Aeternalis Evolution, a custom-engine persistent evolutionary MMO. This first commit establishes the project skeleton: - Architecture and design docs: project vision, system overview, engineering constitution, roadmap, 19 ADRs, and the simulation, world, network, persistence and infrastructure architectures. - Headless engineering proof: deterministic synthetic particles, component lifecycle, explicit PRNG streams, canonical snapshots and restored execution. - Build and tooling: CMake 3.28+ / Ninja presets, C++23 feature check, clang-format and clang-tidy config, format check script, Windows check script, and a Docker build context. - Tests: unit tests, benchmarks, a snapshot fuzz target and CLI integration wiring. No gameplay, renderer, network service or running deployment yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
8.4 KiB
Persistence and region authority
Status: proposed production contract; source-of-truth PostgreSQL and disposable workers follow locked requirements. M0 implements only an in-memory local proof snapshot. It has no durable journal, database, lease or crash-consistent file writer.
Data ownership and recovery
| Data | Proposed owner and retention |
|---|---|
| Accounts/provider subjects, worlds, regions | Versioned PostgreSQL relational metadata |
| Immutable genomes and ancestry/history | PostgreSQL metadata/events; never discard extinct genomes |
| Active simulation | Region worker memory, recoverable from committed checkpoint/replay |
| Snapshot/chunk payloads | Immutable S3-compatible objects, strong content digest |
| Snapshot manifests and committed replay ranges | PostgreSQL transactions; references verified objects |
| Notifications/projections | Transactional outbox; optional JetStream, idempotent consumers |
| Ephemeral routing/cache | In-process first; Redis only with measured need, never sole durable state |
Proposed durable keys: world_id, region_id, epoch, simulation_version, world_format_version, codec_version, generator_version, snapshot_id, tick, covered_sequence, content_digest. Organism, genome and runtime entity identities remain distinct. All cross-world keys include world identity. Migration SQL is versioned when introduced at M7; do not create unused tables in M0.
Recovery is not semantic events alone: deaths/births do not reconstruct intervening movement, external inputs or random decisions. Persist contiguous deterministic replay batches containing ordered admitted commands, exogenous inputs, tick range, versions and resulting state hash. Preserve required RNG/state/checkpoint data. Use strict append-only domain history for irreversible ancestry/lifecycle outcomes; use snapshots plus replay for mutable region state, not event sourcing every scalar. Batch transactions; no PostgreSQL write per simulation tick.
sequenceDiagram
participant W as Worker
participant O as Object store
participant D as PostgreSQL
participant C as Client
W->>W: Simulate ordered ticks into pending batch
W->>O: Upload immutable replay payload if externalized / verify
W->>D: Fenced transaction: replay range + domain outcomes + outbox
D-->>W: Durable commit or ambiguous failure
W-->>C: Final outcome only after confirmed/reconciled commit
W->>O: Upload checkpoint bytes / verify digest
W->>D: Fenced publish manifest with covered sequence
Note over W,D: On recovery: manifest + complete replay range, verify hash before serving
Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment are distinct. Never present an uncommitted permanent death or reproductive success as final. Acknowledged death/reproduction must reconstruct after worker loss. Ordinary speculative movement may roll back within an explicit bounded replay window; that window and batching latency are M7 experiments. Database failure withholds irreversible outcomes, limits pending memory and pauses before budgets or lease safety expire. A disconnected commit may have succeeded: reconcile its idempotency key before retrying.
Decision IDs store canonical request digests: retry with changed contents is rejected. Track simulated_tick separately from committed_tick. Pending irreversible decisions cannot enable dependent actions; pending death cannot bypass logout. For initial M7, pause the entire region at an irreversible-outcome boundary until commit. A future nonblocking alternative must prove deterministic isolation of every affected interaction; freezing only a dying organism is insufficient. CommitRegionBatch requires the current lease and expected predecessor head. Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG versions, resolved exchanges and required nondeterministic outputs. If initial M7 cannot capture complete replay, evaluate full checkpoint batches explicitly rather than claiming incomplete event replay is recovery.
Fenced region lease
stateDiagram-v2
Unassigned --> Acquiring: director assignment hint
Acquiring --> Recovering: transactional epoch increment / lease
Recovering --> Active: checkpoint + replay + validation complete
Active --> Draining: handoff or shutdown
Active --> Fenced: renewal failure / deadline safety margin
Draining --> Released: final committed checkpoint / release
Fenced --> Acquiring: new valid ownership acquisition
The director suggests assignment; PostgreSQL arbitrates. Proposed row:
RegionLease(world, region, holder, epoch, expires_at). Acquisition locks the row,
requires release/expiry using database time, increments a never-reused epoch and
records holder/deadline. Renewal compares holder+epoch and unexpired authority.
Every durable region mutation transaction locks/checks that same lease row and
epoch; DB privileges prevent workers bypassing the guarded interface. The lock
serializes ownership acquisition against commits. A process cannot become
authoritative merely because it has a Pod identity or receives a NATS message.
Workers fence themselves before their conservative local safety deadline; a pause/resume cannot renew a stale epoch. Incoming commands, routing grants, snapshots and transfer operations carry epoch/revision. Gateways/clients reject stale-authority output under the routing/session protocol. Two processes may hold stale routing views too: require bounded-validity authority registration and fail-closed forwarding on uncertainty. Begin with one admission gateway per region; partition/clock tests gate multiple gateways. Carrying an epoch alone does not fence a gateway unaware of a newer epoch. Two processes may retain memory, but cannot both commit authority. Recovery readiness requires validated replay and a valid current lease; liveness is not proof of ownership. Test long pauses, stale renewals, overlapping acquisition and delayed packets.
Entity handoff across regions
- Destination holds a valid compatible region lease and read-only ghost.
- Source freezes the authoritative organism at a tick boundary and durably prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
- Destination validates and durably stages the exact transferred state.
- One transaction checks both epochs and changes the organism ownership mapping from source to destination, recording commit and state reference.
- Destination activates only after observing commit and its current lease; source tombstones its authority. Client reroutes with a fresh grant.
Abort is a durable state transition permitted before commit; source resumes only after observing that abort. Timeout alone does not authorize a second copy. Retries return the recorded transfer outcome. Crash at each arrow must yield one owner or a temporarily unavailable organism, never duplication/loss. Epochs and the organism ownership mapping must be enforced inside persistence transactions.
Snapshots, migrations and retention
Snapshots capture coherent committed state: snapshot.tick <= committed_tick with a matching committed replay watermark. They never publish speculative outcomes. Compression and I/O run outside the simulation writer on immutable buffers. Upload immutable bytes, verify length/digest, then publish metadata and replay coverage transactionally under the current fence. Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all objects reachable from live manifests, clones, backups and retained replay. Never delete source replay until a verified recoverable replacement and backup exist.
Version biology rules, world format and wire protocol independently. Unknown required formats/rules fail closed. Upgrade: quiesce/capture consistent world barrier → clone into isolated identity/subjects → migrate → accelerate → compare ecological distributions and histories → explicit approval → compatible world barrier deployment. Do not run adjacent regions under different physical laws. Migration failure reverts code/state before new writes; downgrade after new writes requires a designed reverse migration, not a blind database restore.
Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive host loss. Exact guarantees for acknowledged outcomes require an explicitly verified durable failure domain containing the complete causal recovery chain and referenced payloads before acknowledgment; restoring an older backup cannot silently undo permadeath. That admission/DR requirement is a World0 launch gate.