Files
aevolution/docs/architecture/PERSISTENCE_ARCHITECTURE.md
T

10 KiB

Persistence and region authority

M7.1 now scopes the first implementation in ADR-0058 and REGION_DURABILITY.md: full checkpoint batches, not recovery from semantic events or an unimplemented input journal. Candidate uploads become published committed snapshots only in the fenced head/outcome transaction. The broader replay and cross-region design below remains a production direction. The state audit explicitly identifies missing lineage/population integration; environmental fixture recovery cannot satisfy those acceptance criteria alone.

Status: proposed production contract; source-of-truth PostgreSQL and disposable workers follow locked requirements. M0 implements only an in-memory local proof snapshot. It has no durable journal, database, lease or crash-consistent file writer.

Data ownership and recovery

Data Proposed owner and retention
Accounts/provider subjects, worlds, regions Versioned PostgreSQL relational metadata
Immutable genomes and ancestry/history PostgreSQL metadata/events; never discard extinct genomes
Active simulation Region worker memory, recoverable from committed checkpoint/replay
Snapshot/chunk payloads Immutable S3-compatible objects, strong content digest
Snapshot manifests and committed replay ranges PostgreSQL transactions; references verified objects
Notifications/projections Transactional outbox; optional JetStream, idempotent consumers
Ephemeral routing/cache In-process first; Redis only with measured need, never sole durable state

Proposed durable keys: world_id, region_id, epoch, simulation_version, world_format_version, codec_version, generator_version, snapshot_id, tick, covered_sequence, content_digest. Organism, genome and runtime entity identities remain distinct. All cross-world keys include world identity. Migration SQL is versioned when introduced at M7; do not create unused tables in M0.

Recovery is not semantic events alone: deaths/births do not reconstruct intervening movement, external inputs or random decisions. Persist contiguous deterministic replay batches containing ordered admitted commands, exogenous inputs, tick range, versions and resulting state hash. Preserve required RNG/state/checkpoint data. Use strict append-only domain history for irreversible ancestry/lifecycle outcomes; use snapshots plus replay for mutable region state, not event sourcing every scalar. Batch transactions; no PostgreSQL write per simulation tick.

sequenceDiagram
  participant W as Worker
  participant O as Object store
  participant D as PostgreSQL
  participant C as Client
  W->>W: Simulate ordered ticks into pending batch
  W->>O: Upload immutable replay payload if externalized / verify
  W->>D: Fenced transaction: replay range + domain outcomes + outbox
  D-->>W: Durable commit or ambiguous failure
  W-->>C: Final outcome only after confirmed/reconciled commit
  W->>O: Upload checkpoint bytes / verify digest
  W->>D: Fenced publish manifest with covered sequence
  Note over W,D: On recovery: manifest + complete replay range, verify hash before serving

Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment are distinct. Never present an uncommitted permanent death or reproductive success as final. Acknowledged death/reproduction must reconstruct after worker loss. Ordinary speculative movement may roll back within an explicit bounded replay window; that window and batching latency are M7 experiments. Database failure withholds irreversible outcomes, limits pending memory and pauses before budgets or lease safety expire. A disconnected commit may have succeeded: reconcile its idempotency key before retrying.

Decision IDs store canonical request digests: retry with changed contents is rejected. Track simulated_tick separately from committed_tick. Pending irreversible decisions cannot enable dependent actions; pending death cannot bypass logout. For initial M7, pause the entire region at an irreversible-outcome boundary until commit. A future nonblocking alternative must prove deterministic isolation of every affected interaction; freezing only a dying organism is insufficient. CommitRegionBatch requires the current lease and expected predecessor head. Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG versions, resolved exchanges and required nondeterministic outputs. If initial M7 cannot capture complete replay, evaluate full checkpoint batches explicitly rather than claiming incomplete event replay is recovery.

Fenced region lease

stateDiagram-v2
  Unassigned --> Acquiring: director assignment hint
  Acquiring --> Recovering: transactional epoch increment / lease
  Recovering --> Active: checkpoint + replay + validation complete
  Active --> Draining: handoff or shutdown
  Active --> Fenced: renewal failure / deadline safety margin
  Draining --> Released: final committed checkpoint / release
  Fenced --> Acquiring: new valid ownership acquisition

The director suggests assignment; PostgreSQL arbitrates. Proposed row: RegionLease(world, region, holder, epoch, expires_at). Acquisition locks the row, requires release/expiry using database time, increments a never-reused epoch and records holder/deadline. Renewal compares holder+epoch and unexpired authority. Every durable region mutation transaction locks/checks that same lease row and epoch; DB privileges prevent workers bypassing the guarded interface. The lock serializes ownership acquisition against commits. A process cannot become authoritative merely because it has a Pod identity or receives a NATS message.

Workers fence themselves before their conservative local safety deadline; a pause/resume cannot renew a stale epoch. Incoming commands, routing grants, snapshots and transfer operations carry epoch/revision. Gateways/clients reject stale-authority output under the routing/session protocol. Two processes may hold stale routing views too: require bounded-validity authority registration and fail-closed forwarding on uncertainty. Begin with one admission gateway per region; partition/clock tests gate multiple gateways. Carrying an epoch alone does not fence a gateway unaware of a newer epoch. Two processes may retain memory, but cannot both commit authority. Recovery readiness requires validated replay and a valid current lease; liveness is not proof of ownership. Test long pauses, stale renewals, overlapping acquisition and delayed packets.

Entity handoff across regions

  1. Destination holds a valid compatible region lease and read-only ghost.
  2. Source freezes the authoritative organism at a tick boundary and durably prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
  3. Destination validates and durably stages the exact transferred state.
  4. One transaction checks both epochs and changes the organism ownership mapping from source to destination, recording commit and state reference.
  5. Destination activates only after observing commit and its current lease; source tombstones its authority. Client reroutes with a fresh grant.

Abort is a durable state transition permitted before commit; source resumes only after observing that abort. Timeout alone does not authorize a second copy. Retries return the recorded transfer outcome. Crash at each arrow must yield one owner or a temporarily unavailable organism, never duplication/loss. Epochs and the organism ownership mapping must be enforced inside persistence transactions.

Snapshots, migrations and retention

Snapshots capture coherent committed state: snapshot.tick <= committed_tick with a matching committed replay watermark. They never publish speculative outcomes. Compression and I/O run outside the simulation writer on immutable buffers. Upload immutable bytes, verify length/digest, then publish metadata and replay coverage transactionally under the current fence. Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all objects reachable from live manifests, clones, backups and retained replay. Never delete source replay until a verified recoverable replacement and backup exist.

Version biology rules, world format and wire protocol independently. Unknown required formats/rules fail closed. Upgrade: quiesce/capture consistent world barrier → clone into isolated identity/subjects → migrate → accelerate → compare ecological distributions and histories → explicit approval → compatible world barrier deployment. Do not run adjacent regions under different physical laws. Migration failure reverts code/state before new writes; downgrade after new writes requires a designed reverse migration, not a blind database restore.

Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive host loss. Exact guarantees for acknowledged outcomes require an explicitly verified durable failure domain containing the complete causal recovery chain and referenced payloads before acknowledgment; restoring an older backup cannot silently undo permadeath. That admission/DR requirement is a World0 launch gate.

Implemented M5.7 local file publication boundary

ADR-0048 and the manifest schema implement a single-writer development file backend over the M5.6 codecs. Immutable checksum-named payloads are installed before alternating manifests; atomic replacement of the inactive manifest is the commit point. Recovery validates the selected snapshot and replays its optional one-segment journal into a detached store. Pre-publication failures retain the previous generation; a post-publication interruption requires reopen and recovers the new one. Orphan files are retained. This is not the M7 fenced, multi-worker or host-loss durability contract above.

Implemented M5.6 local world codec boundary

ADR-0047 and the codec contract implement bounded canonical buffers carrying world/frame IDs, base versions, covered command high-water, resident cells, debris and exact retry state. Detached replay verifies predecessor and final state. This remains an in-memory codec beneath M5.7's local file publication adapter.