Files
aevolution/docs/architecture/PERSISTENCE_ARCHITECTURE.md
dingo 39509ecd40
Foundation / checks (ecs) (push) Successful in 2m51s
Foundation / checks (deploy) (push) Successful in 5m18s
Foundation / checks (network) (push) Successful in 25m49s
Foundation / checks (client) (push) Successful in 26m34s
Foundation / checks (tsan) (push) Successful in 37m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 47m59s
Foundation / checks (postgres) (push) Canceled after 47m53s
feat(persistence): define retained-plan checkpoint upgrade candidates
2026-10-19 14:17:41 -04:00

19 KiB

Persistence and region authority

ADR-0167 defines portable raw-only bundle 12 and the explicit no-tick bundle-11 -> 12 checkpoint candidate. Full source restore preserves expressed biology/history while initializing retained offspring plans. Reader support is separate from SQL publication; there is no population mapping, automatic upgrade or new durable entry point. See the portable upgrade contract.

ADR-0166 adds portable raw trace 4 exclusively for the retained-plan owner 8/rules 10. Replay and action matching preserve expressed/inherited meanings and return outcomes only after the entire candidate succeeds. Existing trace defaults, SQL bundles, population traces and result admission remain unchanged. This is not a durable upgrade or live entry.

ADR-0158/schema 53 admits the explicit same-profile bundle-10 -> 11 maturation upgrade through the existing immutable version journal and lease/head fence. Recovery rederives the complete retained transformation; raw trace 3 and population batch 2 keep their layouts. Profile changes preserve the tuple. No new genesis, automatic upgrade or default change. ADR-0160 adds portable equal-rules-9 transfer candidates and separately identified trace 2. ADR-0161/schema 54 admits exact bundle 10/trace 1 or bundle 11/trace 2 pairs through the existing preparation, staging, paired receipt, recovery and pending-handoff census workflow. All current fences, head bindings, immutable evidence and complete replay remain required. Mixed pairs and cross-rule traces fail; no new table or biological rule is added. ADR-0159 admits its exact retained stocks to census projection 1 and the existing verified history and registered snapshot readers. Mixed rules-8/rules-9 regional cuts fail the complete query. Unavailable evidence cannot become zero life. See the upgrade contract.

M12.7c/schema 52 adds pending migration evidence to registered snapshots. Pending source/destination pointers, namespace, registry and heads share one metadata view. Exact captured capsules/ghosts are fully replayed after capture, even if settlement commits meanwhile. The source head already owns prepared/staged life; ghosts add no census stock. Missing/false evidence fails the entire result. This is existing handoff evidence reconciliation, not coverage of every viable store or an accepted terminal-extinction record.

M12.7b/schema 51 adds database history domains. Registered census evidence identifies a world by (history_domain, world_id); domain, registry and heads share one metadata view. Supported restores fork the domain while quarantined in the installation transaction, retaining parent/request and exact restored head cuts before verification/readmission. Initial adoption and legacy backups cannot reconstruct unknown original context. Arbitrary trusted database copies outside that workflow can copy a domain. Cross-domain clone provenance, viable-store reconciliation and durable accepted extinction remain open.

M12.7a/schema 50 adds registered-world coverage. The existing authority registry supplies every required region/frame, without activity, expiry, profile or checkpoint filters. The bounded read function and head selection share one database snapshot; resources must match the entire registered set. Missing/unusable owners and a 65th region fail the query. There is no new catalog table or registration write. Schema 51 adds the namespace above; other viable stores and accepted global extinction reconciliation remain separate.

M12.6d captures declared regional heads in one short read-only repeatable-read transaction, closes it, then fully replays each exact immutable cut before constructing a scoped census. Atomic migration commits cannot split the selected head set when both endpoints are declared. Failed metadata capture rolls back; failed cleanup or an exception disconnects the session. Full replay remains outside the metadata transaction, with existing per-call evidence locks. See ADR-0153. The caller's list is still not complete world coverage or external branch identity.

M12.6c's portable scoped census set combines a bounded declared set of regional observations, including replayed history entries. It verifies exact common boundaries, complete shared genome records and disjoint resident organism IDs. Callers retain the verified commit references separately. Declared coverage is not a durable world catalog, and same-tick cuts are not by themselves a reconciled migration watermark.

M12.6b derives regional census history from the existing immutable biological commit journal. A commit and its exact payload are the canonical event, already atomic with the head; no duplicate census-write window or new SQL table is added. Explicit historical recovery reuses the full current-head replay checks. Pages are bounded to eight consecutive commits and pinned to an exact regional cut; unavailable/false evidence fails the entire page. Idle connections prevent a writer's uncommitted cuts from appearing durable. See ADR-0151. This is regional history, not branch/world completeness, a client endpoint or extinction authority.

M12.4f defines cellular bundle 10 for owner-7 boundary accounting, with an explicit same-profile, no-tick bundle-9 upgrade. M12.4g/schema 46 admits it through the existing immutable version-upgrade journal and lease/head fence. Recovery rederives the complete source/target transformation. Profile transitions preserve either tuple; bundle 9 requires population batch 1 and bundle 10 requires batch 2. Portable reader support grants no publication authority and no worker selects an upgrade automatically. Atomic paired transfer publication remains required. See ADR-0145 and ADR-0146.

M12.4h/schema 47 retains source migration preparation, immutable explicit abort and a bounded pending freeze across worker replacement. Complete recovery replay uses both retained manifests and a verified destination capsule in the source's object namespace. Both biological heads stay unchanged; destination inspection is optimistic and grants no destination authority. Source publication and new interval/player reservations are blocked until explicit settlement. See ADR-0147.

M12.4i/schema 48 adds destination-authorized staging under ordered endpoint locks and the exact retained preparation/heads. A verified source capsule and derived destination ghost stay read-only in the destination namespace. Both roles pause publication, interval/player admission and host readiness across replacement. Explicit source abort releases both pointers atomically; stage/abort races cannot activate or strand stock. See ADR-0148. M12.4j/schema 49 now publishes both results and an immutable common receipt in one transaction under both current fences. Abort is exclusive with commit; second-insert failure rolls back both heads and pointer settlement. Recovery replays both cuts and binds both results to the stage. Frozen hosts reload and verify a changed head before readiness, rather than inferring activation from pointer absence. See ADR-0149. M12.4k qualifies independent loopback TCP/TLS request/receipt loss and either owner's actual lease expiry, replacement and healed stale fencing in both representations. Independent database observation checks the outcomes. See transport scope. Multi-host network/kernel, object-store/orchestrator failures and linked ecology remain separate. M12.4l extends local qualification with capsule/result directory-sync failures and actual object read-permission denial after commit. Current-payload I/O failure pauses complete recovery for retry; unavailable retained counterpart evidence quarantines. Replacement fully verifies the original pair under newer authority. See storage scope. Remote storage-host, physical media and orchestrator failures remain separate evidence.

M7.1 now scopes the first implementation in ADR-0058 and REGION_DURABILITY.md: full checkpoint batches, not recovery from semantic events or an unimplemented input journal. Candidate uploads become published committed snapshots only in the fenced head/outcome transaction. The broader replay and cross-region design below remains a production direction. The state audit explicitly identifies missing lineage/population integration; environmental fixture recovery cannot satisfy those acceptance criteria alone.

Status: proposed production contract; source-of-truth PostgreSQL and disposable workers follow locked requirements. M0 implements only an in-memory local proof snapshot. It has no durable journal, database, lease or crash-consistent file writer.

Data ownership and recovery

Data Proposed owner and retention
Accounts/provider subjects, worlds, regions Versioned PostgreSQL relational metadata
Immutable genomes and ancestry/history PostgreSQL metadata/events; never discard extinct genomes
Active simulation Region worker memory, recoverable from committed checkpoint/replay
Snapshot/chunk payloads Immutable S3-compatible objects, strong content digest
Snapshot manifests and committed replay ranges PostgreSQL transactions; references verified objects
Notifications/projections Transactional outbox; optional JetStream, idempotent consumers
Ephemeral routing/cache In-process first; Redis only with measured need, never sole durable state

Proposed durable keys: world_id, region_id, epoch, simulation_version, world_format_version, codec_version, generator_version, snapshot_id, tick, covered_sequence, content_digest. Organism, genome and runtime entity identities remain distinct. All cross-world keys include world identity. Migration SQL is versioned when introduced at M7; do not create unused tables in M0.

Recovery is not semantic events alone: deaths/births do not reconstruct intervening movement, external inputs or random decisions. Persist contiguous deterministic replay batches containing ordered admitted commands, exogenous inputs, tick range, versions and resulting state hash. Preserve required RNG/state/checkpoint data. Use strict append-only domain history for irreversible ancestry/lifecycle outcomes; use snapshots plus replay for mutable region state, not event sourcing every scalar. Batch transactions; no PostgreSQL write per simulation tick.

sequenceDiagram
  participant W as Worker
  participant O as Object store
  participant D as PostgreSQL
  participant C as Client
  W->>W: Simulate ordered ticks into pending batch
  W->>O: Upload immutable replay payload if externalized / verify
  W->>D: Fenced transaction: replay range + domain outcomes + outbox
  D-->>W: Durable commit or ambiguous failure
  W-->>C: Final outcome only after confirmed/reconciled commit
  W->>O: Upload checkpoint bytes / verify digest
  W->>D: Fenced publish manifest with covered sequence
  Note over W,D: On recovery: manifest + complete replay range, verify hash before serving

Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment are distinct. Never present an uncommitted permanent death or reproductive success as final. Acknowledged death/reproduction must reconstruct after worker loss. Ordinary speculative movement may roll back within an explicit bounded replay window; that window and batching latency are M7 experiments. Database failure withholds irreversible outcomes, limits pending memory and pauses before budgets or lease safety expire. A disconnected commit may have succeeded: reconcile its idempotency key before retrying.

Decision IDs store canonical request digests: retry with changed contents is rejected. Track simulated_tick separately from committed_tick. Pending irreversible decisions cannot enable dependent actions; pending death cannot bypass logout. For initial M7, pause the entire region at an irreversible-outcome boundary until commit. A future nonblocking alternative must prove deterministic isolation of every affected interaction; freezing only a dying organism is insufficient. CommitRegionBatch requires the current lease and expected predecessor head. Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG versions, resolved exchanges and required nondeterministic outputs. If initial M7 cannot capture complete replay, evaluate full checkpoint batches explicitly rather than claiming incomplete event replay is recovery.

Fenced region lease

stateDiagram-v2
  Unassigned --> Acquiring: director assignment hint
  Acquiring --> Recovering: transactional epoch increment / lease
  Recovering --> Active: checkpoint + replay + validation complete
  Active --> Draining: handoff or shutdown
  Active --> Fenced: renewal failure / deadline safety margin
  Draining --> Released: final committed checkpoint / release
  Fenced --> Acquiring: new valid ownership acquisition

The director suggests assignment; PostgreSQL arbitrates. Proposed row: RegionLease(world, region, holder, epoch, expires_at). Acquisition locks the row, requires release/expiry using database time, increments a never-reused epoch and records holder/deadline. Renewal compares holder+epoch and unexpired authority. Every durable region mutation transaction locks/checks that same lease row and epoch; DB privileges prevent workers bypassing the guarded interface. The lock serializes ownership acquisition against commits. A process cannot become authoritative merely because it has a Pod identity or receives a NATS message.

Workers fence themselves before their conservative local safety deadline; a pause/resume cannot renew a stale epoch. Incoming commands, routing grants, snapshots and transfer operations carry epoch/revision. Gateways/clients reject stale-authority output under the routing/session protocol. Two processes may hold stale routing views too: require bounded-validity authority registration and fail-closed forwarding on uncertainty. Begin with one admission gateway per region; partition/clock tests gate multiple gateways. Carrying an epoch alone does not fence a gateway unaware of a newer epoch. Two processes may retain memory, but cannot both commit authority. Recovery readiness requires validated replay and a valid current lease; liveness is not proof of ownership. Test long pauses, stale renewals, overlapping acquisition and delayed packets.

Entity handoff across regions

  1. Destination holds a valid compatible region lease and read-only ghost.
  2. Source freezes the authoritative organism at a tick boundary and durably prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
  3. Destination validates and durably stages the exact transferred state.
  4. One transaction checks both epochs and changes the organism ownership mapping from source to destination, recording commit and state reference.
  5. Destination activates only after observing commit and its current lease; source tombstones its authority. Client reroutes with a fresh grant.

Abort is a durable state transition permitted before commit; source resumes only after observing that abort. Timeout alone does not authorize a second copy. Retries return the recorded transfer outcome. Crash at each arrow must yield one owner or a temporarily unavailable organism, never duplication/loss. Epochs and the organism ownership mapping must be enforced inside persistence transactions.

Snapshots, migrations and retention

Snapshots capture coherent committed state: snapshot.tick <= committed_tick with a matching committed replay watermark. They never publish speculative outcomes. Compression and I/O run outside the simulation writer on immutable buffers. Upload immutable bytes, verify length/digest, then publish metadata and replay coverage transactionally under the current fence. Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all objects reachable from live manifests, clones, backups and retained replay. Never delete source replay until a verified recoverable replacement and backup exist.

Version biology rules, world format and wire protocol independently. Unknown required formats/rules fail closed. Upgrade: quiesce/capture consistent world barrier → clone into isolated identity/subjects → migrate → accelerate → compare ecological distributions and histories → explicit approval → compatible world barrier deployment. Do not run adjacent regions under different physical laws. Migration failure reverts code/state before new writes; downgrade after new writes requires a designed reverse migration, not a blind database restore.

Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive host loss. Exact guarantees for acknowledged outcomes require an explicitly verified durable failure domain containing the complete causal recovery chain and referenced payloads before acknowledgment; restoring an older backup cannot silently undo permadeath. That admission/DR requirement is a World0 launch gate.

Implemented M5.7 local file publication boundary

ADR-0048 and the manifest schema implement a single-writer development file backend over the M5.6 codecs. Immutable checksum-named payloads are installed before alternating manifests; atomic replacement of the inactive manifest is the commit point. Recovery validates the selected snapshot and replays its optional one-segment journal into a detached store. Pre-publication failures retain the previous generation; a post-publication interruption requires reopen and recovers the new one. Orphan files are retained. This is not the M7 fenced, multi-worker or host-loss durability contract above.

Implemented M5.6 local world codec boundary

ADR-0047 and the codec contract implement bounded canonical buffers carrying world/frame IDs, base versions, covered command high-water, resident cells, debris and exact retry state. Detached replay verifies predecessor and final state. This remains an in-memory codec beneath M5.7's local file publication adapter.