Files
aevolution/docs/architecture/PERSISTENCE_ARCHITECTURE.md
T
dingo 39509ecd40
Foundation / checks (ecs) (push) Successful in 2m51s
Foundation / checks (deploy) (push) Successful in 5m18s
Foundation / checks (network) (push) Successful in 25m49s
Foundation / checks (client) (push) Successful in 26m34s
Foundation / checks (tsan) (push) Successful in 37m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 47m59s
Foundation / checks (postgres) (push) Canceled after 47m53s
feat(persistence): define retained-plan checkpoint upgrade candidates
2026-10-19 14:17:41 -04:00

296 lines
19 KiB
Markdown

# Persistence and region authority
[ADR-0167](../adr/0167-retained-plan-checkpoint-upgrade.md) defines portable
raw-only bundle 12 and the explicit no-tick bundle-11 -> 12 checkpoint candidate.
Full source restore preserves expressed biology/history while initializing
retained offspring plans. Reader support is separate from SQL publication;
there is no population mapping, automatic upgrade or new durable entry point.
See [the portable upgrade contract](../../schemas/cellular-inheritance-upgrade.md).
[ADR-0166](../adr/0166-retained-plan-replay-traces.md) adds portable raw trace 4
exclusively for the retained-plan owner 8/rules 10. Replay and action matching
preserve expressed/inherited meanings and return outcomes only after the entire
candidate succeeds. Existing trace defaults, SQL bundles, population traces and
result admission remain unchanged. This is not a durable upgrade or live entry.
ADR-0158/schema 53 admits the explicit same-profile bundle-10 -> 11 maturation
upgrade through the existing immutable version journal and lease/head fence.
Recovery rederives the complete retained transformation; raw trace 3 and population
batch 2 keep their layouts. Profile changes preserve the tuple. No new genesis,
automatic upgrade or default change. [ADR-0160](../adr/0160-portable-maturing-wildlife-transfer.md)
adds portable equal-rules-9 transfer candidates and separately identified trace 2.
[ADR-0161](../adr/0161-durable-maturing-wildlife-transfer.md)/schema 54 admits exact
bundle 10/trace 1 or bundle 11/trace 2 pairs through the existing preparation,
staging, paired receipt, recovery and pending-handoff census workflow. All current
fences, head bindings, immutable evidence and complete replay remain required.
Mixed pairs and cross-rule traces fail; no new table or biological rule is added.
[ADR-0159](../adr/0159-maturing-cellular-census.md) admits its exact
retained stocks to census projection 1 and the existing verified history and
registered snapshot readers. Mixed rules-8/rules-9 regional cuts fail the complete
query. Unavailable evidence cannot become zero life. See
[the upgrade contract](../../schemas/cellular-maturation-upgrade.md).
M12.7c/schema 52 adds [pending migration evidence](../../schemas/ecology/census-pending-migrations-v1.md)
to registered snapshots. Pending source/destination pointers, namespace, registry
and heads share one metadata view. Exact captured capsules/ghosts are fully
replayed after capture, even if settlement commits meanwhile. The source head
already owns prepared/staged life; ghosts add no census stock. Missing/false
evidence fails the entire result. This is existing handoff evidence reconciliation,
not coverage of every viable store or an accepted terminal-extinction record.
M12.7b/schema 51 adds [database history domains](../../schemas/ecology/database-history-domain-v1.md).
Registered census evidence identifies a world by `(history_domain, world_id)`;
domain, registry and heads share one metadata view. Supported restores fork the
domain while quarantined in the installation transaction, retaining parent/request
and exact restored head cuts before verification/readmission. Initial adoption
and legacy backups cannot reconstruct unknown original context. Arbitrary trusted
database copies outside that workflow can copy a domain. Cross-domain clone
provenance, viable-store reconciliation and durable accepted extinction remain open.
M12.7a/schema 50 adds [registered-world coverage](../../schemas/ecology/registered-world-census-coverage-v1.md).
The existing authority registry supplies every required region/frame, without
activity, expiry, profile or checkpoint filters. The bounded read function and
head selection share one database snapshot; resources must match the entire
registered set. Missing/unusable owners and a 65th region fail the query. There
is no new catalog table or registration write. Schema 51 adds the namespace above;
other viable stores and accepted global extinction reconciliation remain separate.
M12.6d captures declared regional heads in one short read-only repeatable-read
transaction, closes it, then fully replays each exact immutable cut before
constructing a scoped census. Atomic migration commits cannot split the selected
head set when both endpoints are declared. Failed metadata capture rolls back;
failed cleanup or an exception disconnects the session. Full replay remains
outside the metadata transaction, with existing per-call evidence locks.
See [ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md). The caller's
list is still not complete world coverage or external branch identity.
M12.6c's portable [scoped census set](../../schemas/ecology/scoped-cellular-census-set-v1.md)
combines a bounded declared set of regional observations, including replayed
history entries. It verifies exact common boundaries, complete shared genome
records and disjoint resident organism IDs. Callers retain the verified commit
references separately. Declared coverage is not a durable world catalog, and
same-tick cuts are not by themselves a reconciled migration watermark.
M12.6b derives regional census history from the existing immutable biological
commit journal. A commit and its exact payload are the canonical event, already
atomic with the head; no duplicate census-write window or new SQL table is added.
Explicit historical recovery reuses the full current-head replay checks. Pages
are bounded to eight consecutive commits and pinned to an exact regional cut;
unavailable/false evidence fails the entire page. Idle connections prevent a
writer's uncommitted cuts from appearing durable. See
[ADR-0151](../adr/0151-committed-cellular-census-history.md). This is regional
history, not branch/world completeness, a client endpoint or extinction authority.
M12.4f defines cellular bundle 10 for owner-7 boundary accounting, with an explicit
same-profile, no-tick bundle-9 upgrade. M12.4g/schema 46 admits it through the
existing immutable version-upgrade journal and lease/head fence. Recovery
rederives the complete source/target transformation. Profile transitions preserve
either tuple; bundle 9 requires population batch 1 and bundle 10 requires batch 2.
Portable reader support grants no publication authority and no worker selects an
upgrade automatically. Atomic paired transfer publication remains required.
See [ADR-0145](../adr/0145-boundary-accounting-checkpoint-upgrade.md) and
[ADR-0146](../adr/0146-durable-boundary-accounting-upgrade.md).
M12.4h/schema 47 retains source migration preparation, immutable explicit abort
and a bounded pending freeze across worker replacement. Complete recovery replay
uses both retained manifests and a verified destination capsule in the source's
object namespace. Both biological heads stay unchanged; destination inspection
is optimistic and grants no destination authority. Source publication and new
interval/player reservations are blocked until explicit settlement. See
[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md).
M12.4i/schema 48 adds destination-authorized staging under ordered endpoint locks
and the exact retained preparation/heads. A verified source capsule and derived
destination ghost stay read-only in the destination namespace. Both roles pause
publication, interval/player admission and host readiness across replacement.
Explicit source abort releases both pointers atomically; stage/abort races cannot
activate or strand stock. See [ADR-0148](../adr/0148-durable-cellular-migration-staging.md).
M12.4j/schema 49 now publishes both results and an immutable common receipt in
one transaction under both current fences. Abort is exclusive with commit;
second-insert failure rolls back both heads and pointer settlement. Recovery
replays both cuts and binds both results to the stage. Frozen hosts reload and
verify a changed head before readiness, rather than inferring activation from
pointer absence. See [ADR-0149](../adr/0149-atomic-cellular-migration-publication.md).
M12.4k qualifies independent loopback TCP/TLS request/receipt loss and either
owner's actual lease expiry, replacement and healed stale fencing in both
representations. Independent database observation checks the outcomes. See
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
M12.4l extends local qualification with capsule/result directory-sync failures and
actual object read-permission denial after commit. Current-payload I/O failure
pauses complete recovery for retry; unavailable retained counterpart evidence
quarantines. Replacement fully verifies the original pair under newer authority. See
[storage scope](../engineering/M12_MIGRATION_STORAGE_FAULTS.md). Remote storage-host,
physical media and orchestrator failures remain separate evidence.
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
recovery from semantic events or an unimplemented input journal. Candidate uploads
become published committed snapshots only in the fenced head/outcome transaction.
The broader replay and cross-region design below remains a production direction.
The state audit explicitly identifies missing lineage/population integration;
environmental fixture recovery cannot satisfy those acceptance criteria alone.
Status: proposed production contract; source-of-truth PostgreSQL and disposable
workers follow locked requirements. M0 implements only an in-memory local proof
snapshot. It has no durable journal, database, lease or crash-consistent file writer.
## Data ownership and recovery
| Data | Proposed owner and retention |
|---|---|
| Accounts/provider subjects, worlds, regions | Versioned PostgreSQL relational metadata |
| Immutable genomes and ancestry/history | PostgreSQL metadata/events; never discard extinct genomes |
| Active simulation | Region worker memory, recoverable from committed checkpoint/replay |
| Snapshot/chunk payloads | Immutable S3-compatible objects, strong content digest |
| Snapshot manifests and committed replay ranges | PostgreSQL transactions; references verified objects |
| Notifications/projections | Transactional outbox; optional JetStream, idempotent consumers |
| Ephemeral routing/cache | In-process first; Redis only with measured need, never sole durable state |
Proposed durable keys: world_id, region_id, epoch, simulation_version,
world_format_version, codec_version, generator_version, snapshot_id, tick,
covered_sequence, content_digest. Organism, genome and runtime entity identities
remain distinct. All cross-world keys include world identity. Migration SQL is
versioned when introduced at M7; do not create unused tables in M0.
Recovery is not semantic events alone: deaths/births do not reconstruct intervening
movement, external inputs or random decisions. Persist contiguous deterministic
replay batches containing ordered admitted commands, exogenous inputs, tick range,
versions and resulting state hash. Preserve required RNG/state/checkpoint data.
Use strict append-only domain history for irreversible ancestry/lifecycle outcomes;
use snapshots plus replay for mutable region state, not event sourcing every scalar.
Batch transactions; no PostgreSQL write per simulation tick.
```mermaid
sequenceDiagram
participant W as Worker
participant O as Object store
participant D as PostgreSQL
participant C as Client
W->>W: Simulate ordered ticks into pending batch
W->>O: Upload immutable replay payload if externalized / verify
W->>D: Fenced transaction: replay range + domain outcomes + outbox
D-->>W: Durable commit or ambiguous failure
W-->>C: Final outcome only after confirmed/reconciled commit
W->>O: Upload checkpoint bytes / verify digest
W->>D: Fenced publish manifest with covered sequence
Note over W,D: On recovery: manifest + complete replay range, verify hash before serving
```
Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment
are distinct. Never present an uncommitted permanent death or reproductive success
as final. Acknowledged death/reproduction must reconstruct after worker loss.
Ordinary speculative movement may roll back within an explicit bounded replay
window; that window and batching latency are M7 experiments. Database failure
withholds irreversible outcomes, limits pending memory and pauses before budgets
or lease safety expire. A disconnected commit may have succeeded: reconcile its
idempotency key before retrying.
Decision IDs store canonical request digests: retry with changed contents is
rejected. Track simulated_tick separately from committed_tick. Pending irreversible
decisions cannot enable dependent actions; pending death cannot bypass logout.
For initial M7, pause the entire region at an irreversible-outcome boundary until
commit. A future nonblocking alternative must prove deterministic isolation of
every affected interaction; freezing only a dying organism is insufficient.
CommitRegionBatch requires the current lease and expected predecessor head.
Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG
versions, resolved exchanges and required nondeterministic outputs. If initial M7
cannot capture complete replay, evaluate full checkpoint batches explicitly rather
than claiming incomplete event replay is recovery.
## Fenced region lease
```mermaid
stateDiagram-v2
Unassigned --> Acquiring: director assignment hint
Acquiring --> Recovering: transactional epoch increment / lease
Recovering --> Active: checkpoint + replay + validation complete
Active --> Draining: handoff or shutdown
Active --> Fenced: renewal failure / deadline safety margin
Draining --> Released: final committed checkpoint / release
Fenced --> Acquiring: new valid ownership acquisition
```
The director suggests assignment; PostgreSQL arbitrates. Proposed row:
`RegionLease(world, region, holder, epoch, expires_at)`. Acquisition locks the row,
requires release/expiry using database time, increments a never-reused epoch and
records holder/deadline. Renewal compares holder+epoch and unexpired authority.
Every durable region mutation transaction locks/checks that same lease row and
epoch; DB privileges prevent workers bypassing the guarded interface. The lock
serializes ownership acquisition against commits. A process cannot become
authoritative merely because it has a Pod identity or receives a NATS message.
Workers fence themselves before their conservative local safety deadline; a
pause/resume cannot renew a stale epoch. Incoming commands, routing grants,
snapshots and transfer operations carry epoch/revision. Gateways/clients reject
stale-authority output under the routing/session protocol. Two processes may
hold stale routing views too: require bounded-validity authority registration
and fail-closed forwarding on uncertainty. Begin with one admission gateway per
region; partition/clock tests gate multiple gateways. Carrying an epoch alone
does not fence a gateway unaware of a newer epoch. Two processes may
retain memory, but cannot both commit authority. Recovery readiness requires
validated replay and a valid current lease; liveness is not proof of ownership.
Test long pauses, stale renewals, overlapping acquisition and delayed packets.
## Entity handoff across regions
1. Destination holds a valid compatible region lease and read-only ghost.
2. Source freezes the authoritative organism at a tick boundary and durably
prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
3. Destination validates and durably stages the exact transferred state.
4. One transaction checks both epochs and changes the organism ownership mapping
from source to destination, recording commit and state reference.
5. Destination activates only after observing commit and its current lease;
source tombstones its authority. Client reroutes with a fresh grant.
Abort is a durable state transition permitted before commit; source resumes only
after observing that abort. Timeout alone does not authorize a second copy.
Retries return the recorded transfer outcome. Crash at each arrow must yield one
owner or a temporarily unavailable organism, never duplication/loss. Epochs and
the organism ownership mapping must be enforced inside persistence transactions.
## Snapshots, migrations and retention
Snapshots capture coherent committed state: snapshot.tick <= committed_tick with
a matching committed replay watermark. They never publish speculative outcomes.
Compression and I/O run outside the simulation writer on immutable buffers.
Upload immutable bytes, verify length/digest, then
publish metadata and replay coverage transactionally under the current fence.
Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all
objects reachable from live manifests, clones, backups and retained replay. Never
delete source replay until a verified recoverable replacement and backup exist.
Version biology rules, world format and wire protocol independently. Unknown
required formats/rules fail closed. Upgrade: quiesce/capture consistent world
barrier → clone into isolated identity/subjects → migrate → accelerate → compare
ecological distributions and histories → explicit approval → compatible world
barrier deployment. Do not run adjacent regions under different physical laws.
Migration failure reverts code/state before new writes; downgrade after new writes
requires a designed reverse migration, not a blind database restore.
Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive
host loss. Exact guarantees for acknowledged outcomes require an explicitly
verified durable failure domain containing the complete causal recovery chain
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
permadeath. That admission/DR requirement is a World0 launch gate.
## Implemented M5.7 local file publication boundary
[ADR-0048](../adr/0048-local-publication.md) and
[the manifest schema](../../schemas/local-world-manifest.md) implement a single-writer
development file backend over the M5.6 codecs. Immutable checksum-named payloads
are installed before alternating manifests; atomic replacement of the inactive
manifest is the commit point. Recovery validates the selected snapshot and replays
its optional one-segment journal into a detached store. Pre-publication failures
retain the previous generation; a post-publication interruption requires reopen
and recovers the new one. Orphan files are retained. This is not the M7 fenced,
multi-worker or host-loss durability contract above.
## Implemented M5.6 local world codec boundary
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
command high-water, resident cells, debris and exact retry state. Detached replay
verifies predecessor and final state. This remains an in-memory codec beneath
M5.7's local file publication adapter.