Files
aevolution/docs/architecture/PERSISTENCE_ARCHITECTURE.md
T

173 lines
10 KiB
Markdown

# Persistence and region authority
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
recovery from semantic events or an unimplemented input journal. Candidate uploads
become published committed snapshots only in the fenced head/outcome transaction.
The broader replay and cross-region design below remains a production direction.
The state audit explicitly identifies missing lineage/population integration;
environmental fixture recovery cannot satisfy those acceptance criteria alone.
Status: proposed production contract; source-of-truth PostgreSQL and disposable
workers follow locked requirements. M0 implements only an in-memory local proof
snapshot. It has no durable journal, database, lease or crash-consistent file writer.
## Data ownership and recovery
| Data | Proposed owner and retention |
|---|---|
| Accounts/provider subjects, worlds, regions | Versioned PostgreSQL relational metadata |
| Immutable genomes and ancestry/history | PostgreSQL metadata/events; never discard extinct genomes |
| Active simulation | Region worker memory, recoverable from committed checkpoint/replay |
| Snapshot/chunk payloads | Immutable S3-compatible objects, strong content digest |
| Snapshot manifests and committed replay ranges | PostgreSQL transactions; references verified objects |
| Notifications/projections | Transactional outbox; optional JetStream, idempotent consumers |
| Ephemeral routing/cache | In-process first; Redis only with measured need, never sole durable state |
Proposed durable keys: world_id, region_id, epoch, simulation_version,
world_format_version, codec_version, generator_version, snapshot_id, tick,
covered_sequence, content_digest. Organism, genome and runtime entity identities
remain distinct. All cross-world keys include world identity. Migration SQL is
versioned when introduced at M7; do not create unused tables in M0.
Recovery is not semantic events alone: deaths/births do not reconstruct intervening
movement, external inputs or random decisions. Persist contiguous deterministic
replay batches containing ordered admitted commands, exogenous inputs, tick range,
versions and resulting state hash. Preserve required RNG/state/checkpoint data.
Use strict append-only domain history for irreversible ancestry/lifecycle outcomes;
use snapshots plus replay for mutable region state, not event sourcing every scalar.
Batch transactions; no PostgreSQL write per simulation tick.
```mermaid
sequenceDiagram
participant W as Worker
participant O as Object store
participant D as PostgreSQL
participant C as Client
W->>W: Simulate ordered ticks into pending batch
W->>O: Upload immutable replay payload if externalized / verify
W->>D: Fenced transaction: replay range + domain outcomes + outbox
D-->>W: Durable commit or ambiguous failure
W-->>C: Final outcome only after confirmed/reconciled commit
W->>O: Upload checkpoint bytes / verify digest
W->>D: Fenced publish manifest with covered sequence
Note over W,D: On recovery: manifest + complete replay range, verify hash before serving
```
Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment
are distinct. Never present an uncommitted permanent death or reproductive success
as final. Acknowledged death/reproduction must reconstruct after worker loss.
Ordinary speculative movement may roll back within an explicit bounded replay
window; that window and batching latency are M7 experiments. Database failure
withholds irreversible outcomes, limits pending memory and pauses before budgets
or lease safety expire. A disconnected commit may have succeeded: reconcile its
idempotency key before retrying.
Decision IDs store canonical request digests: retry with changed contents is
rejected. Track simulated_tick separately from committed_tick. Pending irreversible
decisions cannot enable dependent actions; pending death cannot bypass logout.
For initial M7, pause the entire region at an irreversible-outcome boundary until
commit. A future nonblocking alternative must prove deterministic isolation of
every affected interaction; freezing only a dying organism is insufficient.
CommitRegionBatch requires the current lease and expected predecessor head.
Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG
versions, resolved exchanges and required nondeterministic outputs. If initial M7
cannot capture complete replay, evaluate full checkpoint batches explicitly rather
than claiming incomplete event replay is recovery.
## Fenced region lease
```mermaid
stateDiagram-v2
Unassigned --> Acquiring: director assignment hint
Acquiring --> Recovering: transactional epoch increment / lease
Recovering --> Active: checkpoint + replay + validation complete
Active --> Draining: handoff or shutdown
Active --> Fenced: renewal failure / deadline safety margin
Draining --> Released: final committed checkpoint / release
Fenced --> Acquiring: new valid ownership acquisition
```
The director suggests assignment; PostgreSQL arbitrates. Proposed row:
`RegionLease(world, region, holder, epoch, expires_at)`. Acquisition locks the row,
requires release/expiry using database time, increments a never-reused epoch and
records holder/deadline. Renewal compares holder+epoch and unexpired authority.
Every durable region mutation transaction locks/checks that same lease row and
epoch; DB privileges prevent workers bypassing the guarded interface. The lock
serializes ownership acquisition against commits. A process cannot become
authoritative merely because it has a Pod identity or receives a NATS message.
Workers fence themselves before their conservative local safety deadline; a
pause/resume cannot renew a stale epoch. Incoming commands, routing grants,
snapshots and transfer operations carry epoch/revision. Gateways/clients reject
stale-authority output under the routing/session protocol. Two processes may
hold stale routing views too: require bounded-validity authority registration
and fail-closed forwarding on uncertainty. Begin with one admission gateway per
region; partition/clock tests gate multiple gateways. Carrying an epoch alone
does not fence a gateway unaware of a newer epoch. Two processes may
retain memory, but cannot both commit authority. Recovery readiness requires
validated replay and a valid current lease; liveness is not proof of ownership.
Test long pauses, stale renewals, overlapping acquisition and delayed packets.
## Entity handoff across regions
1. Destination holds a valid compatible region lease and read-only ghost.
2. Source freezes the authoritative organism at a tick boundary and durably
prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
3. Destination validates and durably stages the exact transferred state.
4. One transaction checks both epochs and changes the organism ownership mapping
from source to destination, recording commit and state reference.
5. Destination activates only after observing commit and its current lease;
source tombstones its authority. Client reroutes with a fresh grant.
Abort is a durable state transition permitted before commit; source resumes only
after observing that abort. Timeout alone does not authorize a second copy.
Retries return the recorded transfer outcome. Crash at each arrow must yield one
owner or a temporarily unavailable organism, never duplication/loss. Epochs and
the organism ownership mapping must be enforced inside persistence transactions.
## Snapshots, migrations and retention
Snapshots capture coherent committed state: snapshot.tick <= committed_tick with
a matching committed replay watermark. They never publish speculative outcomes.
Compression and I/O run outside the simulation writer on immutable buffers.
Upload immutable bytes, verify length/digest, then
publish metadata and replay coverage transactionally under the current fence.
Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all
objects reachable from live manifests, clones, backups and retained replay. Never
delete source replay until a verified recoverable replacement and backup exist.
Version biology rules, world format and wire protocol independently. Unknown
required formats/rules fail closed. Upgrade: quiesce/capture consistent world
barrier → clone into isolated identity/subjects → migrate → accelerate → compare
ecological distributions and histories → explicit approval → compatible world
barrier deployment. Do not run adjacent regions under different physical laws.
Migration failure reverts code/state before new writes; downgrade after new writes
requires a designed reverse migration, not a blind database restore.
Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive
host loss. Exact guarantees for acknowledged outcomes require an explicitly
verified durable failure domain containing the complete causal recovery chain
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
permadeath. That admission/DR requirement is a World0 launch gate.
## Implemented M5.7 local file publication boundary
[ADR-0048](../adr/0048-local-publication.md) and
[the manifest schema](../../schemas/local-world-manifest.md) implement a single-writer
development file backend over the M5.6 codecs. Immutable checksum-named payloads
are installed before alternating manifests; atomic replacement of the inactive
manifest is the commit point. Recovery validates the selected snapshot and replays
its optional one-segment journal into a detached store. Pre-publication failures
retain the previous generation; a post-publication interruption requires reopen
and recovers the new one. Orphan files are retained. This is not the M7 fenced,
multi-worker or host-loss durability contract above.
## Implemented M5.6 local world codec boundary
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
command high-water, resident cells, debris and exact retry state. Detached replay
verifies predecessor and final state. This remains an in-memory codec beneath
M5.7's local file publication adapter.