173 lines
10 KiB
Markdown
173 lines
10 KiB
Markdown
# Persistence and region authority
|
|
|
|
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
|
|
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
|
|
recovery from semantic events or an unimplemented input journal. Candidate uploads
|
|
become published committed snapshots only in the fenced head/outcome transaction.
|
|
The broader replay and cross-region design below remains a production direction.
|
|
The state audit explicitly identifies missing lineage/population integration;
|
|
environmental fixture recovery cannot satisfy those acceptance criteria alone.
|
|
|
|
Status: proposed production contract; source-of-truth PostgreSQL and disposable
|
|
workers follow locked requirements. M0 implements only an in-memory local proof
|
|
snapshot. It has no durable journal, database, lease or crash-consistent file writer.
|
|
|
|
## Data ownership and recovery
|
|
|
|
| Data | Proposed owner and retention |
|
|
|---|---|
|
|
| Accounts/provider subjects, worlds, regions | Versioned PostgreSQL relational metadata |
|
|
| Immutable genomes and ancestry/history | PostgreSQL metadata/events; never discard extinct genomes |
|
|
| Active simulation | Region worker memory, recoverable from committed checkpoint/replay |
|
|
| Snapshot/chunk payloads | Immutable S3-compatible objects, strong content digest |
|
|
| Snapshot manifests and committed replay ranges | PostgreSQL transactions; references verified objects |
|
|
| Notifications/projections | Transactional outbox; optional JetStream, idempotent consumers |
|
|
| Ephemeral routing/cache | In-process first; Redis only with measured need, never sole durable state |
|
|
|
|
Proposed durable keys: world_id, region_id, epoch, simulation_version,
|
|
world_format_version, codec_version, generator_version, snapshot_id, tick,
|
|
covered_sequence, content_digest. Organism, genome and runtime entity identities
|
|
remain distinct. All cross-world keys include world identity. Migration SQL is
|
|
versioned when introduced at M7; do not create unused tables in M0.
|
|
|
|
Recovery is not semantic events alone: deaths/births do not reconstruct intervening
|
|
movement, external inputs or random decisions. Persist contiguous deterministic
|
|
replay batches containing ordered admitted commands, exogenous inputs, tick range,
|
|
versions and resulting state hash. Preserve required RNG/state/checkpoint data.
|
|
Use strict append-only domain history for irreversible ancestry/lifecycle outcomes;
|
|
use snapshots plus replay for mutable region state, not event sourcing every scalar.
|
|
Batch transactions; no PostgreSQL write per simulation tick.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant W as Worker
|
|
participant O as Object store
|
|
participant D as PostgreSQL
|
|
participant C as Client
|
|
W->>W: Simulate ordered ticks into pending batch
|
|
W->>O: Upload immutable replay payload if externalized / verify
|
|
W->>D: Fenced transaction: replay range + domain outcomes + outbox
|
|
D-->>W: Durable commit or ambiguous failure
|
|
W-->>C: Final outcome only after confirmed/reconciled commit
|
|
W->>O: Upload checkpoint bytes / verify digest
|
|
W->>D: Fenced publish manifest with covered sequence
|
|
Note over W,D: On recovery: manifest + complete replay range, verify hash before serving
|
|
```
|
|
|
|
Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment
|
|
are distinct. Never present an uncommitted permanent death or reproductive success
|
|
as final. Acknowledged death/reproduction must reconstruct after worker loss.
|
|
Ordinary speculative movement may roll back within an explicit bounded replay
|
|
window; that window and batching latency are M7 experiments. Database failure
|
|
withholds irreversible outcomes, limits pending memory and pauses before budgets
|
|
or lease safety expire. A disconnected commit may have succeeded: reconcile its
|
|
idempotency key before retrying.
|
|
|
|
Decision IDs store canonical request digests: retry with changed contents is
|
|
rejected. Track simulated_tick separately from committed_tick. Pending irreversible
|
|
decisions cannot enable dependent actions; pending death cannot bypass logout.
|
|
For initial M7, pause the entire region at an irreversible-outcome boundary until
|
|
commit. A future nonblocking alternative must prove deterministic isolation of
|
|
every affected interaction; freezing only a dying organism is insufficient.
|
|
CommitRegionBatch requires the current lease and expected predecessor head.
|
|
Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG
|
|
versions, resolved exchanges and required nondeterministic outputs. If initial M7
|
|
cannot capture complete replay, evaluate full checkpoint batches explicitly rather
|
|
than claiming incomplete event replay is recovery.
|
|
|
|
## Fenced region lease
|
|
|
|
```mermaid
|
|
stateDiagram-v2
|
|
Unassigned --> Acquiring: director assignment hint
|
|
Acquiring --> Recovering: transactional epoch increment / lease
|
|
Recovering --> Active: checkpoint + replay + validation complete
|
|
Active --> Draining: handoff or shutdown
|
|
Active --> Fenced: renewal failure / deadline safety margin
|
|
Draining --> Released: final committed checkpoint / release
|
|
Fenced --> Acquiring: new valid ownership acquisition
|
|
```
|
|
|
|
The director suggests assignment; PostgreSQL arbitrates. Proposed row:
|
|
`RegionLease(world, region, holder, epoch, expires_at)`. Acquisition locks the row,
|
|
requires release/expiry using database time, increments a never-reused epoch and
|
|
records holder/deadline. Renewal compares holder+epoch and unexpired authority.
|
|
Every durable region mutation transaction locks/checks that same lease row and
|
|
epoch; DB privileges prevent workers bypassing the guarded interface. The lock
|
|
serializes ownership acquisition against commits. A process cannot become
|
|
authoritative merely because it has a Pod identity or receives a NATS message.
|
|
|
|
Workers fence themselves before their conservative local safety deadline; a
|
|
pause/resume cannot renew a stale epoch. Incoming commands, routing grants,
|
|
snapshots and transfer operations carry epoch/revision. Gateways/clients reject
|
|
stale-authority output under the routing/session protocol. Two processes may
|
|
hold stale routing views too: require bounded-validity authority registration
|
|
and fail-closed forwarding on uncertainty. Begin with one admission gateway per
|
|
region; partition/clock tests gate multiple gateways. Carrying an epoch alone
|
|
does not fence a gateway unaware of a newer epoch. Two processes may
|
|
retain memory, but cannot both commit authority. Recovery readiness requires
|
|
validated replay and a valid current lease; liveness is not proof of ownership.
|
|
Test long pauses, stale renewals, overlapping acquisition and delayed packets.
|
|
|
|
## Entity handoff across regions
|
|
|
|
1. Destination holds a valid compatible region lease and read-only ghost.
|
|
2. Source freezes the authoritative organism at a tick boundary and durably
|
|
prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
|
|
3. Destination validates and durably stages the exact transferred state.
|
|
4. One transaction checks both epochs and changes the organism ownership mapping
|
|
from source to destination, recording commit and state reference.
|
|
5. Destination activates only after observing commit and its current lease;
|
|
source tombstones its authority. Client reroutes with a fresh grant.
|
|
|
|
Abort is a durable state transition permitted before commit; source resumes only
|
|
after observing that abort. Timeout alone does not authorize a second copy.
|
|
Retries return the recorded transfer outcome. Crash at each arrow must yield one
|
|
owner or a temporarily unavailable organism, never duplication/loss. Epochs and
|
|
the organism ownership mapping must be enforced inside persistence transactions.
|
|
|
|
## Snapshots, migrations and retention
|
|
|
|
Snapshots capture coherent committed state: snapshot.tick <= committed_tick with
|
|
a matching committed replay watermark. They never publish speculative outcomes.
|
|
Compression and I/O run outside the simulation writer on immutable buffers.
|
|
Upload immutable bytes, verify length/digest, then
|
|
publish metadata and replay coverage transactionally under the current fence.
|
|
Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all
|
|
objects reachable from live manifests, clones, backups and retained replay. Never
|
|
delete source replay until a verified recoverable replacement and backup exist.
|
|
|
|
Version biology rules, world format and wire protocol independently. Unknown
|
|
required formats/rules fail closed. Upgrade: quiesce/capture consistent world
|
|
barrier → clone into isolated identity/subjects → migrate → accelerate → compare
|
|
ecological distributions and histories → explicit approval → compatible world
|
|
barrier deployment. Do not run adjacent regions under different physical laws.
|
|
Migration failure reverts code/state before new writes; downgrade after new writes
|
|
requires a designed reverse migration, not a blind database restore.
|
|
|
|
Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive
|
|
host loss. Exact guarantees for acknowledged outcomes require an explicitly
|
|
verified durable failure domain containing the complete causal recovery chain
|
|
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
|
|
permadeath. That admission/DR requirement is a World0 launch gate.
|
|
|
|
## Implemented M5.7 local file publication boundary
|
|
|
|
[ADR-0048](../adr/0048-local-publication.md) and
|
|
[the manifest schema](../../schemas/local-world-manifest.md) implement a single-writer
|
|
development file backend over the M5.6 codecs. Immutable checksum-named payloads
|
|
are installed before alternating manifests; atomic replacement of the inactive
|
|
manifest is the commit point. Recovery validates the selected snapshot and replays
|
|
its optional one-segment journal into a detached store. Pre-publication failures
|
|
retain the previous generation; a post-publication interruption requires reopen
|
|
and recovers the new one. Orphan files are retained. This is not the M7 fenced,
|
|
multi-worker or host-loss durability contract above.
|
|
|
|
## Implemented M5.6 local world codec boundary
|
|
|
|
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
|
|
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
|
|
command high-water, resident cells, debris and exact retry state. Detached replay
|
|
verifies predecessor and final state. This remains an in-memory codec beneath
|
|
M5.7's local file publication adapter.
|