Foundation / checks (ecs) (push) Successful in 2m51s
Foundation / checks (deploy) (push) Successful in 5m18s
Foundation / checks (network) (push) Successful in 25m49s
Foundation / checks (client) (push) Successful in 26m34s
Foundation / checks (tsan) (push) Successful in 37m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 47m59s
Foundation / checks (postgres) (push) Canceled after 47m53s
296 lines
19 KiB
Markdown
296 lines
19 KiB
Markdown
# Persistence and region authority
|
|
|
|
[ADR-0167](../adr/0167-retained-plan-checkpoint-upgrade.md) defines portable
|
|
raw-only bundle 12 and the explicit no-tick bundle-11 -> 12 checkpoint candidate.
|
|
Full source restore preserves expressed biology/history while initializing
|
|
retained offspring plans. Reader support is separate from SQL publication;
|
|
there is no population mapping, automatic upgrade or new durable entry point.
|
|
See [the portable upgrade contract](../../schemas/cellular-inheritance-upgrade.md).
|
|
|
|
[ADR-0166](../adr/0166-retained-plan-replay-traces.md) adds portable raw trace 4
|
|
exclusively for the retained-plan owner 8/rules 10. Replay and action matching
|
|
preserve expressed/inherited meanings and return outcomes only after the entire
|
|
candidate succeeds. Existing trace defaults, SQL bundles, population traces and
|
|
result admission remain unchanged. This is not a durable upgrade or live entry.
|
|
|
|
ADR-0158/schema 53 admits the explicit same-profile bundle-10 -> 11 maturation
|
|
upgrade through the existing immutable version journal and lease/head fence.
|
|
Recovery rederives the complete retained transformation; raw trace 3 and population
|
|
batch 2 keep their layouts. Profile changes preserve the tuple. No new genesis,
|
|
automatic upgrade or default change. [ADR-0160](../adr/0160-portable-maturing-wildlife-transfer.md)
|
|
adds portable equal-rules-9 transfer candidates and separately identified trace 2.
|
|
[ADR-0161](../adr/0161-durable-maturing-wildlife-transfer.md)/schema 54 admits exact
|
|
bundle 10/trace 1 or bundle 11/trace 2 pairs through the existing preparation,
|
|
staging, paired receipt, recovery and pending-handoff census workflow. All current
|
|
fences, head bindings, immutable evidence and complete replay remain required.
|
|
Mixed pairs and cross-rule traces fail; no new table or biological rule is added.
|
|
[ADR-0159](../adr/0159-maturing-cellular-census.md) admits its exact
|
|
retained stocks to census projection 1 and the existing verified history and
|
|
registered snapshot readers. Mixed rules-8/rules-9 regional cuts fail the complete
|
|
query. Unavailable evidence cannot become zero life. See
|
|
[the upgrade contract](../../schemas/cellular-maturation-upgrade.md).
|
|
|
|
M12.7c/schema 52 adds [pending migration evidence](../../schemas/ecology/census-pending-migrations-v1.md)
|
|
to registered snapshots. Pending source/destination pointers, namespace, registry
|
|
and heads share one metadata view. Exact captured capsules/ghosts are fully
|
|
replayed after capture, even if settlement commits meanwhile. The source head
|
|
already owns prepared/staged life; ghosts add no census stock. Missing/false
|
|
evidence fails the entire result. This is existing handoff evidence reconciliation,
|
|
not coverage of every viable store or an accepted terminal-extinction record.
|
|
|
|
M12.7b/schema 51 adds [database history domains](../../schemas/ecology/database-history-domain-v1.md).
|
|
Registered census evidence identifies a world by `(history_domain, world_id)`;
|
|
domain, registry and heads share one metadata view. Supported restores fork the
|
|
domain while quarantined in the installation transaction, retaining parent/request
|
|
and exact restored head cuts before verification/readmission. Initial adoption
|
|
and legacy backups cannot reconstruct unknown original context. Arbitrary trusted
|
|
database copies outside that workflow can copy a domain. Cross-domain clone
|
|
provenance, viable-store reconciliation and durable accepted extinction remain open.
|
|
|
|
M12.7a/schema 50 adds [registered-world coverage](../../schemas/ecology/registered-world-census-coverage-v1.md).
|
|
The existing authority registry supplies every required region/frame, without
|
|
activity, expiry, profile or checkpoint filters. The bounded read function and
|
|
head selection share one database snapshot; resources must match the entire
|
|
registered set. Missing/unusable owners and a 65th region fail the query. There
|
|
is no new catalog table or registration write. Schema 51 adds the namespace above;
|
|
other viable stores and accepted global extinction reconciliation remain separate.
|
|
|
|
M12.6d captures declared regional heads in one short read-only repeatable-read
|
|
transaction, closes it, then fully replays each exact immutable cut before
|
|
constructing a scoped census. Atomic migration commits cannot split the selected
|
|
head set when both endpoints are declared. Failed metadata capture rolls back;
|
|
failed cleanup or an exception disconnects the session. Full replay remains
|
|
outside the metadata transaction, with existing per-call evidence locks.
|
|
See [ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md). The caller's
|
|
list is still not complete world coverage or external branch identity.
|
|
|
|
M12.6c's portable [scoped census set](../../schemas/ecology/scoped-cellular-census-set-v1.md)
|
|
combines a bounded declared set of regional observations, including replayed
|
|
history entries. It verifies exact common boundaries, complete shared genome
|
|
records and disjoint resident organism IDs. Callers retain the verified commit
|
|
references separately. Declared coverage is not a durable world catalog, and
|
|
same-tick cuts are not by themselves a reconciled migration watermark.
|
|
|
|
M12.6b derives regional census history from the existing immutable biological
|
|
commit journal. A commit and its exact payload are the canonical event, already
|
|
atomic with the head; no duplicate census-write window or new SQL table is added.
|
|
Explicit historical recovery reuses the full current-head replay checks. Pages
|
|
are bounded to eight consecutive commits and pinned to an exact regional cut;
|
|
unavailable/false evidence fails the entire page. Idle connections prevent a
|
|
writer's uncommitted cuts from appearing durable. See
|
|
[ADR-0151](../adr/0151-committed-cellular-census-history.md). This is regional
|
|
history, not branch/world completeness, a client endpoint or extinction authority.
|
|
|
|
M12.4f defines cellular bundle 10 for owner-7 boundary accounting, with an explicit
|
|
same-profile, no-tick bundle-9 upgrade. M12.4g/schema 46 admits it through the
|
|
existing immutable version-upgrade journal and lease/head fence. Recovery
|
|
rederives the complete source/target transformation. Profile transitions preserve
|
|
either tuple; bundle 9 requires population batch 1 and bundle 10 requires batch 2.
|
|
Portable reader support grants no publication authority and no worker selects an
|
|
upgrade automatically. Atomic paired transfer publication remains required.
|
|
See [ADR-0145](../adr/0145-boundary-accounting-checkpoint-upgrade.md) and
|
|
[ADR-0146](../adr/0146-durable-boundary-accounting-upgrade.md).
|
|
|
|
M12.4h/schema 47 retains source migration preparation, immutable explicit abort
|
|
and a bounded pending freeze across worker replacement. Complete recovery replay
|
|
uses both retained manifests and a verified destination capsule in the source's
|
|
object namespace. Both biological heads stay unchanged; destination inspection
|
|
is optimistic and grants no destination authority. Source publication and new
|
|
interval/player reservations are blocked until explicit settlement. See
|
|
[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md).
|
|
|
|
M12.4i/schema 48 adds destination-authorized staging under ordered endpoint locks
|
|
and the exact retained preparation/heads. A verified source capsule and derived
|
|
destination ghost stay read-only in the destination namespace. Both roles pause
|
|
publication, interval/player admission and host readiness across replacement.
|
|
Explicit source abort releases both pointers atomically; stage/abort races cannot
|
|
activate or strand stock. See [ADR-0148](../adr/0148-durable-cellular-migration-staging.md).
|
|
M12.4j/schema 49 now publishes both results and an immutable common receipt in
|
|
one transaction under both current fences. Abort is exclusive with commit;
|
|
second-insert failure rolls back both heads and pointer settlement. Recovery
|
|
replays both cuts and binds both results to the stage. Frozen hosts reload and
|
|
verify a changed head before readiness, rather than inferring activation from
|
|
pointer absence. See [ADR-0149](../adr/0149-atomic-cellular-migration-publication.md).
|
|
M12.4k qualifies independent loopback TCP/TLS request/receipt loss and either
|
|
owner's actual lease expiry, replacement and healed stale fencing in both
|
|
representations. Independent database observation checks the outcomes. See
|
|
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
|
|
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
|
|
M12.4l extends local qualification with capsule/result directory-sync failures and
|
|
actual object read-permission denial after commit. Current-payload I/O failure
|
|
pauses complete recovery for retry; unavailable retained counterpart evidence
|
|
quarantines. Replacement fully verifies the original pair under newer authority. See
|
|
[storage scope](../engineering/M12_MIGRATION_STORAGE_FAULTS.md). Remote storage-host,
|
|
physical media and orchestrator failures remain separate evidence.
|
|
|
|
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
|
|
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
|
|
recovery from semantic events or an unimplemented input journal. Candidate uploads
|
|
become published committed snapshots only in the fenced head/outcome transaction.
|
|
The broader replay and cross-region design below remains a production direction.
|
|
The state audit explicitly identifies missing lineage/population integration;
|
|
environmental fixture recovery cannot satisfy those acceptance criteria alone.
|
|
|
|
Status: proposed production contract; source-of-truth PostgreSQL and disposable
|
|
workers follow locked requirements. M0 implements only an in-memory local proof
|
|
snapshot. It has no durable journal, database, lease or crash-consistent file writer.
|
|
|
|
## Data ownership and recovery
|
|
|
|
| Data | Proposed owner and retention |
|
|
|---|---|
|
|
| Accounts/provider subjects, worlds, regions | Versioned PostgreSQL relational metadata |
|
|
| Immutable genomes and ancestry/history | PostgreSQL metadata/events; never discard extinct genomes |
|
|
| Active simulation | Region worker memory, recoverable from committed checkpoint/replay |
|
|
| Snapshot/chunk payloads | Immutable S3-compatible objects, strong content digest |
|
|
| Snapshot manifests and committed replay ranges | PostgreSQL transactions; references verified objects |
|
|
| Notifications/projections | Transactional outbox; optional JetStream, idempotent consumers |
|
|
| Ephemeral routing/cache | In-process first; Redis only with measured need, never sole durable state |
|
|
|
|
Proposed durable keys: world_id, region_id, epoch, simulation_version,
|
|
world_format_version, codec_version, generator_version, snapshot_id, tick,
|
|
covered_sequence, content_digest. Organism, genome and runtime entity identities
|
|
remain distinct. All cross-world keys include world identity. Migration SQL is
|
|
versioned when introduced at M7; do not create unused tables in M0.
|
|
|
|
Recovery is not semantic events alone: deaths/births do not reconstruct intervening
|
|
movement, external inputs or random decisions. Persist contiguous deterministic
|
|
replay batches containing ordered admitted commands, exogenous inputs, tick range,
|
|
versions and resulting state hash. Preserve required RNG/state/checkpoint data.
|
|
Use strict append-only domain history for irreversible ancestry/lifecycle outcomes;
|
|
use snapshots plus replay for mutable region state, not event sourcing every scalar.
|
|
Batch transactions; no PostgreSQL write per simulation tick.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant W as Worker
|
|
participant O as Object store
|
|
participant D as PostgreSQL
|
|
participant C as Client
|
|
W->>W: Simulate ordered ticks into pending batch
|
|
W->>O: Upload immutable replay payload if externalized / verify
|
|
W->>D: Fenced transaction: replay range + domain outcomes + outbox
|
|
D-->>W: Durable commit or ambiguous failure
|
|
W-->>C: Final outcome only after confirmed/reconciled commit
|
|
W->>O: Upload checkpoint bytes / verify digest
|
|
W->>D: Fenced publish manifest with covered sequence
|
|
Note over W,D: On recovery: manifest + complete replay range, verify hash before serving
|
|
```
|
|
|
|
Input receipt, simulated-input acknowledgment and durable gameplay acknowledgment
|
|
are distinct. Never present an uncommitted permanent death or reproductive success
|
|
as final. Acknowledged death/reproduction must reconstruct after worker loss.
|
|
Ordinary speculative movement may roll back within an explicit bounded replay
|
|
window; that window and batching latency are M7 experiments. Database failure
|
|
withholds irreversible outcomes, limits pending memory and pauses before budgets
|
|
or lease safety expire. A disconnected commit may have succeeded: reconcile its
|
|
idempotency key before retrying.
|
|
|
|
Decision IDs store canonical request digests: retry with changed contents is
|
|
rejected. Track simulated_tick separately from committed_tick. Pending irreversible
|
|
decisions cannot enable dependent actions; pending death cannot bypass logout.
|
|
For initial M7, pause the entire region at an irreversible-outcome boundary until
|
|
commit. A future nonblocking alternative must prove deterministic isolation of
|
|
every affected interaction; freezing only a dying organism is insufficient.
|
|
CommitRegionBatch requires the current lease and expected predecessor head.
|
|
Segments identify contiguous ticks, predecessor digest, exact rules/binary/RNG
|
|
versions, resolved exchanges and required nondeterministic outputs. If initial M7
|
|
cannot capture complete replay, evaluate full checkpoint batches explicitly rather
|
|
than claiming incomplete event replay is recovery.
|
|
|
|
## Fenced region lease
|
|
|
|
```mermaid
|
|
stateDiagram-v2
|
|
Unassigned --> Acquiring: director assignment hint
|
|
Acquiring --> Recovering: transactional epoch increment / lease
|
|
Recovering --> Active: checkpoint + replay + validation complete
|
|
Active --> Draining: handoff or shutdown
|
|
Active --> Fenced: renewal failure / deadline safety margin
|
|
Draining --> Released: final committed checkpoint / release
|
|
Fenced --> Acquiring: new valid ownership acquisition
|
|
```
|
|
|
|
The director suggests assignment; PostgreSQL arbitrates. Proposed row:
|
|
`RegionLease(world, region, holder, epoch, expires_at)`. Acquisition locks the row,
|
|
requires release/expiry using database time, increments a never-reused epoch and
|
|
records holder/deadline. Renewal compares holder+epoch and unexpired authority.
|
|
Every durable region mutation transaction locks/checks that same lease row and
|
|
epoch; DB privileges prevent workers bypassing the guarded interface. The lock
|
|
serializes ownership acquisition against commits. A process cannot become
|
|
authoritative merely because it has a Pod identity or receives a NATS message.
|
|
|
|
Workers fence themselves before their conservative local safety deadline; a
|
|
pause/resume cannot renew a stale epoch. Incoming commands, routing grants,
|
|
snapshots and transfer operations carry epoch/revision. Gateways/clients reject
|
|
stale-authority output under the routing/session protocol. Two processes may
|
|
hold stale routing views too: require bounded-validity authority registration
|
|
and fail-closed forwarding on uncertainty. Begin with one admission gateway per
|
|
region; partition/clock tests gate multiple gateways. Carrying an epoch alone
|
|
does not fence a gateway unaware of a newer epoch. Two processes may
|
|
retain memory, but cannot both commit authority. Recovery readiness requires
|
|
validated replay and a valid current lease; liveness is not proof of ownership.
|
|
Test long pauses, stale renewals, overlapping acquisition and delayed packets.
|
|
|
|
## Entity handoff across regions
|
|
|
|
1. Destination holds a valid compatible region lease and read-only ghost.
|
|
2. Source freezes the authoritative organism at a tick boundary and durably
|
|
prepares final state under a transfer ID. Ghost cannot spend/act/reproduce.
|
|
3. Destination validates and durably stages the exact transferred state.
|
|
4. One transaction checks both epochs and changes the organism ownership mapping
|
|
from source to destination, recording commit and state reference.
|
|
5. Destination activates only after observing commit and its current lease;
|
|
source tombstones its authority. Client reroutes with a fresh grant.
|
|
|
|
Abort is a durable state transition permitted before commit; source resumes only
|
|
after observing that abort. Timeout alone does not authorize a second copy.
|
|
Retries return the recorded transfer outcome. Crash at each arrow must yield one
|
|
owner or a temporarily unavailable organism, never duplication/loss. Epochs and
|
|
the organism ownership mapping must be enforced inside persistence transactions.
|
|
|
|
## Snapshots, migrations and retention
|
|
|
|
Snapshots capture coherent committed state: snapshot.tick <= committed_tick with
|
|
a matching committed replay watermark. They never publish speculative outcomes.
|
|
Compression and I/O run outside the simulation writer on immutable buffers.
|
|
Upload immutable bytes, verify length/digest, then
|
|
publish metadata and replay coverage transactionally under the current fence.
|
|
Failed publication leaves an orphan, not a checkpoint. Compaction/GC retains all
|
|
objects reachable from live manifests, clones, backups and retained replay. Never
|
|
delete source replay until a verified recoverable replacement and backup exist.
|
|
|
|
Version biology rules, world format and wire protocol independently. Unknown
|
|
required formats/rules fail closed. Upgrade: quiesce/capture consistent world
|
|
barrier → clone into isolated identity/subjects → migrate → accelerate → compare
|
|
ecological distributions and histories → explicit approval → compatible world
|
|
barrier deployment. Do not run adjacent regions under different physical laws.
|
|
Migration failure reverts code/state before new writes; downgrade after new writes
|
|
requires a designed reverse migration, not a blind database restore.
|
|
|
|
Backup/DR is detailed in OPERATIONS. A replica on the same host does not survive
|
|
host loss. Exact guarantees for acknowledged outcomes require an explicitly
|
|
verified durable failure domain containing the complete causal recovery chain
|
|
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
|
|
permadeath. That admission/DR requirement is a World0 launch gate.
|
|
|
|
## Implemented M5.7 local file publication boundary
|
|
|
|
[ADR-0048](../adr/0048-local-publication.md) and
|
|
[the manifest schema](../../schemas/local-world-manifest.md) implement a single-writer
|
|
development file backend over the M5.6 codecs. Immutable checksum-named payloads
|
|
are installed before alternating manifests; atomic replacement of the inactive
|
|
manifest is the commit point. Recovery validates the selected snapshot and replays
|
|
its optional one-segment journal into a detached store. Pre-publication failures
|
|
retain the previous generation; a post-publication interruption requires reopen
|
|
and recovers the new one. Orphan files are retained. This is not the M7 fenced,
|
|
multi-worker or host-loss durability contract above.
|
|
|
|
## Implemented M5.6 local world codec boundary
|
|
|
|
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
|
|
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
|
|
command high-water, resident cells, debris and exact retry state. Detached replay
|
|
verifies predecessor and final state. This remains an in-memory codec beneath
|
|
M5.7's local file publication adapter.
|