Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m21s
Foundation / checks (network) (push) Successful in 20m51s
Foundation / checks (client) (push) Successful in 22m2s
Foundation / checks (tsan) (push) Successful in 22m59s
Foundation / checks (postgres) (push) Successful in 53m40s
Foundation / checks (checks) (push) Successful in 1h7m19s
Foundation / postgres-tools (push) Successful in 29s
Foundation / server-image (push) Successful in 38s
Foundation / image (push) Successful in 4m31s
309 lines
20 KiB
Markdown
309 lines
20 KiB
Markdown
# M7 region durability contract
|
||
|
||
Design accepted by ADR-0058; no storage/host implementation yet. Locked invariants:
|
||
one committed authority, no lost acknowledged permanent outcome under the declared
|
||
failure model, no fallback empty world, and no database dependency in headless code.
|
||
|
||
## State closure audit
|
||
|
||
| State | Existing evidence | M7 treatment / missing work |
|
||
|---|---|---|
|
||
| M6 environment, two accounts/capacities, terrain and debris | EnvironmentalCommunity envelope v2 and nested codecs | Required exact payload for environmental profile |
|
||
| Config, source/growth/efficiency residuals, lifecycle, tick, last funded interval | Same envelope | Restore all; never infer from tick or defaults |
|
||
| Process retries/audit anchor, resource revisions | Same envelope, 64 records | Preserve; not a durable command deduplication service |
|
||
| Interval scheduler | Stateless M6.7 adapter | No clock/debt to serialize; host queues are separate |
|
||
| Genome | Individual Genome v1 codec | No M6 account binding; unsupported in initial profile |
|
||
| LineageArchive and dead organisms | Separate append-only API/tests, no complete codec | Add bounded export/restore and binding before ancestry recovery is claimed; retain extinct genomes |
|
||
| Population/cohorts | Separate M4 state/codecs | No integration with M6; profile must explicitly add complete ownership before admission |
|
||
| Particle entities/positions/RNG | Independent proof snapshot | Not part of M6; never concatenate and imply unified ownership |
|
||
| Environmental randomness | None in rules 1/2 | Test-driver RNG is not simulation state; future runtime RNG needs versioned streams |
|
||
| Pending host commands | Not implemented | Bounded host queue; uncommitted receipt is provisional; durable decisions stored separately |
|
||
| Durable decisions/history/outbox | Not implemented | Required transactional records with commit references and immutable request bytes |
|
||
| Player accounts, logout, network sessions, cross-region transfers | Not implemented | Unsupported capabilities; initial profile cannot host players |
|
||
| Lease/routing readiness | Not simulation state | Live database authority, never resumed from a saved lease |
|
||
|
||
M7.4 must enumerate any newly attached state and extend its schema and tests before
|
||
admitting it. For a history-capable profile, validate all organism-to-genome bindings,
|
||
parent closure, lifecycle agreement, identity high-waters and archived dead members.
|
||
Both checkpoint and durable history must agree. Empty/unrecognized sections cannot
|
||
stand in for missing history. M7.9's retained-ancestry criterion is not waived by
|
||
the narrower environmental profile. M7.1 completes the audit, not those codecs.
|
||
|
||
## Identity, authority and watermarks
|
||
|
||
ADR-0142 adds opt-in exact population profile 5 (envelope 1, nested cellular
|
||
bundle 9, history profile 5). Raw cellular profile 4 is unchanged. Fenced no-tick
|
||
transitions retain complete biology and source/target digests; normal population
|
||
publication binds recorded intentions and exact membership boundaries. Generic
|
||
publishers cannot bypass dedicated immutable evidence. The host has one owning
|
||
state, materializes before player operations and never treats read-only commit
|
||
resolution as current authority. See [the contract](../../schemas/cellular-population-batch.md)
|
||
and [qualification limits](../engineering/M12_DURABLE_POPULATIONS.md).
|
||
|
||
ADR-0143/schema 45 adds one durable 1..64-tick admission per population head,
|
||
with immutable source/digest/ticks/controller evidence and an atomically maintained
|
||
pending pointer. Settlement and checkpoint publication share one transaction.
|
||
Recovery services that exact pending prefix before fresh work; conversion/profile
|
||
exit cannot skip it. Unknown admission retains the same request and pauses
|
||
readiness. New work requires fully suspended controlled cells, leaving vulnerable
|
||
linger/death on profile 4. Earlier trace semantics remain readable. See the
|
||
[interval contract](../../schemas/ecology/population-interval-v1.md).
|
||
|
||
ADR-0144 adds an exact transfer pair with explicit regional boundary flows and
|
||
nonresident history (owner 7, population envelope 2). ADR-0145/0146 and schema 46
|
||
admit bundle 10 through an explicit same-profile version upgrade. Population trace
|
||
1 retains bundle 9/envelope 1; trace 2 requires bundle 10/envelope 2. No worker
|
||
upgrades automatically. A paired publisher must commit both fenced heads before
|
||
destination activation.
|
||
See [scope and remaining proof](../engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md).
|
||
Its bounded [paired migration trace](../../schemas/cellular-migration-trace-v1.md)
|
||
replays from both source cuts and an independently trusted request, including
|
||
complete cohort tags. Independent-process replay is qualified; no existing
|
||
durable publisher admits that trace or treats it as a transfer receipt.
|
||
|
||
ADR-0147/schema 47 retains that trace as source preparation evidence, without
|
||
admitting either migration result. The source stays authoritative at its exact
|
||
head; a durable pending pointer blocks its new publications/admissions and
|
||
survives lease expiry or replacement. Explicit fenced abort appends its outcome
|
||
and releases the pointer with both biological heads unchanged. A verified
|
||
destination predecessor capsule in the source namespace allows complete pair
|
||
replay without consulting destination live state. The owning host pauses
|
||
readiness, retains unknown requests and never treats timeout/drain as abort.
|
||
See [the freeze contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
|
||
ADR-0148/schema 48 now stages a verified destination ghost and source replay
|
||
capsule under current destination authority and ordered endpoint locks. The
|
||
destination head remains unchanged and is durably paused alongside the source.
|
||
Recovery verifies the complete ghost without activating it; explicit source abort
|
||
releases both pointers atomically. Stage/abort races serialize on source authority.
|
||
See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md).
|
||
ADR-0149/schema 49 now atomically publishes both exact heads and a common immutable
|
||
receipt under both current fences. Failure at either insert rolls back both
|
||
heads and both pending pointers. Commit and abort have one exclusive outcome;
|
||
exact committed retries preserve both original manifests under fresh authority.
|
||
Recovery replays both candidates from local capsules and checks both result
|
||
digests and the complete local payload. A frozen host verifies its changed head
|
||
before readiness. See [the paired commit contract](../../schemas/ecology/cellular-migration-commit-v1.md).
|
||
No atlas, independent-database protocol or automatic ecological movement follows.
|
||
|
||
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
|
||
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
|
||
of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them.
|
||
Simulation tick is local to the fixed versioned region rules. `simulated_tick`
|
||
may lead `committed_tick` only within the current bounded pending batch.
|
||
`commit_sequence` counts publications, not ticks or field commands. Genesis has
|
||
sequence 0, tick 0, no predecessor and an explicit authorized opening payload.
|
||
Only provisioning can create genesis; recovery cannot do so.
|
||
|
||
The database arbitrates acquire/renew/commit/release under the same region lease
|
||
row lock, using database expiry and matching holder/epoch. Commit additionally
|
||
compares expected predecessor sequence, digest and tick. No direct worker table
|
||
writes may bypass this guard. Renewal never revives expired authority. A stale
|
||
process may retain memory but cannot publish a head or originate current output.
|
||
The gateway must have bounded-validity registration, not just an epoch number.
|
||
M7 has no gateway implementation; readiness alone does not prove remote fencing.
|
||
|
||
M7.3 selected lease duration, renewal cadence, local safety margin and store
|
||
deadlines together in ADR-0060 (20 s / 5 s / 2 s / 2 s renew / 5 s commit). The
|
||
latency behind them was measured only on a local socket. The database functions
|
||
and runtime grants in migration v2 implement the guard described here.
|
||
Any uncertain lease stops admission, simulation and final output. A commit that
|
||
already linearized can remain durable after its writer expires; query can return
|
||
its receipt, but does not grant that writer authority to resume.
|
||
|
||
## Batch and I/O boundary
|
||
|
||
Profile limits: one pending batch, 1..64 contiguous ticks, payload <=1MiB for the
|
||
environmental envelope, <=64 durable requests/outcomes and <=64KiB total canonical
|
||
request/result bytes per batch.
|
||
The batch additionally carries <=64KiB of ordered input/report validation trace;
|
||
charge it separately from decision bytes. The full canonical commit request is
|
||
bounded by 160KiB including trace, decisions and metadata. One host backlog holds <=64 future inputs and
|
||
<=64KiB encoded input bytes. Overflow backpressures/rejects before admission; no
|
||
dropping acknowledged inputs. Implementations must charge payload/output buffers,
|
||
one upload buffer and the process budget separately. Compression is initially off.
|
||
No additional queue may silently defeat these bounds.
|
||
|
||
The host drives **one tick at a time** when permanent outcomes are possible. M6.7
|
||
committed-prefix results describe in-memory ticks, not database commits. The host
|
||
cannot blindly execute 64 ticks and discover a death afterward: after each tick,
|
||
detect terminal transitions, include the immutable outcome, seal the current batch
|
||
and pause the whole region before any dependent next tick. The terminal tick's
|
||
ordered phases commit together, including decay. No subsequent tick runs until
|
||
durable confirmation. Normal flush occurs by 64 ticks or an earlier host deadline;
|
||
there is no SQL call from the tick itself. Exceptionally a death can force a
|
||
one-tick batch; this is not a per-tick persistence loop.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant H as Single-owner region host
|
||
participant O as Immutable payload store
|
||
participant D as Fenced database authority
|
||
H->>H: Execute bounded prefix; seal exact end state and outcomes
|
||
H->>O: Put immutable candidate; verify length and SHA-256
|
||
H->>D: Commit expected head + epoch + candidate + decisions/outbox
|
||
D-->>H: Confirmed receipt / rejected / unknown
|
||
alt unknown
|
||
H->>D: Resolve same commit ID and canonical request
|
||
Note over H: Remain paused; never assume rollback
|
||
end
|
||
Note over H: Only confirmed durable receipt permits final outcome and next tick
|
||
```
|
||
|
||
Interface sketches (not compiled stubs): `put_immutable(bytes)->VerifiedObject`,
|
||
`commit_region_batch(lease, expected_head, batch)->Confirmed|Rejected|Unknown`,
|
||
`resolve_commit(world,region,commit_id,canonical_request)->Committed|Absent|Conflict|Unavailable`,
|
||
`load_head(world,region)->CommittedManifest`. Adapter errors carry categories and
|
||
bounded diagnostic context, never credentials. M7.2 chooses concrete C++ types.
|
||
|
||
Candidate state is untrusted to the durable validator: check closure, versions,
|
||
identities, end tick, resource invariants and outcome consistency before publishing.
|
||
Reject changed terminal histories and impossible outcomes. This does not establish
|
||
general anti-cheat against a fully compromised simulation process; database roles
|
||
limit its authority and M7 tests verify the selected transition validators.
|
||
|
||
## Acknowledgments, retries and recovery
|
||
|
||
Received input and simulated input ACKs are explicitly provisional. Only a receipt
|
||
for the database transaction that references durably stored bytes can finalize
|
||
death/reproduction or another permanent decision. Durable request IDs are scoped
|
||
by world/region and are never reused. Retain canonical request and result, compare
|
||
exact bytes on retry; changed contents conflict. A process's 64-entry retry cache
|
||
cannot authorize pruning the durable record. Permanent history/decision IDs remain
|
||
retained for the world lifetime in this initial contract, with archival later.
|
||
|
||
On timeout/disconnect, freeze and query the same commit ID. If committed, reconcile
|
||
exact digest/predecessor/outcomes and adopt the receipt. If absent, retry the same
|
||
request only while still authorized; concurrent delayed originals converge under
|
||
the unique ID and head guard. Absence does not prove a request can never arrive.
|
||
If fenced, abandon mutation and let the current owner recover; do not undo a death
|
||
or ACK speculation. Conflicting bytes or unavailable reconciliation fail closed.
|
||
|
||
Recovery acquires fresh authority, reads the committed head, verifies object length
|
||
and strong digest, checks the explicit version/profile allowlist, restores the full
|
||
payload and reconciles durable outcomes/history to the commit. Missing/corrupt
|
||
objects quarantine the region; never select an older head and silently undo ACKs.
|
||
Unpublished uploads and uncommitted memory are not recovery inputs. No replay tail
|
||
exists in this profile. Required historical binaries/generators remain available.
|
||
Readiness requires successful closure checks and a still-valid lease.
|
||
|
||
No orphan collection in initial M7. Retention/GC must prove reachability across
|
||
heads, commit/decision records, clones and backups before deleting anything. A
|
||
clone gets a new identity/authority domain; references cannot permit source writes.
|
||
Database plus referenced-object backups need a verified common consistency cut.
|
||
Worker/pod loss with surviving durable stores is M7's test model. Host/storage loss
|
||
requires separate M8 replication/restore evidence; old backups cannot silently
|
||
resurrect acknowledged deaths. World0 stays gated.
|
||
|
||
## Crash-point acceptance matrix (required later, not executed by M7.1)
|
||
|
||
| Interruption | Required durable result / recovery | Gate |
|
||
|---|---|---|
|
||
| Before/within a tick | Previous committed head; no permanent ACK | 7.6/7.7 |
|
||
| After simulation, before upload | Previous head; pending outcomes not final | 7.5/7.7 |
|
||
| During upload or failed verification | No manifest; partial/orphan object unreachable | 7.4 |
|
||
| After verified upload, before transaction | Previous head; immutable orphan retained | 7.4/7.7 |
|
||
| Lease expires / new owner before transaction | Old writer rejected; new owner loads committed head | 7.3 |
|
||
| After manifest insert before transaction commit | Entire transaction rolls back, including outcomes/outbox | 7.5 |
|
||
| Commit succeeds, response lost | Resolve same ID returns exact receipt, no second death/birth | 7.5/7.7 |
|
||
| Timeout while original request still running | Retry converges or conflicts safely; never infer rollback | 7.5 |
|
||
| After final ACK, pod killed | Latest head reconstructs ACKed state and history | 7.7 |
|
||
| Outbox delivered twice / broker down | Idempotent projection or queued delivery; no second simulation mutation | 7.5 |
|
||
| Payload missing/corrupt, unknown versions | Quarantine/not ready, no fallback genesis or older world | 7.4/7.7 |
|
||
| Paused stale worker resumes | Renew/commit/output rejected; cannot skip recovery | 7.3/7.6 |
|
||
| Shutdown with pending/unknown commit | Drain boundedly or leave unavailable; next owner reconciles | 7.6/7.7 |
|
||
| Clone/migration fails mid-publication | Source untouched; target not ready, no partial version switch | 7.9 |
|
||
|
||
M7.1 verification is a source/state inventory and contract review. Real database,
|
||
object-store, process-kill and network partition tests are still required.
|
||
|
||
M7.3 executed its rows against real PostgreSQL. After lease expiry, a new owner
|
||
acquires a newer epoch. The old writer's renew, release and head advance are then
|
||
rejected, and the head is unchanged. A paused stale worker resuming past its lease
|
||
is fenced both locally and by the database. A writer queued on the lease lock past
|
||
expiry is also rejected. These are in-process pauses, not killed pods or network
|
||
partitions. The 7.6 parts of those rows remain open.
|
||
|
||
M7.4a (ADR-0061) executed its rows for environmental-v1 against real PostgreSQL and
|
||
a local filesystem object store:
|
||
- A partial upload leaves only an unresolvable temporary name.
|
||
- A failed upload sends no manifest.
|
||
- A verified upload whose transaction is rejected stays an unreferenced orphan,
|
||
and recovery returns the previous head.
|
||
- Missing, corrupt, symlinked or resized payloads quarantine the region.
|
||
- Unsupported builds are refused by both the database and recovery.
|
||
- Recovery never creates genesis or picks an older head.
|
||
|
||
M7.4b (ADR-0062) closes the checkpoint side of the history audit. The lineage
|
||
archive codec and the `environmental-lineage-v1` profile validate:
|
||
- organism-to-genome bindings and parent closure (by archive replay);
|
||
- lifecycle agreement, with deaths recorded in the causing tick;
|
||
- the living set, identity high-waters and retained dead members.
|
||
Recovery restores all of these. Durable history rows (M7.5) and process-kill
|
||
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
|
||
|
||
M7.5 (ADR-0063) executed its rows against real PostgreSQL:
|
||
- A rejected publication (repeated death, partial or reordered decision block,
|
||
gap) leaves the head, commits and decisions unchanged. Publication is a single
|
||
statement; an injected failure between insert and commit is not yet tested.
|
||
- A lost response resolves to the original receipt, and an exact retry adds no
|
||
second death.
|
||
- A client timeout while queued resolved as committed after the lock cleared,
|
||
and was adopted.
|
||
- A terminated orphan resolved as absent and was retried with the same bytes.
|
||
- Changed bytes conflict.
|
||
- Outbox redelivery is idempotent and never touches the simulation.
|
||
Recovery now reconciles durable deaths with the restored lifecycle, so the
|
||
checkpoint and durable history must agree. Kill tests at each boundary remain
|
||
M7.7.
|
||
|
||
M7.6 (ADR-0064) executed its host rows:
|
||
- A paused stale host fences itself when renewals cannot reach the database. It
|
||
reports unconfirmed inputs as never final and re-acquires from the head.
|
||
- A death pauses the region in the same step as its commit.
|
||
- A storage outage keeps the sealed batch waiting with nothing acknowledged.
|
||
- Shutdown drains, commits and releases.
|
||
- A real SIGKILLed `evo-server` left no acknowledged tick uncommitted, and its
|
||
successor became ready only after lease expiry, from the committed head.
|
||
|
||
M7.7 (ADR-0065) executed the matrix with real `evo-server` processes SIGKILLed at
|
||
each boundary: before the transaction (first and second batch), before upload,
|
||
after commit but before acknowledgement, and after acknowledgement.
|
||
- In every case acknowledgements stayed at or below the committed head.
|
||
- Deaths were final only once committed, and verified orphans remained.
|
||
- Each successor verified the head and its tail before readiness.
|
||
A SIGSTOPped owner resumed fenced: it was not ready, refused admission and
|
||
acknowledged nothing new. Corrupt payloads and missing traces quarantined with
|
||
no fallback. Host loss and clone or migration remain M8 and M7.9.
|
||
|
||
M7.9a (ADR-0067) clones a verified committed head into another world and
|
||
database:
|
||
- The payload is re-identified, and the byte round trip is proven.
|
||
- The clone genesis names the source commit and digest, and inherits the source's
|
||
outcomes and ancestry.
|
||
- The source session is read-only.
|
||
- A clone killed before its target transaction left the source untouched and the
|
||
target unprovisioned (it quarantined rather than serve), and a retry completed.
|
||
- The same inputs from the cut reproduced the source trajectory exactly.
|
||
|
||
`audit_region_history` re-verifies every retained link across owners and kills.
|
||
It names the exact link whose evidence is missing, even though head recovery
|
||
still succeeds. Migration, fencing of old writers, and measured recovery and
|
||
batching latency are still M7.9b.
|
||
|
||
M7.9b (ADR-0068) closes the clone/migration row.
|
||
- **Failed migration while serving.** A v8 migration that failed at its last
|
||
statement left v7 exactly, and the lease-holding writer kept publishing. The
|
||
real migration then committed under that same lease.
|
||
- **Old writers.** The upgrade barrier retires the old build and fences every
|
||
holder. After it:
|
||
- the old writer cannot publish, renew or re-acquire, by either acquisition
|
||
form;
|
||
- no link naming the retired build can commit, even under a current lease;
|
||
- a host of the old image quarantines;
|
||
- the new build recovered the whole history and continued.
|
||
- **Rejection.** Rule, format and profile changes are refused; no converter
|
||
exists.
|
||
- **Qualification.** Real SIGKILLs under the production lease were recovered in
|
||
19.5–19.6 s against the proposed 60 s target, with no acknowledged input lost
|
||
(see engineering/DURABILITY_QUALIFICATION.md).
|