Files
dingo 0f071f3f8b
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m21s
Foundation / checks (network) (push) Successful in 20m51s
Foundation / checks (client) (push) Successful in 22m2s
Foundation / checks (tsan) (push) Successful in 22m59s
Foundation / checks (postgres) (push) Successful in 53m40s
Foundation / checks (checks) (push) Successful in 1h7m19s
Foundation / postgres-tools (push) Successful in 29s
Foundation / server-image (push) Successful in 38s
Foundation / image (push) Successful in 4m31s
M12.4: atomically publish and recover biological migration pairs
2026-10-18 10:02:55 -04:00

309 lines
20 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# M7 region durability contract
Design accepted by ADR-0058; no storage/host implementation yet. Locked invariants:
one committed authority, no lost acknowledged permanent outcome under the declared
failure model, no fallback empty world, and no database dependency in headless code.
## State closure audit
| State | Existing evidence | M7 treatment / missing work |
|---|---|---|
| M6 environment, two accounts/capacities, terrain and debris | EnvironmentalCommunity envelope v2 and nested codecs | Required exact payload for environmental profile |
| Config, source/growth/efficiency residuals, lifecycle, tick, last funded interval | Same envelope | Restore all; never infer from tick or defaults |
| Process retries/audit anchor, resource revisions | Same envelope, 64 records | Preserve; not a durable command deduplication service |
| Interval scheduler | Stateless M6.7 adapter | No clock/debt to serialize; host queues are separate |
| Genome | Individual Genome v1 codec | No M6 account binding; unsupported in initial profile |
| LineageArchive and dead organisms | Separate append-only API/tests, no complete codec | Add bounded export/restore and binding before ancestry recovery is claimed; retain extinct genomes |
| Population/cohorts | Separate M4 state/codecs | No integration with M6; profile must explicitly add complete ownership before admission |
| Particle entities/positions/RNG | Independent proof snapshot | Not part of M6; never concatenate and imply unified ownership |
| Environmental randomness | None in rules 1/2 | Test-driver RNG is not simulation state; future runtime RNG needs versioned streams |
| Pending host commands | Not implemented | Bounded host queue; uncommitted receipt is provisional; durable decisions stored separately |
| Durable decisions/history/outbox | Not implemented | Required transactional records with commit references and immutable request bytes |
| Player accounts, logout, network sessions, cross-region transfers | Not implemented | Unsupported capabilities; initial profile cannot host players |
| Lease/routing readiness | Not simulation state | Live database authority, never resumed from a saved lease |
M7.4 must enumerate any newly attached state and extend its schema and tests before
admitting it. For a history-capable profile, validate all organism-to-genome bindings,
parent closure, lifecycle agreement, identity high-waters and archived dead members.
Both checkpoint and durable history must agree. Empty/unrecognized sections cannot
stand in for missing history. M7.9's retained-ancestry criterion is not waived by
the narrower environmental profile. M7.1 completes the audit, not those codecs.
## Identity, authority and watermarks
ADR-0142 adds opt-in exact population profile 5 (envelope 1, nested cellular
bundle 9, history profile 5). Raw cellular profile 4 is unchanged. Fenced no-tick
transitions retain complete biology and source/target digests; normal population
publication binds recorded intentions and exact membership boundaries. Generic
publishers cannot bypass dedicated immutable evidence. The host has one owning
state, materializes before player operations and never treats read-only commit
resolution as current authority. See [the contract](../../schemas/cellular-population-batch.md)
and [qualification limits](../engineering/M12_DURABLE_POPULATIONS.md).
ADR-0143/schema 45 adds one durable 1..64-tick admission per population head,
with immutable source/digest/ticks/controller evidence and an atomically maintained
pending pointer. Settlement and checkpoint publication share one transaction.
Recovery services that exact pending prefix before fresh work; conversion/profile
exit cannot skip it. Unknown admission retains the same request and pauses
readiness. New work requires fully suspended controlled cells, leaving vulnerable
linger/death on profile 4. Earlier trace semantics remain readable. See the
[interval contract](../../schemas/ecology/population-interval-v1.md).
ADR-0144 adds an exact transfer pair with explicit regional boundary flows and
nonresident history (owner 7, population envelope 2). ADR-0145/0146 and schema 46
admit bundle 10 through an explicit same-profile version upgrade. Population trace
1 retains bundle 9/envelope 1; trace 2 requires bundle 10/envelope 2. No worker
upgrades automatically. A paired publisher must commit both fenced heads before
destination activation.
See [scope and remaining proof](../engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md).
Its bounded [paired migration trace](../../schemas/cellular-migration-trace-v1.md)
replays from both source cuts and an independently trusted request, including
complete cohort tags. Independent-process replay is qualified; no existing
durable publisher admits that trace or treats it as a transfer receipt.
ADR-0147/schema 47 retains that trace as source preparation evidence, without
admitting either migration result. The source stays authoritative at its exact
head; a durable pending pointer blocks its new publications/admissions and
survives lease expiry or replacement. Explicit fenced abort appends its outcome
and releases the pointer with both biological heads unchanged. A verified
destination predecessor capsule in the source namespace allows complete pair
replay without consulting destination live state. The owning host pauses
readiness, retains unknown requests and never treats timeout/drain as abort.
See [the freeze contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
ADR-0148/schema 48 now stages a verified destination ghost and source replay
capsule under current destination authority and ordered endpoint locks. The
destination head remains unchanged and is durably paused alongside the source.
Recovery verifies the complete ghost without activating it; explicit source abort
releases both pointers atomically. Stage/abort races serialize on source authority.
See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md).
ADR-0149/schema 49 now atomically publishes both exact heads and a common immutable
receipt under both current fences. Failure at either insert rolls back both
heads and both pending pointers. Commit and abort have one exclusive outcome;
exact committed retries preserve both original manifests under fresh authority.
Recovery replays both candidates from local capsules and checks both result
digests and the complete local payload. A frozen host verifies its changed head
before readiness. See [the paired commit contract](../../schemas/ecology/cellular-migration-commit-v1.md).
No atlas, independent-database protocol or automatic ecological movement follows.
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them.
Simulation tick is local to the fixed versioned region rules. `simulated_tick`
may lead `committed_tick` only within the current bounded pending batch.
`commit_sequence` counts publications, not ticks or field commands. Genesis has
sequence 0, tick 0, no predecessor and an explicit authorized opening payload.
Only provisioning can create genesis; recovery cannot do so.
The database arbitrates acquire/renew/commit/release under the same region lease
row lock, using database expiry and matching holder/epoch. Commit additionally
compares expected predecessor sequence, digest and tick. No direct worker table
writes may bypass this guard. Renewal never revives expired authority. A stale
process may retain memory but cannot publish a head or originate current output.
The gateway must have bounded-validity registration, not just an epoch number.
M7 has no gateway implementation; readiness alone does not prove remote fencing.
M7.3 selected lease duration, renewal cadence, local safety margin and store
deadlines together in ADR-0060 (20 s / 5 s / 2 s / 2 s renew / 5 s commit). The
latency behind them was measured only on a local socket. The database functions
and runtime grants in migration v2 implement the guard described here.
Any uncertain lease stops admission, simulation and final output. A commit that
already linearized can remain durable after its writer expires; query can return
its receipt, but does not grant that writer authority to resume.
## Batch and I/O boundary
Profile limits: one pending batch, 1..64 contiguous ticks, payload <=1MiB for the
environmental envelope, <=64 durable requests/outcomes and <=64KiB total canonical
request/result bytes per batch.
The batch additionally carries <=64KiB of ordered input/report validation trace;
charge it separately from decision bytes. The full canonical commit request is
bounded by 160KiB including trace, decisions and metadata. One host backlog holds <=64 future inputs and
<=64KiB encoded input bytes. Overflow backpressures/rejects before admission; no
dropping acknowledged inputs. Implementations must charge payload/output buffers,
one upload buffer and the process budget separately. Compression is initially off.
No additional queue may silently defeat these bounds.
The host drives **one tick at a time** when permanent outcomes are possible. M6.7
committed-prefix results describe in-memory ticks, not database commits. The host
cannot blindly execute 64 ticks and discover a death afterward: after each tick,
detect terminal transitions, include the immutable outcome, seal the current batch
and pause the whole region before any dependent next tick. The terminal tick's
ordered phases commit together, including decay. No subsequent tick runs until
durable confirmation. Normal flush occurs by 64 ticks or an earlier host deadline;
there is no SQL call from the tick itself. Exceptionally a death can force a
one-tick batch; this is not a per-tick persistence loop.
```mermaid
sequenceDiagram
participant H as Single-owner region host
participant O as Immutable payload store
participant D as Fenced database authority
H->>H: Execute bounded prefix; seal exact end state and outcomes
H->>O: Put immutable candidate; verify length and SHA-256
H->>D: Commit expected head + epoch + candidate + decisions/outbox
D-->>H: Confirmed receipt / rejected / unknown
alt unknown
H->>D: Resolve same commit ID and canonical request
Note over H: Remain paused; never assume rollback
end
Note over H: Only confirmed durable receipt permits final outcome and next tick
```
Interface sketches (not compiled stubs): `put_immutable(bytes)->VerifiedObject`,
`commit_region_batch(lease, expected_head, batch)->Confirmed|Rejected|Unknown`,
`resolve_commit(world,region,commit_id,canonical_request)->Committed|Absent|Conflict|Unavailable`,
`load_head(world,region)->CommittedManifest`. Adapter errors carry categories and
bounded diagnostic context, never credentials. M7.2 chooses concrete C++ types.
Candidate state is untrusted to the durable validator: check closure, versions,
identities, end tick, resource invariants and outcome consistency before publishing.
Reject changed terminal histories and impossible outcomes. This does not establish
general anti-cheat against a fully compromised simulation process; database roles
limit its authority and M7 tests verify the selected transition validators.
## Acknowledgments, retries and recovery
Received input and simulated input ACKs are explicitly provisional. Only a receipt
for the database transaction that references durably stored bytes can finalize
death/reproduction or another permanent decision. Durable request IDs are scoped
by world/region and are never reused. Retain canonical request and result, compare
exact bytes on retry; changed contents conflict. A process's 64-entry retry cache
cannot authorize pruning the durable record. Permanent history/decision IDs remain
retained for the world lifetime in this initial contract, with archival later.
On timeout/disconnect, freeze and query the same commit ID. If committed, reconcile
exact digest/predecessor/outcomes and adopt the receipt. If absent, retry the same
request only while still authorized; concurrent delayed originals converge under
the unique ID and head guard. Absence does not prove a request can never arrive.
If fenced, abandon mutation and let the current owner recover; do not undo a death
or ACK speculation. Conflicting bytes or unavailable reconciliation fail closed.
Recovery acquires fresh authority, reads the committed head, verifies object length
and strong digest, checks the explicit version/profile allowlist, restores the full
payload and reconciles durable outcomes/history to the commit. Missing/corrupt
objects quarantine the region; never select an older head and silently undo ACKs.
Unpublished uploads and uncommitted memory are not recovery inputs. No replay tail
exists in this profile. Required historical binaries/generators remain available.
Readiness requires successful closure checks and a still-valid lease.
No orphan collection in initial M7. Retention/GC must prove reachability across
heads, commit/decision records, clones and backups before deleting anything. A
clone gets a new identity/authority domain; references cannot permit source writes.
Database plus referenced-object backups need a verified common consistency cut.
Worker/pod loss with surviving durable stores is M7's test model. Host/storage loss
requires separate M8 replication/restore evidence; old backups cannot silently
resurrect acknowledged deaths. World0 stays gated.
## Crash-point acceptance matrix (required later, not executed by M7.1)
| Interruption | Required durable result / recovery | Gate |
|---|---|---|
| Before/within a tick | Previous committed head; no permanent ACK | 7.6/7.7 |
| After simulation, before upload | Previous head; pending outcomes not final | 7.5/7.7 |
| During upload or failed verification | No manifest; partial/orphan object unreachable | 7.4 |
| After verified upload, before transaction | Previous head; immutable orphan retained | 7.4/7.7 |
| Lease expires / new owner before transaction | Old writer rejected; new owner loads committed head | 7.3 |
| After manifest insert before transaction commit | Entire transaction rolls back, including outcomes/outbox | 7.5 |
| Commit succeeds, response lost | Resolve same ID returns exact receipt, no second death/birth | 7.5/7.7 |
| Timeout while original request still running | Retry converges or conflicts safely; never infer rollback | 7.5 |
| After final ACK, pod killed | Latest head reconstructs ACKed state and history | 7.7 |
| Outbox delivered twice / broker down | Idempotent projection or queued delivery; no second simulation mutation | 7.5 |
| Payload missing/corrupt, unknown versions | Quarantine/not ready, no fallback genesis or older world | 7.4/7.7 |
| Paused stale worker resumes | Renew/commit/output rejected; cannot skip recovery | 7.3/7.6 |
| Shutdown with pending/unknown commit | Drain boundedly or leave unavailable; next owner reconciles | 7.6/7.7 |
| Clone/migration fails mid-publication | Source untouched; target not ready, no partial version switch | 7.9 |
M7.1 verification is a source/state inventory and contract review. Real database,
object-store, process-kill and network partition tests are still required.
M7.3 executed its rows against real PostgreSQL. After lease expiry, a new owner
acquires a newer epoch. The old writer's renew, release and head advance are then
rejected, and the head is unchanged. A paused stale worker resuming past its lease
is fenced both locally and by the database. A writer queued on the lease lock past
expiry is also rejected. These are in-process pauses, not killed pods or network
partitions. The 7.6 parts of those rows remain open.
M7.4a (ADR-0061) executed its rows for environmental-v1 against real PostgreSQL and
a local filesystem object store:
- A partial upload leaves only an unresolvable temporary name.
- A failed upload sends no manifest.
- A verified upload whose transaction is rejected stays an unreferenced orphan,
and recovery returns the previous head.
- Missing, corrupt, symlinked or resized payloads quarantine the region.
- Unsupported builds are refused by both the database and recovery.
- Recovery never creates genesis or picks an older head.
M7.4b (ADR-0062) closes the checkpoint side of the history audit. The lineage
archive codec and the `environmental-lineage-v1` profile validate:
- organism-to-genome bindings and parent closure (by archive replay);
- lifecycle agreement, with deaths recorded in the causing tick;
- the living set, identity high-waters and retained dead members.
Recovery restores all of these. Durable history rows (M7.5) and process-kill
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
M7.5 (ADR-0063) executed its rows against real PostgreSQL:
- A rejected publication (repeated death, partial or reordered decision block,
gap) leaves the head, commits and decisions unchanged. Publication is a single
statement; an injected failure between insert and commit is not yet tested.
- A lost response resolves to the original receipt, and an exact retry adds no
second death.
- A client timeout while queued resolved as committed after the lock cleared,
and was adopted.
- A terminated orphan resolved as absent and was retried with the same bytes.
- Changed bytes conflict.
- Outbox redelivery is idempotent and never touches the simulation.
Recovery now reconciles durable deaths with the restored lifecycle, so the
checkpoint and durable history must agree. Kill tests at each boundary remain
M7.7.
M7.6 (ADR-0064) executed its host rows:
- A paused stale host fences itself when renewals cannot reach the database. It
reports unconfirmed inputs as never final and re-acquires from the head.
- A death pauses the region in the same step as its commit.
- A storage outage keeps the sealed batch waiting with nothing acknowledged.
- Shutdown drains, commits and releases.
- A real SIGKILLed `evo-server` left no acknowledged tick uncommitted, and its
successor became ready only after lease expiry, from the committed head.
M7.7 (ADR-0065) executed the matrix with real `evo-server` processes SIGKILLed at
each boundary: before the transaction (first and second batch), before upload,
after commit but before acknowledgement, and after acknowledgement.
- In every case acknowledgements stayed at or below the committed head.
- Deaths were final only once committed, and verified orphans remained.
- Each successor verified the head and its tail before readiness.
A SIGSTOPped owner resumed fenced: it was not ready, refused admission and
acknowledged nothing new. Corrupt payloads and missing traces quarantined with
no fallback. Host loss and clone or migration remain M8 and M7.9.
M7.9a (ADR-0067) clones a verified committed head into another world and
database:
- The payload is re-identified, and the byte round trip is proven.
- The clone genesis names the source commit and digest, and inherits the source's
outcomes and ancestry.
- The source session is read-only.
- A clone killed before its target transaction left the source untouched and the
target unprovisioned (it quarantined rather than serve), and a retry completed.
- The same inputs from the cut reproduced the source trajectory exactly.
`audit_region_history` re-verifies every retained link across owners and kills.
It names the exact link whose evidence is missing, even though head recovery
still succeeds. Migration, fencing of old writers, and measured recovery and
batching latency are still M7.9b.
M7.9b (ADR-0068) closes the clone/migration row.
- **Failed migration while serving.** A v8 migration that failed at its last
statement left v7 exactly, and the lease-holding writer kept publishing. The
real migration then committed under that same lease.
- **Old writers.** The upgrade barrier retires the old build and fences every
holder. After it:
- the old writer cannot publish, renew or re-acquire, by either acquisition
form;
- no link naming the retired build can commit, even under a current lease;
- a host of the old image quarantines;
- the new build recovered the whole history and continued.
- **Rejection.** Rule, format and profile changes are refused; no converter
exists.
- **Qualification.** Real SIGKILLs under the production lease were recovered in
19.5–19.6 s against the proposed 60 s target, with no acknowledged input lost
(see engineering/DURABILITY_QUALIFICATION.md).