Files
dingo 0f071f3f8b
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m21s
Foundation / checks (network) (push) Successful in 20m51s
Foundation / checks (client) (push) Successful in 22m2s
Foundation / checks (tsan) (push) Successful in 22m59s
Foundation / checks (postgres) (push) Successful in 53m40s
Foundation / checks (checks) (push) Successful in 1h7m19s
Foundation / postgres-tools (push) Successful in 29s
Foundation / server-image (push) Successful in 38s
Foundation / image (push) Successful in 4m31s
M12.4: atomically publish and recover biological migration pairs
2026-10-18 10:02:55 -04:00

20 KiB
Raw Permalink Blame History

M7 region durability contract

Design accepted by ADR-0058; no storage/host implementation yet. Locked invariants: one committed authority, no lost acknowledged permanent outcome under the declared failure model, no fallback empty world, and no database dependency in headless code.

State closure audit

State Existing evidence M7 treatment / missing work
M6 environment, two accounts/capacities, terrain and debris EnvironmentalCommunity envelope v2 and nested codecs Required exact payload for environmental profile
Config, source/growth/efficiency residuals, lifecycle, tick, last funded interval Same envelope Restore all; never infer from tick or defaults
Process retries/audit anchor, resource revisions Same envelope, 64 records Preserve; not a durable command deduplication service
Interval scheduler Stateless M6.7 adapter No clock/debt to serialize; host queues are separate
Genome Individual Genome v1 codec No M6 account binding; unsupported in initial profile
LineageArchive and dead organisms Separate append-only API/tests, no complete codec Add bounded export/restore and binding before ancestry recovery is claimed; retain extinct genomes
Population/cohorts Separate M4 state/codecs No integration with M6; profile must explicitly add complete ownership before admission
Particle entities/positions/RNG Independent proof snapshot Not part of M6; never concatenate and imply unified ownership
Environmental randomness None in rules 1/2 Test-driver RNG is not simulation state; future runtime RNG needs versioned streams
Pending host commands Not implemented Bounded host queue; uncommitted receipt is provisional; durable decisions stored separately
Durable decisions/history/outbox Not implemented Required transactional records with commit references and immutable request bytes
Player accounts, logout, network sessions, cross-region transfers Not implemented Unsupported capabilities; initial profile cannot host players
Lease/routing readiness Not simulation state Live database authority, never resumed from a saved lease

M7.4 must enumerate any newly attached state and extend its schema and tests before admitting it. For a history-capable profile, validate all organism-to-genome bindings, parent closure, lifecycle agreement, identity high-waters and archived dead members. Both checkpoint and durable history must agree. Empty/unrecognized sections cannot stand in for missing history. M7.9's retained-ancestry criterion is not waived by the narrower environmental profile. M7.1 completes the audit, not those codecs.

Identity, authority and watermarks

ADR-0142 adds opt-in exact population profile 5 (envelope 1, nested cellular bundle 9, history profile 5). Raw cellular profile 4 is unchanged. Fenced no-tick transitions retain complete biology and source/target digests; normal population publication binds recorded intentions and exact membership boundaries. Generic publishers cannot bypass dedicated immutable evidence. The host has one owning state, materializes before player operations and never treats read-only commit resolution as current authority. See the contract and qualification limits.

ADR-0143/schema 45 adds one durable 1..64-tick admission per population head, with immutable source/digest/ticks/controller evidence and an atomically maintained pending pointer. Settlement and checkpoint publication share one transaction. Recovery services that exact pending prefix before fresh work; conversion/profile exit cannot skip it. Unknown admission retains the same request and pauses readiness. New work requires fully suspended controlled cells, leaving vulnerable linger/death on profile 4. Earlier trace semantics remain readable. See the interval contract.

ADR-0144 adds an exact transfer pair with explicit regional boundary flows and nonresident history (owner 7, population envelope 2). ADR-0145/0146 and schema 46 admit bundle 10 through an explicit same-profile version upgrade. Population trace 1 retains bundle 9/envelope 1; trace 2 requires bundle 10/envelope 2. No worker upgrades automatically. A paired publisher must commit both fenced heads before destination activation. See scope and remaining proof. Its bounded paired migration trace replays from both source cuts and an independently trusted request, including complete cohort tags. Independent-process replay is qualified; no existing durable publisher admits that trace or treats it as a transfer receipt.

ADR-0147/schema 47 retains that trace as source preparation evidence, without admitting either migration result. The source stays authoritative at its exact head; a durable pending pointer blocks its new publications/admissions and survives lease expiry or replacement. Explicit fenced abort appends its outcome and releases the pointer with both biological heads unchanged. A verified destination predecessor capsule in the source namespace allows complete pair replay without consulting destination live state. The owning host pauses readiness, retains unknown requests and never treats timeout/drain as abort. See the freeze contract. ADR-0148/schema 48 now stages a verified destination ghost and source replay capsule under current destination authority and ordered endpoint locks. The destination head remains unchanged and is durably paused alongside the source. Recovery verifies the complete ghost without activating it; explicit source abort releases both pointers atomically. Stage/abort races serialize on source authority. See the staging contract. ADR-0149/schema 49 now atomically publishes both exact heads and a common immutable receipt under both current fences. Failure at either insert rolls back both heads and both pending pointers. Commit and abort have one exclusive outcome; exact committed retries preserve both original manifests under fresh authority. Recovery replays both candidates from local capsules and checks both result digests and the complete local payload. A frozen host verifies its changed head before readiness. See the paired commit contract. No atlas, independent-database protocol or automatic ecological movement follows.

World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them. Simulation tick is local to the fixed versioned region rules. simulated_tick may lead committed_tick only within the current bounded pending batch. commit_sequence counts publications, not ticks or field commands. Genesis has sequence 0, tick 0, no predecessor and an explicit authorized opening payload. Only provisioning can create genesis; recovery cannot do so.

The database arbitrates acquire/renew/commit/release under the same region lease row lock, using database expiry and matching holder/epoch. Commit additionally compares expected predecessor sequence, digest and tick. No direct worker table writes may bypass this guard. Renewal never revives expired authority. A stale process may retain memory but cannot publish a head or originate current output. The gateway must have bounded-validity registration, not just an epoch number. M7 has no gateway implementation; readiness alone does not prove remote fencing.

M7.3 selected lease duration, renewal cadence, local safety margin and store deadlines together in ADR-0060 (20 s / 5 s / 2 s / 2 s renew / 5 s commit). The latency behind them was measured only on a local socket. The database functions and runtime grants in migration v2 implement the guard described here. Any uncertain lease stops admission, simulation and final output. A commit that already linearized can remain durable after its writer expires; query can return its receipt, but does not grant that writer authority to resume.

Batch and I/O boundary

Profile limits: one pending batch, 1..64 contiguous ticks, payload <=1MiB for the environmental envelope, <=64 durable requests/outcomes and <=64KiB total canonical request/result bytes per batch. The batch additionally carries <=64KiB of ordered input/report validation trace; charge it separately from decision bytes. The full canonical commit request is bounded by 160KiB including trace, decisions and metadata. One host backlog holds <=64 future inputs and <=64KiB encoded input bytes. Overflow backpressures/rejects before admission; no dropping acknowledged inputs. Implementations must charge payload/output buffers, one upload buffer and the process budget separately. Compression is initially off. No additional queue may silently defeat these bounds.

The host drives one tick at a time when permanent outcomes are possible. M6.7 committed-prefix results describe in-memory ticks, not database commits. The host cannot blindly execute 64 ticks and discover a death afterward: after each tick, detect terminal transitions, include the immutable outcome, seal the current batch and pause the whole region before any dependent next tick. The terminal tick's ordered phases commit together, including decay. No subsequent tick runs until durable confirmation. Normal flush occurs by 64 ticks or an earlier host deadline; there is no SQL call from the tick itself. Exceptionally a death can force a one-tick batch; this is not a per-tick persistence loop.

sequenceDiagram
    participant H as Single-owner region host
    participant O as Immutable payload store
    participant D as Fenced database authority
    H->>H: Execute bounded prefix; seal exact end state and outcomes
    H->>O: Put immutable candidate; verify length and SHA-256
    H->>D: Commit expected head + epoch + candidate + decisions/outbox
    D-->>H: Confirmed receipt / rejected / unknown
    alt unknown
        H->>D: Resolve same commit ID and canonical request
        Note over H: Remain paused; never assume rollback
    end
    Note over H: Only confirmed durable receipt permits final outcome and next tick

Interface sketches (not compiled stubs): put_immutable(bytes)->VerifiedObject, commit_region_batch(lease, expected_head, batch)->Confirmed|Rejected|Unknown, resolve_commit(world,region,commit_id,canonical_request)->Committed|Absent|Conflict|Unavailable, load_head(world,region)->CommittedManifest. Adapter errors carry categories and bounded diagnostic context, never credentials. M7.2 chooses concrete C++ types.

Candidate state is untrusted to the durable validator: check closure, versions, identities, end tick, resource invariants and outcome consistency before publishing. Reject changed terminal histories and impossible outcomes. This does not establish general anti-cheat against a fully compromised simulation process; database roles limit its authority and M7 tests verify the selected transition validators.

Acknowledgments, retries and recovery

Received input and simulated input ACKs are explicitly provisional. Only a receipt for the database transaction that references durably stored bytes can finalize death/reproduction or another permanent decision. Durable request IDs are scoped by world/region and are never reused. Retain canonical request and result, compare exact bytes on retry; changed contents conflict. A process's 64-entry retry cache cannot authorize pruning the durable record. Permanent history/decision IDs remain retained for the world lifetime in this initial contract, with archival later.

On timeout/disconnect, freeze and query the same commit ID. If committed, reconcile exact digest/predecessor/outcomes and adopt the receipt. If absent, retry the same request only while still authorized; concurrent delayed originals converge under the unique ID and head guard. Absence does not prove a request can never arrive. If fenced, abandon mutation and let the current owner recover; do not undo a death or ACK speculation. Conflicting bytes or unavailable reconciliation fail closed.

Recovery acquires fresh authority, reads the committed head, verifies object length and strong digest, checks the explicit version/profile allowlist, restores the full payload and reconciles durable outcomes/history to the commit. Missing/corrupt objects quarantine the region; never select an older head and silently undo ACKs. Unpublished uploads and uncommitted memory are not recovery inputs. No replay tail exists in this profile. Required historical binaries/generators remain available. Readiness requires successful closure checks and a still-valid lease.

No orphan collection in initial M7. Retention/GC must prove reachability across heads, commit/decision records, clones and backups before deleting anything. A clone gets a new identity/authority domain; references cannot permit source writes. Database plus referenced-object backups need a verified common consistency cut. Worker/pod loss with surviving durable stores is M7's test model. Host/storage loss requires separate M8 replication/restore evidence; old backups cannot silently resurrect acknowledged deaths. World0 stays gated.

Crash-point acceptance matrix (required later, not executed by M7.1)

Interruption Required durable result / recovery Gate
Before/within a tick Previous committed head; no permanent ACK 7.6/7.7
After simulation, before upload Previous head; pending outcomes not final 7.5/7.7
During upload or failed verification No manifest; partial/orphan object unreachable 7.4
After verified upload, before transaction Previous head; immutable orphan retained 7.4/7.7
Lease expires / new owner before transaction Old writer rejected; new owner loads committed head 7.3
After manifest insert before transaction commit Entire transaction rolls back, including outcomes/outbox 7.5
Commit succeeds, response lost Resolve same ID returns exact receipt, no second death/birth 7.5/7.7
Timeout while original request still running Retry converges or conflicts safely; never infer rollback 7.5
After final ACK, pod killed Latest head reconstructs ACKed state and history 7.7
Outbox delivered twice / broker down Idempotent projection or queued delivery; no second simulation mutation 7.5
Payload missing/corrupt, unknown versions Quarantine/not ready, no fallback genesis or older world 7.4/7.7
Paused stale worker resumes Renew/commit/output rejected; cannot skip recovery 7.3/7.6
Shutdown with pending/unknown commit Drain boundedly or leave unavailable; next owner reconciles 7.6/7.7
Clone/migration fails mid-publication Source untouched; target not ready, no partial version switch 7.9

M7.1 verification is a source/state inventory and contract review. Real database, object-store, process-kill and network partition tests are still required.

M7.3 executed its rows against real PostgreSQL. After lease expiry, a new owner acquires a newer epoch. The old writer's renew, release and head advance are then rejected, and the head is unchanged. A paused stale worker resuming past its lease is fenced both locally and by the database. A writer queued on the lease lock past expiry is also rejected. These are in-process pauses, not killed pods or network partitions. The 7.6 parts of those rows remain open.

M7.4a (ADR-0061) executed its rows for environmental-v1 against real PostgreSQL and a local filesystem object store:

  • A partial upload leaves only an unresolvable temporary name.
  • A failed upload sends no manifest.
  • A verified upload whose transaction is rejected stays an unreferenced orphan, and recovery returns the previous head.
  • Missing, corrupt, symlinked or resized payloads quarantine the region.
  • Unsupported builds are refused by both the database and recovery.
  • Recovery never creates genesis or picks an older head.

M7.4b (ADR-0062) closes the checkpoint side of the history audit. The lineage archive codec and the environmental-lineage-v1 profile validate:

  • organism-to-genome bindings and parent closure (by archive replay);
  • lifecycle agreement, with deaths recorded in the causing tick;
  • the living set, identity high-waters and retained dead members. Recovery restores all of these. Durable history rows (M7.5) and process-kill recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.

M7.5 (ADR-0063) executed its rows against real PostgreSQL:

  • A rejected publication (repeated death, partial or reordered decision block, gap) leaves the head, commits and decisions unchanged. Publication is a single statement; an injected failure between insert and commit is not yet tested.
  • A lost response resolves to the original receipt, and an exact retry adds no second death.
  • A client timeout while queued resolved as committed after the lock cleared, and was adopted.
  • A terminated orphan resolved as absent and was retried with the same bytes.
  • Changed bytes conflict.
  • Outbox redelivery is idempotent and never touches the simulation. Recovery now reconciles durable deaths with the restored lifecycle, so the checkpoint and durable history must agree. Kill tests at each boundary remain M7.7.

M7.6 (ADR-0064) executed its host rows:

  • A paused stale host fences itself when renewals cannot reach the database. It reports unconfirmed inputs as never final and re-acquires from the head.
  • A death pauses the region in the same step as its commit.
  • A storage outage keeps the sealed batch waiting with nothing acknowledged.
  • Shutdown drains, commits and releases.
  • A real SIGKILLed evo-server left no acknowledged tick uncommitted, and its successor became ready only after lease expiry, from the committed head.

M7.7 (ADR-0065) executed the matrix with real evo-server processes SIGKILLed at each boundary: before the transaction (first and second batch), before upload, after commit but before acknowledgement, and after acknowledgement.

  • In every case acknowledgements stayed at or below the committed head.
  • Deaths were final only once committed, and verified orphans remained.
  • Each successor verified the head and its tail before readiness. A SIGSTOPped owner resumed fenced: it was not ready, refused admission and acknowledged nothing new. Corrupt payloads and missing traces quarantined with no fallback. Host loss and clone or migration remain M8 and M7.9.

M7.9a (ADR-0067) clones a verified committed head into another world and database:

  • The payload is re-identified, and the byte round trip is proven.
  • The clone genesis names the source commit and digest, and inherits the source's outcomes and ancestry.
  • The source session is read-only.
  • A clone killed before its target transaction left the source untouched and the target unprovisioned (it quarantined rather than serve), and a retry completed.
  • The same inputs from the cut reproduced the source trajectory exactly.

audit_region_history re-verifies every retained link across owners and kills. It names the exact link whose evidence is missing, even though head recovery still succeeds. Migration, fencing of old writers, and measured recovery and batching latency are still M7.9b.

M7.9b (ADR-0068) closes the clone/migration row.

  • Failed migration while serving. A v8 migration that failed at its last statement left v7 exactly, and the lease-holding writer kept publishing. The real migration then committed under that same lease.
  • Old writers. The upgrade barrier retires the old build and fences every holder. After it:
    • the old writer cannot publish, renew or re-acquire, by either acquisition form;
    • no link naming the retired build can commit, even under a current lease;
    • a host of the old image quarantines;
    • the new build recovered the whole history and continued.
  • Rejection. Rule, format and profile changes are refused; no converter exists.
  • Qualification. Real SIGKILLs under the production lease were recovered in 19.5–19.6 s against the proposed 60 s target, with no acknowledged input lost (see engineering/DURABILITY_QUALIFICATION.md).