Add M10.8a probe handoff schema v12 and store
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (client) (push) Successful in 12m32s
Foundation / checks (network) (push) Successful in 12m17s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (tsan) (push) Canceled after 2m2s
Foundation / checks (postgres) (push) Canceled after 3m48s
Foundation / checks (checks) (push) Canceled after 16m29s

ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable
probe records hold one owning region per account, so ownership is unique and
the test inventory is conserved by construction. A fenced transfer state
machine (prepared, staged, committed, aborted) under each caller's exact
region lease returns the recorded outcome on every retry; a commit requires
staging by the destination's current epoch; aborts are durable and only
before commit. handoff_account_control moves account control on a committed
transfer.

A PostgreSQL fault matrix covers restarts at each transition, stale epochs,
failed destinations, partitions and commit/abort races, checking one owner and
the conserved inventory after every scenario.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-06 18:15:49 -04:00
co-authored by Claude Opus 5.5
parent 5728e7682e
commit 6a645ef7e4
18 changed files with 1355 additions and 16 deletions
+4 -1
View File
@@ -533,7 +533,10 @@ probe-motion-v1 rule, server markers, session protocol v3 marker records and the
predictor done; 10.6b wiring (sessions, evo-server, probe, GUI marker) done; 10.6c impaired
traces done (networking/M10_PREDICTION_TRACES.md). 10.7 split (ADR-0095): 10.7a tick clock,
remote-marker interpolation and fractional prediction done; 10.7b client wiring, stale loss
view and two-client interpolation traces done. 10.8-10.9 planned.
view and two-client interpolation traces done. 10.8 split (ADR-0096, the persistence
handoff ADR; EXPERIMENT durable probe records): 10.8a schema v12 probe records and the
transfer state machine with a PostgreSQL fault matrix done; 10.8b handoff tickets, 10.8c
region wiring and 10.8d two-process kill tests follow. 10.9 planned.
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
+9 -2
View File
@@ -64,8 +64,15 @@ interpolation of remote markers (2-6 ticks behind, at most 2 ticks extrapolated,
stale; organisms never) and sub-tick own-marker presentation. M10.7b wires it into the
engineering client. A lost session's view stays frozen and labelled stale until the
reconnect's fresh baseline. In two-client traces, remote markers shown as current stay
within about 0.1 m of the true path, and outages show stale. M10.7 is complete. Next:
M10.8 authority transfer between two regions (needs a ratified handoff ADR).
within about 0.1 m of the true path, and outages show stale. M10.7 is complete. M10.8a (ADR-0096, the ratified handoff ADR) adds schema v12:
- durable probe records, one owning region per account;
- a fenced transfer state machine (prepared, staged, committed, aborted) whose every retry
returns the recorded outcome;
- an account-control handoff for committed transfers.
A PostgreSQL fault matrix covers restarts at each transition, stale epochs, aborts,
partitions and commit/abort races, checking a unique owner and the conserved test inventory.
Deploying v12 needs a rebuilt migrations image. Next: 10.8b handoff tickets.
## Previous gate - 2026-10-02
+116
View File
@@ -0,0 +1,116 @@
# ADR-0096: M10 cross-region handoff of probe records
## Status
Accepted for M10.8 (2026-10-06). This is the persistence handoff ADR the roadmap requires
before 10.8. Decisions confirmed with the maintainer on 2026-10-06:
- the transferred entity is a durable probe record with a conserved test inventory
(EXPERIMENT, amends ADR-0094);
- regions own adjacent disjoint areas, and crossing the edge starts the handoff, the
destination holding a read-only ghost first;
- the client is admitted at the destination with a short-lived transfer ticket signed by
the source region;
- evidence is a library-level fault matrix plus two real region processes on PostgreSQL,
with a kill at each durable step.
Delivery:
1. **10.8a:** schema v12 and the store, with the fault matrix (this text).
2. **10.8b:** handoff keys and tickets, and ticket admission.
3. **10.8c:** region-host wiring (edge crossing, ghost, freeze, activation, client
rerouting without a loading screen).
4. **10.8d:** two-process kill tests and conservation evidence.
Each segment amends this ADR.
**EXPERIMENT.** A probe is a non-biological engineering entity. Nothing here decides how
organisms move between regions; M11 reuses the state machine for organisms only through
its own ADR.
## Context
PERSISTENCE_ARCHITECTURE "Entity handoff across regions" outlines a handoff:
1. the destination holds a lease and a read-only ghost;
2. the source freezes and durably prepares the final state;
3. the destination durably stages it;
4. one transaction switches ownership;
5. the destination activates and the source tombstones.
Abort is durable and only before commit. Retries return the recorded outcome, and a crash
at any step yields one owner, never duplication or loss. OPEN_QUESTIONS 24 proposes fenced
leases plus an explicit handoff state machine. Until M10.8 no entity was durable and
controllable: probe markers lived in region memory (ADR-0094), and organisms are not
controllable before M11.
## Decision (10.8a, schema v12)
- **Probe record** (`evolution.probe_records`). One row per (world, account): the owning
region, an ownership generation, position (micrometres, cell slice) and an EXPERIMENT
test inventory, plus the open transfer while frozen. Ownership is a single column of a
single row, so the owner is unique and the inventory is conserved by construction: a
transfer moves the row's ownership, never copies it. A region acts for a probe only if
it owns it and it is not frozen. Positions are saved durably at session end, at
preparation and on commit, never per tick (PROJECT_VISION).
- **Transfer** (`evolution.probe_transfers`), state machine:
- **prepared:** the source froze the probe at its final state for a named destination.
- **staged:** the destination validated and durably accepted it under its current
epoch; again after its own restart.
- **committed:** the destination switched ownership in one transaction, with generation
+ 1 and the final state.
- **aborted:** by the source or the destination, before commit only; the source resumes.
- At most one open transfer per probe.
- **Retries.** A retry of any call with the same transfer id returns the recorded outcome:
prepare by its source, stage and commit by its destination, abort by either. After a
commit, an abort returns the commit; after an abort, stage and commit return the abort.
- **Fencing.** Every function first requires the caller's exact unexpired region lease.
- A commit requires the transfer staged by the caller's current epoch: a destination
restarted after staging must stage again.
- No function locks another region's lease row (lock order: upgrade lock, own lease,
record, transfer). The frozen record is what stops the source, under any of its
epochs, from acting while a transfer is open.
- Timeout alone never authorises a second copy: the source resumes only through a
durable abort.
- **Account control.** `handoff_account_control` gives the destination the account's
durable control (schema v10) for a transfer committed to it, instead of a newer grant
serial. Another region's unexpired control is still outlived; the source releases its
control when it prepares, so normally there is no wait.
- **SQLSTATE:**
- EVL13 probe fenced;
- EVL14 unknown transfer;
- EVL15 not staged by this epoch;
- EVL12 for the control handoff.
- The runtime reaches both tables only through the eight functions, enforced by the
effective-privilege audit.
## Planned (10.8b-d, amended when delivered)
- **Tickets.**
- The source signs a short-lived ticket with its region handoff key (ECDSA P-256, as
grants, domain-separated): world, account, transfer id, source, destination, expiry.
- The destination verifies it against the world's configured region keys and admits
the client only for a transfer it can stage and commit.
- Reconnects after a handoff use grants for the owning region; a dev bundle may hold
grants for several regions.
- **Crossing.**
- Regions own adjacent disjoint areas. A marker crossing its region's edge is frozen at
the crossing tick and prepared; the source releases its account control and offers
the client the ticket.
- The client keeps its last view (no loading screen), connects to the destination, and
the destination stages, commits, takes control and opens the session.
- Until the commit the destination holds the staged state as a read-only ghost.
- If the client never arrives, the source aborts after the ticket expires and resumes.
- **Evidence.** Two evo-server processes on one PostgreSQL:
- a crossing and a crossing back;
- SIGKILL of the source or the destination around each durable step, duplicate and
stale retries, and a failed destination;
- unique-owner and conservation audits throughout.
## Alternatives considered
- Copying the state into the transfer and deleting it from the source: conservation then
needs a cross-row invariant instead of holding by construction.
- Locking the source's lease row at commit: breaks the lock order (deadlock between
crossing regions) and adds nothing, since the frozen record already fences the source.
- World-scoped grants or per-region bundles for admission: a grant would open any region
(or bundles would not scale); the ticket names the one destination.
- Durable markers every tick: forbidden (no per-tick database writes).
## Consequences
- Schema v12 must be deployed with the tools image before a region uses probe records.
The dev migrations image must be rebuilt (the v10 lesson).
- A probe's durable position can lag its in-memory position by up to a session:
presentation-only state, never biological.
+47
View File
@@ -4679,3 +4679,50 @@ The Postgres suite was not rerun: no server code changed in M10.7.
Not done:
- interpolating organisms (none move);
- a GUI run against evolution-dev (needs a v3 server).
## 2026-10-06 — M10.8a probe handoff schema v12 (ADR-0096)
Implemented:
- `migrations/0012_probe_handoff.sql`:
- `probe_records` and `probe_transfers`;
- `open_probe`, `save_probe`, `prepare`, `stage`, `commit`, `abort`, `find`;
- `handoff_account_control`;
- SQLSTATEs EVL13-15;
- the apply chain, migration runner, privilege audit allowlist, CI version checks and
tamper tests;
- the C++ store (`include/evo/persistence/probe_store.hpp`, and `handoff_account_control`
in `account_control_store.hpp`);
- `schemas/probe-handoff-v1.md` and the POSTGRESQL.md section.
`tests/probe_handoff_store_tests.cpp` (world 9200, real PostgreSQL) covers:
- ownership and fencing; argument refusals;
- the happy path with retries of every call before and after commit;
- one open transfer per probe; transfer ids bound to one account;
- crossing back;
- source restart after prepare;
- destination restart after staging (re-stage required; the old epoch fenced) and after
commit (the retry answers);
- a failed destination (source abort); a destination abort;
- a partition (the source lease lapses while the destination commits; the recovered source
is fenced);
- 24 commit/abort races on separate connections (7 commits, 17 aborts; both sides always
observed the same outcome);
- the account-control handoff (refused before commit, for another region or account; a wait
while the source holds control; none after release).
After every scenario: one row per probe, the test inventory conserved (n x 1000), and frozen
exactly when an open transfer names the owner as source.
Executed:
- **Linux GCC 14:** `postgres-check.sh` with QUIC: 21 suites, including probe handoff,
migrations (v1-v11 upgrade paths to v12), tamper and audit checks;
- preset `quic`: 4/4.
- **Linux Clang 19:** clang-tidy 19 on `region_lease_store.cpp` and the new test (two
findings fixed); clang-format 19.
The suite was rerun on the final code after the tidy fixes: 21 suites passed again; that
run's races ended 11 commits, 13 aborts.
Not done:
- tickets (10.8b), region wiring (10.8c) and two-process kills (10.8d);
- deploying v12 (needs a rebuilt migrations image; a maintainer step).
+19
View File
@@ -200,6 +200,25 @@ updated). Exact retries and `resolve_region_commit(..., actions)` compare the bl
`find_region_action` reads one record under the caller's exact region lease. The privilege
audit covers all three functions; the tables and `check_action_block` are owner-only.
## Probe records and handoff (schema v12, ADR-0096)
EXPERIMENT (non-biological probes, ADR-0094). `evolution.probe_records` holds one row per
(world, account): owning region, generation, position, test inventory and the open
transfer while frozen. `evolution.probe_transfers` holds the transfer state machine:
prepared, staged, committed, aborted. Runtime access is only through `open_probe`,
`save_probe`, `prepare_probe_transfer`, `stage_probe_transfer`, `commit_probe_transfer`,
`abort_probe_transfer`, `find_probe_transfer` and `handoff_account_control`, each under the
caller's exact unexpired region lease. Every transfer call can be retried with its id and
returns the recorded outcome. SQLSTATEs:
- `EVL13`: probe fenced;
- `EVL14`: unknown transfer;
- `EVL15`: not staged by this epoch.
Lock order: upgrade lock (shared), own region lease row, probe record, transfer;
`handoff_account_control` reads the transfer, then locks the account control row. The
privilege audit covers all eight functions, and the tables are owner-only. Deploying v12
needs a rebuilt migrations (tools) image.
## Host lease discipline
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal