Add M10.8a probe handoff schema v12 and store
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (client) (push) Successful in 12m32s
Foundation / checks (network) (push) Successful in 12m17s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (tsan) (push) Canceled after 2m2s
Foundation / checks (postgres) (push) Canceled after 3m48s
Foundation / checks (checks) (push) Canceled after 16m29s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (client) (push) Successful in 12m32s
Foundation / checks (network) (push) Successful in 12m17s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (tsan) (push) Canceled after 2m2s
Foundation / checks (postgres) (push) Canceled after 3m48s
Foundation / checks (checks) (push) Canceled after 16m29s
ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable probe records hold one owning region per account, so ownership is unique and the test inventory is conserved by construction. A fenced transfer state machine (prepared, staged, committed, aborted) under each caller's exact region lease returns the recorded outcome on every retry; a commit requires staging by the destination's current epoch; aborts are durable and only before commit. handoff_account_control moves account control on a committed transfer. A PostgreSQL fault matrix covers restarts at each transition, stale epochs, failed destinations, partitions and commit/abort races, checking one owner and the conserved inventory after every scenario. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+4
-1
@@ -533,7 +533,10 @@ probe-motion-v1 rule, server markers, session protocol v3 marker records and the
|
||||
predictor done; 10.6b wiring (sessions, evo-server, probe, GUI marker) done; 10.6c impaired
|
||||
traces done (networking/M10_PREDICTION_TRACES.md). 10.7 split (ADR-0095): 10.7a tick clock,
|
||||
remote-marker interpolation and fractional prediction done; 10.7b client wiring, stale loss
|
||||
view and two-client interpolation traces done. 10.8-10.9 planned.
|
||||
view and two-client interpolation traces done. 10.8 split (ADR-0096, the persistence
|
||||
handoff ADR; EXPERIMENT durable probe records): 10.8a schema v12 probe records and the
|
||||
transfer state machine with a PostgreSQL fault matrix done; 10.8b handoff tickets, 10.8c
|
||||
region wiring and 10.8d two-process kill tests follow. 10.9 planned.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+9
-2
@@ -64,8 +64,15 @@ interpolation of remote markers (2-6 ticks behind, at most 2 ticks extrapolated,
|
||||
stale; organisms never) and sub-tick own-marker presentation. M10.7b wires it into the
|
||||
engineering client. A lost session's view stays frozen and labelled stale until the
|
||||
reconnect's fresh baseline. In two-client traces, remote markers shown as current stay
|
||||
within about 0.1 m of the true path, and outages show stale. M10.7 is complete. Next:
|
||||
M10.8 authority transfer between two regions (needs a ratified handoff ADR).
|
||||
within about 0.1 m of the true path, and outages show stale. M10.7 is complete. M10.8a (ADR-0096, the ratified handoff ADR) adds schema v12:
|
||||
- durable probe records, one owning region per account;
|
||||
- a fenced transfer state machine (prepared, staged, committed, aborted) whose every retry
|
||||
returns the recorded outcome;
|
||||
- an account-control handoff for committed transfers.
|
||||
|
||||
A PostgreSQL fault matrix covers restarts at each transition, stale epochs, aborts,
|
||||
partitions and commit/abort races, checking a unique owner and the conserved test inventory.
|
||||
Deploying v12 needs a rebuilt migrations image. Next: 10.8b handoff tickets.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# ADR-0096: M10 cross-region handoff of probe records
|
||||
|
||||
## Status
|
||||
Accepted for M10.8 (2026-10-06). This is the persistence handoff ADR the roadmap requires
|
||||
before 10.8. Decisions confirmed with the maintainer on 2026-10-06:
|
||||
- the transferred entity is a durable probe record with a conserved test inventory
|
||||
(EXPERIMENT, amends ADR-0094);
|
||||
- regions own adjacent disjoint areas, and crossing the edge starts the handoff, the
|
||||
destination holding a read-only ghost first;
|
||||
- the client is admitted at the destination with a short-lived transfer ticket signed by
|
||||
the source region;
|
||||
- evidence is a library-level fault matrix plus two real region processes on PostgreSQL,
|
||||
with a kill at each durable step.
|
||||
|
||||
Delivery:
|
||||
1. **10.8a:** schema v12 and the store, with the fault matrix (this text).
|
||||
2. **10.8b:** handoff keys and tickets, and ticket admission.
|
||||
3. **10.8c:** region-host wiring (edge crossing, ghost, freeze, activation, client
|
||||
rerouting without a loading screen).
|
||||
4. **10.8d:** two-process kill tests and conservation evidence.
|
||||
|
||||
Each segment amends this ADR.
|
||||
|
||||
**EXPERIMENT.** A probe is a non-biological engineering entity. Nothing here decides how
|
||||
organisms move between regions; M11 reuses the state machine for organisms only through
|
||||
its own ADR.
|
||||
|
||||
## Context
|
||||
PERSISTENCE_ARCHITECTURE "Entity handoff across regions" outlines a handoff:
|
||||
1. the destination holds a lease and a read-only ghost;
|
||||
2. the source freezes and durably prepares the final state;
|
||||
3. the destination durably stages it;
|
||||
4. one transaction switches ownership;
|
||||
5. the destination activates and the source tombstones.
|
||||
|
||||
Abort is durable and only before commit. Retries return the recorded outcome, and a crash
|
||||
at any step yields one owner, never duplication or loss. OPEN_QUESTIONS 24 proposes fenced
|
||||
leases plus an explicit handoff state machine. Until M10.8 no entity was durable and
|
||||
controllable: probe markers lived in region memory (ADR-0094), and organisms are not
|
||||
controllable before M11.
|
||||
|
||||
## Decision (10.8a, schema v12)
|
||||
- **Probe record** (`evolution.probe_records`). One row per (world, account): the owning
|
||||
region, an ownership generation, position (micrometres, cell slice) and an EXPERIMENT
|
||||
test inventory, plus the open transfer while frozen. Ownership is a single column of a
|
||||
single row, so the owner is unique and the inventory is conserved by construction: a
|
||||
transfer moves the row's ownership, never copies it. A region acts for a probe only if
|
||||
it owns it and it is not frozen. Positions are saved durably at session end, at
|
||||
preparation and on commit, never per tick (PROJECT_VISION).
|
||||
- **Transfer** (`evolution.probe_transfers`), state machine:
|
||||
- **prepared:** the source froze the probe at its final state for a named destination.
|
||||
- **staged:** the destination validated and durably accepted it under its current
|
||||
epoch; again after its own restart.
|
||||
- **committed:** the destination switched ownership in one transaction, with generation
|
||||
+ 1 and the final state.
|
||||
- **aborted:** by the source or the destination, before commit only; the source resumes.
|
||||
- At most one open transfer per probe.
|
||||
- **Retries.** A retry of any call with the same transfer id returns the recorded outcome:
|
||||
prepare by its source, stage and commit by its destination, abort by either. After a
|
||||
commit, an abort returns the commit; after an abort, stage and commit return the abort.
|
||||
- **Fencing.** Every function first requires the caller's exact unexpired region lease.
|
||||
- A commit requires the transfer staged by the caller's current epoch: a destination
|
||||
restarted after staging must stage again.
|
||||
- No function locks another region's lease row (lock order: upgrade lock, own lease,
|
||||
record, transfer). The frozen record is what stops the source, under any of its
|
||||
epochs, from acting while a transfer is open.
|
||||
- Timeout alone never authorises a second copy: the source resumes only through a
|
||||
durable abort.
|
||||
- **Account control.** `handoff_account_control` gives the destination the account's
|
||||
durable control (schema v10) for a transfer committed to it, instead of a newer grant
|
||||
serial. Another region's unexpired control is still outlived; the source releases its
|
||||
control when it prepares, so normally there is no wait.
|
||||
- **SQLSTATE:**
|
||||
- EVL13 probe fenced;
|
||||
- EVL14 unknown transfer;
|
||||
- EVL15 not staged by this epoch;
|
||||
- EVL12 for the control handoff.
|
||||
- The runtime reaches both tables only through the eight functions, enforced by the
|
||||
effective-privilege audit.
|
||||
|
||||
## Planned (10.8b-d, amended when delivered)
|
||||
- **Tickets.**
|
||||
- The source signs a short-lived ticket with its region handoff key (ECDSA P-256, as
|
||||
grants, domain-separated): world, account, transfer id, source, destination, expiry.
|
||||
- The destination verifies it against the world's configured region keys and admits
|
||||
the client only for a transfer it can stage and commit.
|
||||
- Reconnects after a handoff use grants for the owning region; a dev bundle may hold
|
||||
grants for several regions.
|
||||
- **Crossing.**
|
||||
- Regions own adjacent disjoint areas. A marker crossing its region's edge is frozen at
|
||||
the crossing tick and prepared; the source releases its account control and offers
|
||||
the client the ticket.
|
||||
- The client keeps its last view (no loading screen), connects to the destination, and
|
||||
the destination stages, commits, takes control and opens the session.
|
||||
- Until the commit the destination holds the staged state as a read-only ghost.
|
||||
- If the client never arrives, the source aborts after the ticket expires and resumes.
|
||||
- **Evidence.** Two evo-server processes on one PostgreSQL:
|
||||
- a crossing and a crossing back;
|
||||
- SIGKILL of the source or the destination around each durable step, duplicate and
|
||||
stale retries, and a failed destination;
|
||||
- unique-owner and conservation audits throughout.
|
||||
|
||||
## Alternatives considered
|
||||
- Copying the state into the transfer and deleting it from the source: conservation then
|
||||
needs a cross-row invariant instead of holding by construction.
|
||||
- Locking the source's lease row at commit: breaks the lock order (deadlock between
|
||||
crossing regions) and adds nothing, since the frozen record already fences the source.
|
||||
- World-scoped grants or per-region bundles for admission: a grant would open any region
|
||||
(or bundles would not scale); the ticket names the one destination.
|
||||
- Durable markers every tick: forbidden (no per-tick database writes).
|
||||
|
||||
## Consequences
|
||||
- Schema v12 must be deployed with the tools image before a region uses probe records.
|
||||
The dev migrations image must be rebuilt (the v10 lesson).
|
||||
- A probe's durable position can lag its in-memory position by up to a session:
|
||||
presentation-only state, never biological.
|
||||
@@ -4679,3 +4679,50 @@ The Postgres suite was not rerun: no server code changed in M10.7.
|
||||
Not done:
|
||||
- interpolating organisms (none move);
|
||||
- a GUI run against evolution-dev (needs a v3 server).
|
||||
|
||||
## 2026-10-06 — M10.8a probe handoff schema v12 (ADR-0096)
|
||||
|
||||
Implemented:
|
||||
- `migrations/0012_probe_handoff.sql`:
|
||||
- `probe_records` and `probe_transfers`;
|
||||
- `open_probe`, `save_probe`, `prepare`, `stage`, `commit`, `abort`, `find`;
|
||||
- `handoff_account_control`;
|
||||
- SQLSTATEs EVL13-15;
|
||||
- the apply chain, migration runner, privilege audit allowlist, CI version checks and
|
||||
tamper tests;
|
||||
- the C++ store (`include/evo/persistence/probe_store.hpp`, and `handoff_account_control`
|
||||
in `account_control_store.hpp`);
|
||||
- `schemas/probe-handoff-v1.md` and the POSTGRESQL.md section.
|
||||
|
||||
`tests/probe_handoff_store_tests.cpp` (world 9200, real PostgreSQL) covers:
|
||||
- ownership and fencing; argument refusals;
|
||||
- the happy path with retries of every call before and after commit;
|
||||
- one open transfer per probe; transfer ids bound to one account;
|
||||
- crossing back;
|
||||
- source restart after prepare;
|
||||
- destination restart after staging (re-stage required; the old epoch fenced) and after
|
||||
commit (the retry answers);
|
||||
- a failed destination (source abort); a destination abort;
|
||||
- a partition (the source lease lapses while the destination commits; the recovered source
|
||||
is fenced);
|
||||
- 24 commit/abort races on separate connections (7 commits, 17 aborts; both sides always
|
||||
observed the same outcome);
|
||||
- the account-control handoff (refused before commit, for another region or account; a wait
|
||||
while the source holds control; none after release).
|
||||
|
||||
After every scenario: one row per probe, the test inventory conserved (n x 1000), and frozen
|
||||
exactly when an open transfer names the owner as source.
|
||||
|
||||
Executed:
|
||||
- **Linux GCC 14:** `postgres-check.sh` with QUIC: 21 suites, including probe handoff,
|
||||
migrations (v1-v11 upgrade paths to v12), tamper and audit checks;
|
||||
- preset `quic`: 4/4.
|
||||
- **Linux Clang 19:** clang-tidy 19 on `region_lease_store.cpp` and the new test (two
|
||||
findings fixed); clang-format 19.
|
||||
|
||||
The suite was rerun on the final code after the tidy fixes: 21 suites passed again; that
|
||||
run's races ended 11 commits, 13 aborts.
|
||||
|
||||
Not done:
|
||||
- tickets (10.8b), region wiring (10.8c) and two-process kills (10.8d);
|
||||
- deploying v12 (needs a rebuilt migrations image; a maintainer step).
|
||||
|
||||
@@ -200,6 +200,25 @@ updated). Exact retries and `resolve_region_commit(..., actions)` compare the bl
|
||||
`find_region_action` reads one record under the caller's exact region lease. The privilege
|
||||
audit covers all three functions; the tables and `check_action_block` are owner-only.
|
||||
|
||||
## Probe records and handoff (schema v12, ADR-0096)
|
||||
|
||||
EXPERIMENT (non-biological probes, ADR-0094). `evolution.probe_records` holds one row per
|
||||
(world, account): owning region, generation, position, test inventory and the open
|
||||
transfer while frozen. `evolution.probe_transfers` holds the transfer state machine:
|
||||
prepared, staged, committed, aborted. Runtime access is only through `open_probe`,
|
||||
`save_probe`, `prepare_probe_transfer`, `stage_probe_transfer`, `commit_probe_transfer`,
|
||||
`abort_probe_transfer`, `find_probe_transfer` and `handoff_account_control`, each under the
|
||||
caller's exact unexpired region lease. Every transfer call can be retried with its id and
|
||||
returns the recorded outcome. SQLSTATEs:
|
||||
- `EVL13`: probe fenced;
|
||||
- `EVL14`: unknown transfer;
|
||||
- `EVL15`: not staged by this epoch.
|
||||
|
||||
Lock order: upgrade lock (shared), own region lease row, probe record, transfer;
|
||||
`handoff_account_control` reads the transfer, then locks the account control row. The
|
||||
privilege audit covers all eight functions, and the tables are owner-only. Deploying v12
|
||||
needs a rebuilt migrations (tools) image.
|
||||
|
||||
## Host lease discipline
|
||||
|
||||
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
|
||||
|
||||
Reference in New Issue
Block a user