16 KiB
[Reviewed]
ADR-0096: M10 cross-region handoff of probe records
Status
Accepted for M10.8 (2026-10-06). This is the persistence handoff ADR the roadmap requires before 10.8. Decisions confirmed with the maintainer on 2026-10-06:
- the transferred entity is a durable probe record with a conserved test inventory (EXPERIMENT, amends ADR-0094);
- regions own adjacent disjoint areas, and crossing the edge starts the handoff, the destination holding a read-only ghost first;
- the client is admitted at the destination with a short-lived transfer ticket signed by the source region;
- evidence is a library-level fault matrix plus two real region processes on PostgreSQL, with a kill at each durable step.
Delivery:
- 10.8a: schema v12 and the store, with the fault matrix (this text).
- 10.8b: handoff keys and tickets, and ticket admission.
- 10.8c: region-host wiring (edge crossing, ghost, freeze, activation, client rerouting without a loading screen).
- 10.8d: two-process kill tests and conservation evidence.
Each segment amends this ADR.
EXPERIMENT. A probe is a non-biological engineering entity. Nothing here decides how organisms move between regions; M11 reuses the state machine for organisms only through its own ADR.
Context
PERSISTENCE_ARCHITECTURE "Entity handoff across regions" outlines a handoff:
- the destination holds a lease and a read-only ghost;
- the source freezes and durably prepares the final state;
- the destination durably stages it;
- one transaction switches ownership;
- the destination activates and the source tombstones.
Abort is durable and only before commit. Retries return the recorded outcome, and a crash at any step yields one owner, never duplication or loss. OPEN_QUESTIONS 24 proposes fenced leases plus an explicit handoff state machine. Until M10.8 no entity was durable and controllable: probe markers lived in region memory (ADR-0094), and organisms are not controllable before M11.
Decision (10.8a, schema v12)
- Probe record (
evolution.probe_records). One row per (world, account): the owning region, an ownership generation, position (micrometres, cell slice) and an EXPERIMENT test inventory, plus the open transfer while frozen. Ownership is a single column of a single row, so the owner is unique and the inventory is conserved by construction: a transfer moves the row's ownership, never copies it. A region acts for a probe only if it owns it and it is not frozen. Positions are saved durably at session end, at preparation and on commit, never per tick (PROJECT_VISION). - Transfer (
evolution.probe_transfers), state machine:- prepared: the source froze the probe at its final state for a named destination.
- staged: the destination validated and durably accepted it under its current epoch; again after its own restart.
- committed: the destination switched ownership in one transaction, with generation
- 1 and the final state.
- aborted: by the source or the destination, before commit only; the source resumes.
- At most one open transfer per probe.
- Retries. A retry of any call with the same transfer id returns the recorded outcome: prepare by its source, stage and commit by its destination, abort by either. After a commit, an abort returns the commit; after an abort, stage and commit return the abort.
- Fencing. Every function first requires the caller's exact unexpired region lease.
- A commit requires the transfer staged by the caller's current epoch: a destination restarted after staging must stage again.
- No function locks another region's lease row (lock order: upgrade lock, own lease, record, transfer). The frozen record is what stops the source, under any of its epochs, from acting while a transfer is open.
- Timeout alone never authorises a second copy: the source resumes only through a durable abort.
- Account control.
handoff_account_controlgives the destination the account's durable control (schema v10) for a transfer committed to it, instead of a newer grant serial. Another region's unexpired control is still outlived; the source releases its control when it prepares, so normally there is no wait. - SQLSTATE:
- EVL13 probe fenced;
- EVL14 unknown transfer;
- EVL15 not staged by this epoch;
- EVL12 for the control handoff.
- The runtime reaches both tables only through the eight functions, enforced by the effective-privilege audit.
Planned (10.8b-d, amended when delivered)
- Tickets.
- The source signs a short-lived ticket with its region handoff key (ECDSA P-256, as grants, domain-separated): world, account, transfer id, source, destination, expiry.
- The destination verifies it against the world's configured region keys and admits the client only for a transfer it can stage and commit.
- Reconnects after a handoff use grants for the owning region; a dev bundle may hold grants for several regions.
- Crossing.
- Regions own adjacent disjoint areas. A marker crossing its region's edge is frozen at the crossing tick and prepared; the source releases its account control and offers the client the ticket.
- The client keeps its last view (no loading screen), connects to the destination, and the destination stages, commits, takes control and opens the session.
- Until the commit the destination holds the staged state as a read-only ghost.
- If the client never arrives, the source aborts after the ticket expires and resumes.
- Evidence. Two evo-server processes on one PostgreSQL:
- a crossing and a crossing back;
- SIGKILL of the source or the destination around each durable step, duplicate and stale retries, and a failed destination;
- unique-owner and conservation audits throughout.
Alternatives considered
- Copying the state into the transfer and deleting it from the source: conservation then needs a cross-row invariant instead of holding by construction.
- Locking the source's lease row at commit: breaks the lock order (deadlock between crossing regions) and adds nothing, since the frozen record already fences the source.
- World-scoped grants or per-region bundles for admission: a grant would open any region (or bundles would not scale); the ticket names the one destination.
- Durable markers every tick: forbidden (no per-tick database writes).
Consequences
- Schema v12 must be deployed with the tools image before a region uses probe records. The dev migrations image must be rebuilt (the v10 lesson).
- A probe's durable position can lag its in-memory position by up to a session: presentation-only state, never biological.
M10.8b tickets and ticket admission (2026-10-06)
-
Ticket (handoff-ticket-v1): 160 bytes, the same as a grant, so it rides the
Hellogrant slot with no protocol change. Fields:- an "EVO-HANDOFF" marker, version, environment and key id;
- the transfer id, account, world, frame, source and destination;
- the ticket expiry (at most 120 s ahead) and the session expiry, copied from the client's grant at the source so a crossing never extends a session;
- an ECDSA P-256 signature.
Each region has its own handoff key (
HandoffSigningKey). A destination registers its neighbours' public keys, each bound to its source region (HandoffVerifier, at most 32), and accepts a ticket only if the signing key belongs to the claimed source, the world and destination are its own, and the times hold. Grants and tickets never verify as each other. -
Admission.
RegionSessionsroutes a Hello by the ticket marker. A ticket is verified, thenSessionControl::check_handoffruns: single use per region, the attempt counted, supersedes like a grant, serials untouched. Then the host'sHandoffLedgermakes it durable, and only then does the localopen_handoffrun. The session expiry is the ticket's, capped at the maximum grant lifetime from now. Renewal and release use the ControlLedger under the generation the handoff returned. A handoff ledger without a control ledger is a configuration error. -
PostgreSQL (
PostgresHandoffLedger): stage (under the current lease, again after a restart), commit, thenhandoff_account_control, all idempotent, so a retried login converges. Unknown, aborted, foreign-account, foreign-source or foreign-region transfers are refused (unauthenticated); a fenced lease givesshutdown; undecided givescapacity. -
Not yet: the source side (freeze at the edge, prepare, release control, offer the ticket), the client switch, keys in evo-server and the dev tool (10.8c).
M10.8c-1 region wiring (2026-10-06)
-
Session protocol v4 (session-v4; v3 was deployed and is frozen) adds:
HandoffOffer(transfer id, destination address, ticket);SessionRedirect(where the probe is, before Welcome);- close reasons
handed_offandmoved; - marker kind 2 (ghost).
-
Durable probes in sessions (
ProbeLedger,include/evo/server/probe_ledger.hpp).- A grant login opens the account's probe record after the durable control claim.
- Owned here and active: the marker opens at its saved position, clamped into this region's area, so a probe left beyond the edge by an aborted handoff does not cross again at once.
- Owned elsewhere: Redirect (when that region is a known neighbour), then
moved. - Frozen:
capacity(retry once the transfer is decided).
- The position is saved at session end, but not after a handoff.
- A grant login opens the account's probe record after the durable control claim.
-
Neighbours. A region owns its area; markers move within the bounding box of its area and its neighbours'. Every poll, a marker that left the area into a neighbour's area is frozen and its transfer prepared. Then the session's control is released, a ticket is signed (lifetime 30 s; its session expiry is the session's), the HandoffOffer sent, and the session closed with
handed_off. Anywhere else, the marker is put back at the edge. An outgoing transfer not committed within its ticket's lifetime plus 5 s is aborted (the probe resumes here); undecided aborts retry. -
Destination. The ticket admission now returns the probe's state after the switch (
HandoffAdmission), and the marker opens there. Every second (configurable) the region reads the open transfers addressed to it (schema v13incoming_probe_transfers) and shows them as ghosts: stable ids per transfer, read-only, replicated to sessions whose square holds them, and gone once decided. -
Configuration. Handoffs need probe records, this region's handoff key and neighbours. A handoff key without probe records is a configuration error.
-
Evidence.
two_region_handoff(tests/quic_sessions.cpp): two regions with adjacent areas in one process, over a fake world database with the v12 state machine. It covers:- a crossing at the border;
- a ghost seen by another client at the destination before the switch;
- the ticket login (owner and generation moved, marker where it crossed);
- the ghost giving way;
- a redirect at the old region;
- a crossing back;
- an abandoned handoff aborted after its deadline, with the probe resumed at the edge and the late ticket refused as expired.
It passed 3 times in a row on Linux and also passes on Windows.
-
Not yet: the engineering client's switch (offer, redirect, multi-region grants), evo-server and probe configuration (10.8c-2), and real processes on PostgreSQL (10.8d).
M10.8c-2 client switch and configuration (2026-10-06)
-
Engineering client (
apps/evo-client/remote_session.hpp):- On
Close(handed_off)after a HandoffOffer it learns the destination's address (one lookup), keeps the frozen view, uses no grant and connects at once with the ticket. The panel shows "handing off", then the region and the count of crossings. - On
Close(moved)after a Redirect it connects to the owner with a grant for that region. - Grants are picked by their region claim. They are used in rising serial order world-wide (durable control refuses a serial not above the newest), and a grant outdated by one already used is discarded.
- A new region or epoch resets the views.
- On
-
evo-server (dev only, with the M10 listener):
--m10-area X0,Y0,X1,Y1: the region's cells.--m10-handoff-key FILE: this region's P-256 scalar; the key id is the region id.--m10-neighbour REGION=X0,Y0,X1,Y1=HOST:PORT=SERVER_NAME=PUBLIC_KEY_FILE: up to 8; the neighbour's key id is its region id.
A handoff key needs neighbours and an area, and runs with the PostgreSQL probe and handoff ledgers on the host's connection slot. Without these flags behaviour is as before (in-memory markers).
-
Probe tool:
grantsaccepts--regionseveral times (COUNT per region, serials interleaved in order);keysmakes handoff key pairs too;runprintshandoffandredirectlines.
-
Evidence:
client-remote-session: the GUI session crosses into region 2 with the ticket (no grant used, connected, marker past the border, probe owned there); a second client with a newer bundle logging in at region 1 is redirected and takes over at region 2. It passed 3 times in a row on Linux and also passes on Windows.- The test fixture
tests/fixtures/fake_handoff_world.hpp(the v12 state machine) is shared by both handoff tests.
M10.8d two-process fault matrix (2026-10-06)
Real evo-server processes on PostgreSQL found two recovery gaps. Both are fixed within this ADR's decisions, with no change to the protocol or the ticket format:
- A source restarted after a prepare left the probe frozen for good. The abort deadline
for a client that never arrives was kept only in the source process's memory. Schema v14
adds
overdue_probe_transfers: read-only, under the lease, it lists the caller's open outgoing transfers prepared more than a given age ago by the database clock. Everyoverdue_poll(5 s), sessions abort those older than the ticket lifetime plus 10 s, whichever process prepared them. The in-memory deadline (lifetime plus 5 s) stays as the fast path. Abort is serialized with the destination's stage and commit, so a transfer committed meanwhile stays committed: clock skew can only delay an abort. - A ticket replayed at a restarted destination could take control back. The
destination's used-ticket list is in memory. After its restart, a ticket for a transfer
committed earlier (still within its lifetime) was admitted durably even after the probe
had moved on: it moved the account's control away from the live owner and opened a
phantom probe. Ownership in the database stayed unique.
PostgresHandoffLedger::admitnow admits a committed transfer only while the record is as that commit left it: owned here, at the transfer's generation plus 1, and not frozen. Otherwise it refuses (unauthenticated). An undecided retry right after the commit still converges.
Evidence: tests/region_handoff_process_tests.cpp (in evo-postgres-tests, regions 7/161
and 7/162 with adjacent areas, handoff keys and each other as neighbour), with these
scenarios:
- A crossing and a crossing back (generation 1, 2, 3). A stale grant login at the old
region is redirected (
moved). - Destination SIGKILL after stage (staged under its lease, then killed). The restarted destination stages again under its new epoch and commits on the ticket. After the probe crosses back, a replay of that ticket at a destination process that never saw it is refused, and control stays with the owner.
- Destination SIGKILL after commit (client admitted). A retry of the same ticket at the
restarted destination converges: same commit, and control moves again. A duplicate in
the same process is refused locally (
stale_grant). - Source SIGKILL after prepare. One client's ticket still commits at the destination. The
other client never arrives, and its probe is refused as frozen (
capacity) at the restarted source until the recovery sweep aborts it. Then it logs in where it stopped. - Destination down. The ticket cannot be used, and the live source aborts after the
ticket's lifetime. The late ticket at the restarted destination is refused (
expired).
After every scenario, SQL checks that:
- each probe has one owning region (A or B);
- the inventory total is conserved, in records and in transfers;
- each record is frozen exactly while its single open transfer exists.
Library tests add the v14 call (age bounds, source only, decided transfers excluded), the refused replay in the ledger test, and the sweep over the fake world (never before the store reports a transfer overdue).