Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped

ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.

The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
  abort deadline was in memory): schema v14 overdue_probe_transfers and a
  session-layer sweep abort open outgoing transfers past the ticket
  lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
  took control back: admission of a committed transfer now requires the
  record as that commit left it.

M10.8 is complete. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-06 23:17:49 -04:00
co-authored by Claude Opus 5.5
parent dabeee8e8b
commit 265163a9a6
26 changed files with 856 additions and 20 deletions
+46
View File
@@ -4858,3 +4858,49 @@ Not done:
- real two-process handoffs on PostgreSQL with kills (10.8d);
- a GUI run across regions on evolution-dev (one dev region only; needs v4 and v13
deployed).
## 2026-10-06 — M10.8d two-process handoff fault matrix (ADR-0096)
Implemented:
- `tests/region_handoff_process_tests.cpp`: two evo-server processes (regions 7/161 and
7/162, adjacent `--m10-area`s, handoff keys and each other as `--m10-neighbour`, fixed
loopback ports) with in-process session clients that follow offers and tickets. It covers:
- a crossing and a crossing back, plus a redirected stale login;
- destination SIGKILL after stage, then a re-stage and commit under the new epoch;
- destination SIGKILL after commit, then a converging ticket retry;
- a refused ticket replay after the probe moved on;
- a refused same-process duplicate;
- source SIGKILL after prepare: one client still arrives, the other never does and is
aborted by the restarted source's sweep;
- a destination that is down, with the live source aborting and the late ticket expired.
After every scenario, SQL checks one owner per probe, the conserved inventory, and
frozen-iff-open-transfer.
- Schema v14 `overdue_probe_transfers` (apply chain, audit allowlist, upgrade and drift
checks), `ProbeLedger::overdue`, and the session layer's recovery sweep
(`overdue_poll`, ticket lifetime plus 10 s).
- The replay guard in `PostgresHandoffLedger::admit`: a committed transfer admits only
while the record is still owned here at generation plus 1 and not frozen.
- Library tests:
- the v14 store call (age bounds, source only, aborted transfers excluded);
- the refused retry in the ledger test;
- the sweep over the fake world in `two_region_handoff`.
Findings: the two recovery gaps above were found by this matrix, before any deployment of
v4/v12/v13.
Executed:
- **Linux GCC 14:** `postgres-check.sh` with QUIC, all suites including the new one (restart
waits 0.03-4 s; the replay ran at 8 s of the 30 s ticket lifetime). It passed 3 times in a
row, plus a final run after a tidy-only change. Presets `quic` 4/4 and `client-network`
16/16.
- **Linux Clang 19:** clang-tidy 19 on `region_sessions.cpp`, `postgres_handoff_ledger.cpp`,
`region_lease_store.cpp`, the QUIC session, remote client, probe store, process,
migration and Postgres runner tests (one finding fixed); clang-format 19.
- **Windows MSVC 19.44:** presets `quic` 4/4 and `client-network` 14/14. The PostgreSQL
process tests are Linux-only.
Not done:
- deployment of protocol v4 and schemas v12-v14 (needs a rebuilt migrations image; not done
from routine development);
- a GUI run across two regions on evolution-dev (only one dev region exists).