Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and 7/162, adjacent areas) run crossings and crossings back, SIGKILL of the source after prepare and of the destination after stage and after commit, duplicate and stale ticket retries, a client that never arrives and a destination that is down. After every scenario SQL checks one owner per probe, the conserved inventory and frozen-iff-open-transfer. The matrix found two recovery gaps, now fixed: - a source restarted after a prepare left the probe frozen for good (its abort deadline was in memory): schema v14 overdue_probe_transfers and a session-layer sweep abort open outgoing transfers past the ticket lifetime plus 10 s by the database clock; - a ticket replayed at a restarted destination after the probe moved on took control back: admission of a committed transfer now requires the record as that commit left it. M10.8 is complete. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4858,3 +4858,49 @@ Not done:
|
||||
- real two-process handoffs on PostgreSQL with kills (10.8d);
|
||||
- a GUI run across regions on evolution-dev (one dev region only; needs v4 and v13
|
||||
deployed).
|
||||
|
||||
## 2026-10-06 — M10.8d two-process handoff fault matrix (ADR-0096)
|
||||
|
||||
Implemented:
|
||||
- `tests/region_handoff_process_tests.cpp`: two evo-server processes (regions 7/161 and
|
||||
7/162, adjacent `--m10-area`s, handoff keys and each other as `--m10-neighbour`, fixed
|
||||
loopback ports) with in-process session clients that follow offers and tickets. It covers:
|
||||
- a crossing and a crossing back, plus a redirected stale login;
|
||||
- destination SIGKILL after stage, then a re-stage and commit under the new epoch;
|
||||
- destination SIGKILL after commit, then a converging ticket retry;
|
||||
- a refused ticket replay after the probe moved on;
|
||||
- a refused same-process duplicate;
|
||||
- source SIGKILL after prepare: one client still arrives, the other never does and is
|
||||
aborted by the restarted source's sweep;
|
||||
- a destination that is down, with the live source aborting and the late ticket expired.
|
||||
|
||||
After every scenario, SQL checks one owner per probe, the conserved inventory, and
|
||||
frozen-iff-open-transfer.
|
||||
- Schema v14 `overdue_probe_transfers` (apply chain, audit allowlist, upgrade and drift
|
||||
checks), `ProbeLedger::overdue`, and the session layer's recovery sweep
|
||||
(`overdue_poll`, ticket lifetime plus 10 s).
|
||||
- The replay guard in `PostgresHandoffLedger::admit`: a committed transfer admits only
|
||||
while the record is still owned here at generation plus 1 and not frozen.
|
||||
- Library tests:
|
||||
- the v14 store call (age bounds, source only, aborted transfers excluded);
|
||||
- the refused retry in the ledger test;
|
||||
- the sweep over the fake world in `two_region_handoff`.
|
||||
|
||||
Findings: the two recovery gaps above were found by this matrix, before any deployment of
|
||||
v4/v12/v13.
|
||||
|
||||
Executed:
|
||||
- **Linux GCC 14:** `postgres-check.sh` with QUIC, all suites including the new one (restart
|
||||
waits 0.03-4 s; the replay ran at 8 s of the 30 s ticket lifetime). It passed 3 times in a
|
||||
row, plus a final run after a tidy-only change. Presets `quic` 4/4 and `client-network`
|
||||
16/16.
|
||||
- **Linux Clang 19:** clang-tidy 19 on `region_sessions.cpp`, `postgres_handoff_ledger.cpp`,
|
||||
`region_lease_store.cpp`, the QUIC session, remote client, probe store, process,
|
||||
migration and Postgres runner tests (one finding fixed); clang-format 19.
|
||||
- **Windows MSVC 19.44:** presets `quic` 4/4 and `client-network` 14/14. The PostgreSQL
|
||||
process tests are Linux-only.
|
||||
|
||||
Not done:
|
||||
- deployment of protocol v4 and schemas v12-v14 (needs a rebuilt migrations image; not done
|
||||
from routine development);
|
||||
- a GUI run across two regions on evolution-dev (only one dev region exists).
|
||||
|
||||
Reference in New Issue
Block a user