Complete M7.7 recovery verification before readiness

ADR-0065: before any readiness, recovery now verifies the head's tail.
The trace object must exist with its manifest digest, its header must
bind profile, region, predecessor and range; the predecessor (schema v6
load_region_commit) must hash to the digest the head names; replaying
the trace from the predecessor's verified payload must reproduce every
report and the exact head bytes; and trace-derived decisions must equal
the head's durable decisions. Missing, corrupt or non-reproducing tails
quarantine (RecoveryError::tail); nothing older or empty is served.

Test-only crash seams (HostConfig::crash_hook, PublishHooks) and
evo-server --crash-at deliver real SIGKILL at before-upload,
before-transaction, after-commit and after-ack. The kill matrix,
a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as
the durability contract requires; successors verify before readiness.

Validated: PostgreSQL suites under GCC Release, clang-tidy and
ASan/UBSan; tail-skip and replay-equality mutations each caught. No
Docker/Gitea CI, production-lease kill test or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 22:31:40 -04:00
co-authored by Claude Opus 5.5
parent d6d0408aae
commit 30411a83f9
24 changed files with 990 additions and 284 deletions
+42
View File
@@ -2339,3 +2339,45 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
- Not executed: Docker/Gitea CI, TSan (the host is single-threaded; libpq is
uninstrumented), remote PostgreSQL/TLS, a production 20 s lease kill test,
durable client deduplication, and a network gateway.
## M7.7 recovery before readiness - 2026-09-28
- WSL Linux: GCC14 Release, clang19 analysis (clang-tidy, clean on the first
build) and clang19 ASan/UBSan builds of `evo-postgres-tests` and `evo-server`
passed the full PostgreSQL suite, including the M7.7 recovery tests. Runs:
Release 2, analysis 1, ASan 2. The repository format check passed after
reordering one fixture include.
- Windows was not rerun. This change touches only Linux persistence and host
code, SQL, tests and docs; M7.6's Windows run (58/58) covers the portable
targets, which are unchanged.
- Kill matrix (real SIGKILL via `--crash-at`, inputs 1..40 with a death at tick
20, 5 s flush):
| Crash point | Head (seq/tick) | Acked up to | Decisions | Objects |
|---|---|---|---|---|
| before-transaction:1 | 0/0 | none | 0 | 3 (verified orphans) |
| before-upload:2 | 1/20 | 20 | 1 | 3 |
| before-transaction:2 | 1/20 | 20 | 1 | 5 (orphans) |
| after-commit:2 | 2/40 | 20 (lost acks are the safe direction) | 1 | 5 |
| after-ack:2 | 2/40 | 40 | 1 | 5 |
The lease stayed held after every kill. Each successor became ready at epoch 2
from the committed head (with tail verification), then drained and released.
- Resumed stale owner: the owner was SIGSTOPped for 4.5 s (4 s lease). The
successor became ready at epoch 2 with committed=10 and processed ticks 11–20.
After SIGCONT the old owner printed `ready 0`, rejected new inputs as
`not_ready`, acknowledged nothing beyond tick 10, and exited 0 on SIGTERM
without touching the successor's chain.
- Damaged heads: a flipped head payload byte and a missing head trace each made
`evo-server` exit 4 (quarantined) without ever printing `ready 1`, and the head
was unchanged. After repair a successor became ready at epoch 4.
- Tail verification (in-process): a missing or corrupt head trace and a missing
predecessor payload are `tail`, and recovery succeeds again after repair. A
published trace of different inputs over the same range (a host bypass the
database cannot detect) is `tail`.
- Mutations (restored and rerun green): skipping tail verification made the
quarantine test time out because the server became ready. Dropping replay
byte-equality failed the forged-trace test (line 262).
- Not executed: Docker/Gitea CI, the production 20 s lease, host loss or VM
freeze, multi-host network partitions, or verification of links older than
the head.