Complete M7.7 recovery verification before readiness
ADR-0065: before any readiness, recovery now verifies the head's tail. The trace object must exist with its manifest digest, its header must bind profile, region, predecessor and range; the predecessor (schema v6 load_region_commit) must hash to the digest the head names; replaying the trace from the predecessor's verified payload must reproduce every report and the exact head bytes; and trace-derived decisions must equal the head's durable decisions. Missing, corrupt or non-reproducing tails quarantine (RecoveryError::tail); nothing older or empty is served. Test-only crash seams (HostConfig::crash_hook, PublishHooks) and evo-server --crash-at deliver real SIGKILL at before-upload, before-transaction, after-commit and after-ack. The kill matrix, a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as the durability contract requires; successors verify before readiness. Validated: PostgreSQL suites under GCC Release, clang-tidy and ASan/UBSan; tail-skip and replay-equality mutations each caught. No Docker/Gitea CI, production-lease kill test or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2339,3 +2339,45 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
- Not executed: Docker/Gitea CI, TSan (the host is single-threaded; libpq is
|
||||
uninstrumented), remote PostgreSQL/TLS, a production 20 s lease kill test,
|
||||
durable client deduplication, and a network gateway.
|
||||
|
||||
## M7.7 recovery before readiness - 2026-09-28
|
||||
|
||||
- WSL Linux: GCC14 Release, clang19 analysis (clang-tidy, clean on the first
|
||||
build) and clang19 ASan/UBSan builds of `evo-postgres-tests` and `evo-server`
|
||||
passed the full PostgreSQL suite, including the M7.7 recovery tests. Runs:
|
||||
Release 2, analysis 1, ASan 2. The repository format check passed after
|
||||
reordering one fixture include.
|
||||
- Windows was not rerun. This change touches only Linux persistence and host
|
||||
code, SQL, tests and docs; M7.6's Windows run (58/58) covers the portable
|
||||
targets, which are unchanged.
|
||||
- Kill matrix (real SIGKILL via `--crash-at`, inputs 1..40 with a death at tick
|
||||
20, 5 s flush):
|
||||
|
||||
| Crash point | Head (seq/tick) | Acked up to | Decisions | Objects |
|
||||
|---|---|---|---|---|
|
||||
| before-transaction:1 | 0/0 | none | 0 | 3 (verified orphans) |
|
||||
| before-upload:2 | 1/20 | 20 | 1 | 3 |
|
||||
| before-transaction:2 | 1/20 | 20 | 1 | 5 (orphans) |
|
||||
| after-commit:2 | 2/40 | 20 (lost acks are the safe direction) | 1 | 5 |
|
||||
| after-ack:2 | 2/40 | 40 | 1 | 5 |
|
||||
|
||||
The lease stayed held after every kill. Each successor became ready at epoch 2
|
||||
from the committed head (with tail verification), then drained and released.
|
||||
- Resumed stale owner: the owner was SIGSTOPped for 4.5 s (4 s lease). The
|
||||
successor became ready at epoch 2 with committed=10 and processed ticks 11–20.
|
||||
After SIGCONT the old owner printed `ready 0`, rejected new inputs as
|
||||
`not_ready`, acknowledged nothing beyond tick 10, and exited 0 on SIGTERM
|
||||
without touching the successor's chain.
|
||||
- Damaged heads: a flipped head payload byte and a missing head trace each made
|
||||
`evo-server` exit 4 (quarantined) without ever printing `ready 1`, and the head
|
||||
was unchanged. After repair a successor became ready at epoch 4.
|
||||
- Tail verification (in-process): a missing or corrupt head trace and a missing
|
||||
predecessor payload are `tail`, and recovery succeeds again after repair. A
|
||||
published trace of different inputs over the same range (a host bypass the
|
||||
database cannot detect) is `tail`.
|
||||
- Mutations (restored and rerun green): skipping tail verification made the
|
||||
quarantine test time out because the server became ready. Dropping replay
|
||||
byte-equality failed the forged-trace test (line 262).
|
||||
- Not executed: Docker/Gitea CI, the production 20 s lease, host loss or VM
|
||||
freeze, multi-host network partitions, or verification of links older than
|
||||
the head.
|
||||
|
||||
Reference in New Issue
Block a user