Complete M7.7 recovery verification before readiness

ADR-0065: before any readiness, recovery now verifies the head's tail.
The trace object must exist with its manifest digest, its header must
bind profile, region, predecessor and range; the predecessor (schema v6
load_region_commit) must hash to the digest the head names; replaying
the trace from the predecessor's verified payload must reproduce every
report and the exact head bytes; and trace-derived decisions must equal
the head's durable decisions. Missing, corrupt or non-reproducing tails
quarantine (RecoveryError::tail); nothing older or empty is served.

Test-only crash seams (HostConfig::crash_hook, PublishHooks) and
evo-server --crash-at deliver real SIGKILL at before-upload,
before-transaction, after-commit and after-ack. The kill matrix,
a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as
the durability contract requires; successors verify before readiness.

Validated: PostgreSQL suites under GCC Release, clang-tidy and
ASan/UBSan; tail-skip and replay-equality mutations each caught. No
Docker/Gitea CI, production-lease kill test or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 22:31:40 -04:00
co-authored by Claude Opus 5.5
parent d6d0408aae
commit 30411a83f9
24 changed files with 990 additions and 284 deletions
+9 -1
View File
@@ -228,6 +228,8 @@ template <typename State> persistence::Uuid RegionHost<State>::next_commit_id()
template <typename State>
void RegionHost<State>::confirmed(const persistence::PublishedBatch& published, SchedulerTime now) {
if (config_.crash_hook)
config_.crash_hook(CrashPoint::after_commit);
const auto& head = published.committed.head;
if (!batch_ || !batch_->confirm(head.end_tick)) {
lose_authority(Unacknowledged::quarantined, HostPhase::quarantined);
@@ -255,9 +257,15 @@ template <typename State> void RegionHost<State>::commit(SchedulerTime now) {
return;
const auto commit_id = next_commit_id();
batch_->seal();
const auto& crash = config_.crash_hook;
if (crash)
crash(CrashPoint::before_upload);
persistence::PublishHooks hooks;
if (crash)
hooks.before_transaction = [&crash] { crash(CrashPoint::before_transaction); };
auto published =
Ops<State>::publish(*db_, *objects_, *grant, *committed_, *batch_, commit_id, config_.build,
config_.restore_budget, config_.lease.commit_timeout);
config_.restore_budget, config_.lease.commit_timeout, hooks);
if (published) {
confirmed(*published, now);
return;