Files
aevolution/docs/adr/0065-recovery-before-readiness.md
T
2026-10-08 12:08:53 -04:00

3.8 KiB

[Reviewed]

ADR-0065: Recovery verification before readiness

Status

Accepted for M7.7, 2026-09-28. It extends ADR-0061 (recovery), ADR-0063 (batch evidence) and ADR-0064 (host). Clone, migration and durability qualification are M7.9.

Context

The M7.7 criterion is: load a verified committed checkpoint, verify the tail, hash, inventories and history, and hold valid current authority before serving. Kill tests must cover every publication and acknowledgement boundary; a corrupt or missing tail must quarantine; there must be no empty fallback world.

The full-checkpoint profile (ADR-0058) has no replay tail beyond the head. The head's own batch evidence (trace, decisions) was published in M7.5, but recovery did not re-verify it. The crash matrix rows were exercised in process, but not by killing a real process at each boundary.

Decision

Tail verification. Recovery, which the host completes before readiness under a freshly acquired lease, now also verifies the head's batch evidence after the manifest, build, payload and decision checks. For any head after genesis:

  1. The manifest must be v2.
  2. The trace object must exist with its manifest length and SHA-256, and its header must bind the profile, region, predecessor commit and digest, and the tick range.
  3. The predecessor is loaded by sequence through the new read-only evolution.load_region_commit (schema v6; runtime function set is eleven). Its manifest must hash to the digest the head names.
  4. The predecessor's payload is fetched, verified and restored. Replaying the trace must reproduce every report and the head payload bytes exactly.
  5. The decisions derived from the trace must equal the head commit's durable decisions.

A missing, corrupt or non-reproducing trace, or a missing predecessor payload, is RecoveryError::tail; a decision mismatch is history. Either quarantines the region: no older head, genesis or empty world is ever served.

The replay equality is the inventory check. It proves the head state follows, under the pinned rules, from a predecessor that was itself verified when it was published. The envelope's own restore audit covers internal totals. Recovery cost grows by one predecessor restore and at most 64 replayed ticks.

Kill boundaries. HostConfig::crash_hook and PublishHooks are test-only seams at three points: before upload, before the transaction (after verified uploads), and after commit (before any acknowledgement). evo-server --crash-at <point>:<n> delivers a real SIGKILL there, or right after a step's acknowledgements are flushed (after-ack).

Alternatives Considered

  • Verifying the whole chain on every recovery: unbounded cost. Earlier links were verified when published, and the chain digests bind them.
  • Skipping trace verification: leaves a published but unreproducible head servable.
  • Kill points chosen by sleeps and signals: nondeterministic; hooks fire exactly at the boundary.

Benefits

  • Real killed processes at every boundary show the contract end to end.
  • A resumed stale owner cannot serve.
  • Damaged heads and tails never become ready.

Costs

  • Recovery reads two payloads and one trace, and replays up to 64 ticks.
  • Predecessor objects must be retained (there is no GC).
  • The crash seams exist in production binaries but are inert unless configured.

Risks

  • Tail verification covers the head link only; a damaged older link is detected only when it becomes a predecessor, or by a future audit.
  • The kill tests use the development 4 s lease on a single host with a local database.
  • VM freezes and host loss remain M8.

Consequences

  • M7.8 exposes readiness and quarantine to probes and metrics.
  • M7.9 measures recovery latency with the production lease, and qualifies clones and migrations against this verification.