Files
aevolution/docs/adr/0075-world0-acknowledgment-gate.md
T
dingoandClaude Opus 5.5 68da5654ff fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)
The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 06:14:34 -04:00

8.0 KiB

ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion

Status

Accepted for M8.8, 2026-09-30. It builds on:

  • ADR-0068 (upgrade barrier);
  • ADR-0069 (World0 protection);
  • ADR-0073 (fault drills);
  • ADR-0074 (backup bundles).

It resolves founding open question 38 (DR objectives) for dev and stage. For World0, it records why launch stays blocked.

Context

M8.8 asks for three things:

  • ratify the measured RPO and RTO, and host-loss causal durability, before any irreversible acknowledgement;
  • an independent durable chain, or else keep launch blocked;
  • an explicit promotion of a reviewed digest and state.

The named risk is an old backup that resurrects dead organisms or duplicates resources.

The facts:

  • One host. Every AEvolution component runs on one physical host: the four cluster VMs and the PostgreSQL VM pg-forge-01 (INVENTORY.md). A replica on that host does not survive the host's loss (PERSISTENCE_ARCHITECTURE.md).
  • Acknowledgement. The host acknowledges an outcome once it is committed in that one database, with its payload on that one host's volume (ADR-0061/0064).
  • Backups. Bundles (ADR-0074) are copies taken at a point in time. A restore undoes every acknowledgement made after the bundle's cut.

Decision

1. Ratified objectives for dev and stage. They rest on these measurements:

Failure Objective Evidence
Worker/pod loss, stores healthy RPO 0 for acknowledged outcomes; RTO ≤ 60 s dev drill: 3/3 kills lost no acknowledgement; RTO 20.1, 19.8 and 20.2 s (lease-bound). M7.9 locally: 19.6 s
Object volume full Nothing acknowledged while full; lossless resume dev drill; retries now back off (M8.6 follow-up)
Object volume empty or wrong Never ready, never a fresh world; lossless resume dev drill (quarantine evidence re-run pending)
Database outage Not ready, nothing acknowledged; a newer epoch after the outage Local tests only. The dev drill did not partition the database. Stays a proposal until a valid dev run
Host loss RPO = age of the last verified bundle; every later acknowledgement is rewound, explicitly Local rehearsal (below). There is no schedule yet, so the dev/stage host-loss RPO is currently unbounded

The 60 s RTO is ratified for dev and stage only. The OPERATIONS.md proposals for World0 (24 h host-loss RTO, 15-minute RPO) are rejected as launch criteria. A nonzero RPO for acknowledged deaths, births and ancestry contradicts permadeath.

2. World0 launch is blocked. World0 must not run a region host or acknowledge anything until acknowledged outcomes survive host loss. The concrete requirement is that, before acknowledging, the host has the complete causal recovery chain durable in a second, independent failure domain:

  • the commit row, its decisions and its outbox rows, for example through synchronous PostgreSQL replication (synchronous_commit = remote_apply) to a standby on other hardware;
  • every payload and trace object the commit references, verified on storage in that same second failure domain.

A host-loss drill must then show that no acknowledged outcome is rewound. None of this hardware exists. The policy check enforces the block:

  • no workload may target World0;
  • the World0 Application keeps an unresolved path;
  • no region/world0 or stores/world0 overlay may exist.

Lifting the block is the launch decision. It needs its own accepted ADR with that evidence. Dev and stage proceed, as the roadmap allows.

3. Host-loss recovery for dev and stage (restore in place). restore-world.sh --in-place --runtime-role ROLE --accept-rewind restores a bundle under the world's own identity into a replacement database:

  • It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
  • It fences every restored lease, so the next owner acquires a newer epoch. The world stays unservable (every build retired) until verification publishes it (ADR-0074 amendment).
  • It re-runs the migration rights audit for the environment's runtime role.
  • It prints a REWIND line per region naming the tick it was restored to.

--accept-rewind is required: it states that undoing later acknowledgements is acceptable, which it is only for disposable environments. The restored world does not resurrect anything silently:

  • the rewind is explicit and measured;
  • history up to the cut is audited;
  • the synthetic driver (or a future gateway) resumes after the cut, so a death at or before the cut is never repeated.

4. Fencing the old environment. fence-database.sh --superseded-by LABEL --confirm, run as owner on the old database, uses the ADR-0068 barrier:

  • it retains the supersession identity SHA-256("AEVOLUTION-SUPERSEDED-v1"), which no image digest can equal;
  • it retires every other build in one transaction.

The effects:

  • every lease is fenced;
  • a running old host's next lease check fails and its re-acquisition is refused (unsupported build), so it quarantines (exit 4);
  • no later host can acquire;
  • a commit naming a retired build is refused by the trigger.

The script prints the old environment's final heads, which measure the rewind. Run it before any old host can return. If the old database is gone, run it the moment it is reachable, and keep its credentials revoked until then. No schema change is needed: supersession is "no current build except one that no image can be".

5. Promotion. Nothing reaches World0 except by promotion:

  • A World0 overlay may pin only the digests that the stage overlay pins for the same image. The policy check enforces this. The digest is promoted after stage qualification, never rebuilt.
  • World0's opening is a reviewed world-design decision. The provisioning tool accepts only development-v1, which may never provision World0.
  • A promotion is one reviewed change that pins the digests and names:
    • the stage evidence for that exact digest (drills, a verified backup);
    • the reviewed opening;
    • the accepted ADR that lifts the gate.

While the gate stands, promotion is defined and its policy is tested, but it cannot happen.

Evidence

tools/ci/dr-check.sh runs in every PostgreSQL build:

  1. An old host serves world 400 past the tick-20 death. An online backup cuts at tick 40, and the host goes on acknowledging, to 110.
  2. The world is restored in place into a replacement database, while the old host keeps running (a partition, the worst case).
  3. A new host resumes at exactly epoch=2 committed=40 and acknowledges from tick 41.
  4. The old database is fenced. The still-running old host drops its unacknowledged inputs as fenced and quarantines (exit 4), and the old head never advances again. A host started later on the old database quarantines at once.
  5. The restored world's full history audit passes: deaths, the founder and descendant lineage, inventory conservation and ancestry. The death is durable exactly once and never repeated.
  6. The rewind is measured: 70 acknowledged ticks were undone. That number is the reason World0 stays blocked.

The deploy policy check is tested negatively: a World0 workload, a resolved World0 Application path, and a World0 overlay digest not pinned in stage are each rejected.

Consequences

  • AEvolution can run dev and stage, and qualify releases there. It cannot launch World0 on this hardware.
  • Host-loss recovery for dev and stage is a deliberate, recorded rewind. Scheduling bundles (backlog) bounds its RPO.
  • M8.9 (upgrade and growth) can proceed in dev and stage.

Alternatives rejected

  • Launch World0 with a backup RPO. Old backups would undo acknowledged deaths and births. The founding brief forbids this, and a silent RPO change is not a workaround.
  • Replicas on the same host (a second PostgreSQL VM or a RAID mirror). These are not an independent failure domain.
  • A schema-level "superseded" flag. It would add a migration and a new error path to the host. The existing barrier already gives the same fence, serialized with every acquisition.
  • Restore in place without --accept-rewind. It would make the rewind implicit, which is exactly the named risk.