Files
aevolution/docs/adr/0075-world0-acknowledgment-gate.md
T
dingoandClaude Opus 5.5 68da5654ff fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)
The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 06:14:34 -04:00

157 lines
8.0 KiB
Markdown

# ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion
## Status
Accepted for M8.8, 2026-09-30. It builds on:
- ADR-0068 (upgrade barrier);
- ADR-0069 (World0 protection);
- ADR-0073 (fault drills);
- ADR-0074 (backup bundles).
It resolves founding open question 38 (DR objectives) for dev and stage. For
World0, it records why launch stays blocked.
## Context
M8.8 asks for three things:
- ratify the measured RPO and RTO, and host-loss causal durability, before any
irreversible acknowledgement;
- an independent durable chain, or else keep launch blocked;
- an explicit promotion of a reviewed digest and state.
The named risk is an old backup that resurrects dead organisms or duplicates
resources.
The facts:
- **One host.** Every AEvolution component runs on one physical host: the four
cluster VMs and the PostgreSQL VM pg-forge-01 (INVENTORY.md). A replica on that
host does not survive the host's loss (PERSISTENCE_ARCHITECTURE.md).
- **Acknowledgement.** The host acknowledges an outcome once it is committed in
that one database, with its payload on that one host's volume (ADR-0061/0064).
- **Backups.** Bundles (ADR-0074) are copies taken at a point in time. A restore
undoes every acknowledgement made after the bundle's cut.
## Decision
**1. Ratified objectives for dev and stage.** They rest on these measurements:
| Failure | Objective | Evidence |
|---|---|---|
| Worker/pod loss, stores healthy | RPO 0 for acknowledged outcomes; RTO ≤ 60 s | dev drill: 3/3 kills lost no acknowledgement; RTO 20.1, 19.8 and 20.2 s (lease-bound). M7.9 locally: 19.6 s |
| Object volume full | Nothing acknowledged while full; lossless resume | dev drill; retries now back off (M8.6 follow-up) |
| Object volume empty or wrong | Never ready, never a fresh world; lossless resume | dev drill (quarantine evidence re-run pending) |
| Database outage | Not ready, nothing acknowledged; a newer epoch after the outage | Local tests only. The dev drill did not partition the database. Stays a proposal until a valid dev run |
| Host loss | RPO = age of the last verified bundle; every later acknowledgement is rewound, explicitly | Local rehearsal (below). There is no schedule yet, so the dev/stage host-loss RPO is currently unbounded |
The 60 s RTO is ratified for dev and stage only. The OPERATIONS.md proposals for
World0 (24 h host-loss RTO, 15-minute RPO) are **rejected as launch criteria**. A
nonzero RPO for acknowledged deaths, births and ancestry contradicts permadeath.
**2. World0 launch is blocked.** World0 must not run a region host or acknowledge
anything until acknowledged outcomes survive host loss. The concrete requirement
is that, before acknowledging, the host has the complete causal recovery chain
durable in a second, independent failure domain:
- the commit row, its decisions and its outbox rows, for example through
synchronous PostgreSQL replication (`synchronous_commit = remote_apply`) to a
standby on other hardware;
- every payload and trace object the commit references, verified on storage in
that same second failure domain.
A host-loss drill must then show that no acknowledged outcome is rewound. None of
this hardware exists. The policy check enforces the block:
- no workload may target World0;
- the World0 Application keeps an unresolved path;
- no `region/world0` or `stores/world0` overlay may exist.
Lifting the block is the launch decision. It needs its own accepted ADR with that
evidence. Dev and stage proceed, as the roadmap allows.
**3. Host-loss recovery for dev and stage (restore in place).**
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
bundle under the world's own identity into a replacement database:
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
- It fences every restored lease, so the next owner acquires a newer epoch. The
world stays unservable (every build retired) until verification publishes it
(ADR-0074 amendment).
- It re-runs the migration rights audit for the environment's runtime role.
- It prints a `REWIND` line per region naming the tick it was restored to.
`--accept-rewind` is required: it states that undoing later acknowledgements is
acceptable, which it is only for disposable environments. The restored world does
**not** resurrect anything silently:
- the rewind is explicit and measured;
- history up to the cut is audited;
- the synthetic driver (or a future gateway) resumes after the cut, so a death at
or before the cut is never repeated.
**4. Fencing the old environment.** `fence-database.sh --superseded-by LABEL
--confirm`, run as owner on the old database, uses the ADR-0068 barrier:
- it retains the supersession identity SHA-256("AEVOLUTION-SUPERSEDED-v1"), which
no image digest can equal;
- it retires every other build in one transaction.
The effects:
- every lease is fenced;
- a running old host's next lease check fails and its re-acquisition is refused
(unsupported build), so it quarantines (exit 4);
- no later host can acquire;
- a commit naming a retired build is refused by the trigger.
The script prints the old environment's final heads, which measure the rewind.
Run it before any old host can return. If the old database is gone, run it the
moment it is reachable, and keep its credentials revoked until then. No schema
change is needed: supersession is "no current build except one that no image can
be".
**5. Promotion.** Nothing reaches World0 except by promotion:
- A World0 overlay may pin only the digests that the stage overlay pins for the
same image. The policy check enforces this. The digest is promoted after stage
qualification, never rebuilt.
- World0's opening is a reviewed world-design decision. The provisioning tool
accepts only `development-v1`, which may never provision World0.
- A promotion is one reviewed change that pins the digests and names:
- the stage evidence for that exact digest (drills, a verified backup);
- the reviewed opening;
- the accepted ADR that lifts the gate.
While the gate stands, promotion is defined and its policy is tested, but it
cannot happen.
## Evidence
`tools/ci/dr-check.sh` runs in every PostgreSQL build:
1. An old host serves world 400 past the tick-20 death. An online backup cuts at
tick 40, and the host goes on acknowledging, to 110.
2. The world is restored in place into a replacement database, while the old host
keeps running (a partition, the worst case).
3. A new host resumes at exactly `epoch=2 committed=40` and acknowledges from
tick 41.
4. The old database is fenced. The still-running old host drops its
unacknowledged inputs as `fenced` and quarantines (exit 4), and the old head
never advances again. A host started later on the old database quarantines at
once.
5. The restored world's full history audit passes: deaths, the founder and
descendant lineage, inventory conservation and ancestry. The death is durable
exactly once and never repeated.
6. The rewind is measured: 70 acknowledged ticks were undone. That number is the
reason World0 stays blocked.
The deploy policy check is tested negatively: a World0 workload, a resolved World0
Application path, and a World0 overlay digest not pinned in stage are each
rejected.
## Consequences
- AEvolution can run dev and stage, and qualify releases there. It cannot launch
World0 on this hardware.
- Host-loss recovery for dev and stage is a deliberate, recorded rewind.
Scheduling bundles (backlog) bounds its RPO.
- M8.9 (upgrade and growth) can proceed in dev and stage.
## Alternatives rejected
- **Launch World0 with a backup RPO.** Old backups would undo acknowledged deaths
and births. The founding brief forbids this, and a silent RPO change is not a
workaround.
- **Replicas on the same host** (a second PostgreSQL VM or a RAID mirror). These
are not an independent failure domain.
- **A schema-level "superseded" flag.** It would add a migration and a new error
path to the host. The existing barrier already gives the same fence, serialized
with every acquisition.
- **Restore in place without `--accept-rewind`.** It would make the rewind
implicit, which is exactly the named risk.