158 lines
8.0 KiB
Markdown
158 lines
8.0 KiB
Markdown
[Reviewed]
|
|
# ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion
|
|
|
|
## Status
|
|
Accepted for M8.8, 2026-09-30. It builds on:
|
|
- ADR-0068 (upgrade barrier);
|
|
- ADR-0069 (World0 protection);
|
|
- ADR-0073 (fault drills);
|
|
- ADR-0074 (backup bundles).
|
|
|
|
It resolves founding open question 38 (DR objectives) for dev and stage. For
|
|
World0, it records why launch stays blocked.
|
|
|
|
## Context
|
|
M8.8 asks for three things:
|
|
- ratify the measured RPO and RTO, and host-loss causal durability, before any
|
|
irreversible acknowledgement;
|
|
- an independent durable chain, or else keep launch blocked;
|
|
- an explicit promotion of a reviewed digest and state.
|
|
|
|
The named risk is an old backup that resurrects dead organisms or duplicates
|
|
resources.
|
|
|
|
The facts:
|
|
- **One host.** Every AEvolution component runs on one physical host: the four
|
|
cluster VMs and the PostgreSQL VM pg-forge-01 (INVENTORY.md). A replica on that
|
|
host does not survive the host's loss (PERSISTENCE_ARCHITECTURE.md).
|
|
- **Acknowledgement.** The host acknowledges an outcome once it is committed in
|
|
that one database, with its payload on that one host's volume (ADR-0061/0064).
|
|
- **Backups.** Bundles (ADR-0074) are copies taken at a point in time. A restore
|
|
undoes every acknowledgement made after the bundle's cut.
|
|
|
|
## Decision
|
|
**1. Ratified objectives for dev and stage.** They rest on these measurements:
|
|
|
|
| Failure | Objective | Evidence |
|
|
|---|---|---|
|
|
| Worker/pod loss, stores healthy | RPO 0 for acknowledged outcomes; RTO ≤ 60 s | dev drill: 3/3 kills lost no acknowledgement; RTO 20.1, 19.8 and 20.2 s (lease-bound). M7.9 locally: 19.6 s |
|
|
| Object volume full | Nothing acknowledged while full; lossless resume | dev drill; retries now back off (M8.6 follow-up) |
|
|
| Object volume empty or wrong | Never ready, never a fresh world; lossless resume | dev drill (quarantine evidence re-run pending) |
|
|
| Database outage | Not ready, nothing acknowledged; a newer epoch after the outage | Local tests only. The dev drill did not partition the database. Stays a proposal until a valid dev run |
|
|
| Host loss | RPO = age of the last verified bundle; every later acknowledgement is rewound, explicitly | Local rehearsal (below). There is no schedule yet, so the dev/stage host-loss RPO is currently unbounded |
|
|
|
|
The 60 s RTO is ratified for dev and stage only. The OPERATIONS.md proposals for
|
|
World0 (24 h host-loss RTO, 15-minute RPO) are **rejected as launch criteria**. A
|
|
nonzero RPO for acknowledged deaths, births and ancestry contradicts permadeath.
|
|
|
|
**2. World0 launch is blocked.** World0 must not run a region host or acknowledge
|
|
anything until acknowledged outcomes survive host loss. The concrete requirement
|
|
is that, before acknowledging, the host has the complete causal recovery chain
|
|
durable in a second, independent failure domain:
|
|
- the commit row, its decisions and its outbox rows, for example through
|
|
synchronous PostgreSQL replication (`synchronous_commit = remote_apply`) to a
|
|
standby on other hardware;
|
|
- every payload and trace object the commit references, verified on storage in
|
|
that same second failure domain.
|
|
|
|
A host-loss drill must then show that no acknowledged outcome is rewound. None of
|
|
this hardware exists. The policy check enforces the block:
|
|
- no workload may target World0;
|
|
- the World0 Application keeps an unresolved path;
|
|
- no `region/world0` or `stores/world0` overlay may exist.
|
|
|
|
Lifting the block is the launch decision. It needs its own accepted ADR with that
|
|
evidence. Dev and stage proceed, as the roadmap allows.
|
|
|
|
**3. Host-loss recovery for dev and stage (restore in place).**
|
|
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
|
|
bundle under the world's own identity into a replacement database:
|
|
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
|
|
- It fences every restored lease, so the next owner acquires a newer epoch. The
|
|
world stays unservable (every build retired) until verification publishes it
|
|
(ADR-0074 amendment).
|
|
- It re-runs the migration rights audit for the environment's runtime role.
|
|
- It prints a `REWIND` line per region naming the tick it was restored to.
|
|
|
|
`--accept-rewind` is required: it states that undoing later acknowledgements is
|
|
acceptable, which it is only for disposable environments. The restored world does
|
|
**not** resurrect anything silently:
|
|
- the rewind is explicit and measured;
|
|
- history up to the cut is audited;
|
|
- the synthetic driver (or a future gateway) resumes after the cut, so a death at
|
|
or before the cut is never repeated.
|
|
|
|
**4. Fencing the old environment.** `fence-database.sh --superseded-by LABEL
|
|
--confirm`, run as owner on the old database, uses the ADR-0068 barrier:
|
|
- it retains the supersession identity SHA-256("AEVOLUTION-SUPERSEDED-v1"), which
|
|
no image digest can equal;
|
|
- it retires every other build in one transaction.
|
|
|
|
The effects:
|
|
- every lease is fenced;
|
|
- a running old host's next lease check fails and its re-acquisition is refused
|
|
(unsupported build), so it quarantines (exit 4);
|
|
- no later host can acquire;
|
|
- a commit naming a retired build is refused by the trigger.
|
|
|
|
The script prints the old environment's final heads, which measure the rewind.
|
|
Run it before any old host can return. If the old database is gone, run it the
|
|
moment it is reachable, and keep its credentials revoked until then. No schema
|
|
change is needed: supersession is "no current build except one that no image can
|
|
be".
|
|
|
|
**5. Promotion.** Nothing reaches World0 except by promotion:
|
|
- A World0 overlay may pin only the digests that the stage overlay pins for the
|
|
same image. The policy check enforces this. The digest is promoted after stage
|
|
qualification, never rebuilt.
|
|
- World0's opening is a reviewed world-design decision. The provisioning tool
|
|
accepts only `development-v1`, which may never provision World0.
|
|
- A promotion is one reviewed change that pins the digests and names:
|
|
- the stage evidence for that exact digest (drills, a verified backup);
|
|
- the reviewed opening;
|
|
- the accepted ADR that lifts the gate.
|
|
|
|
While the gate stands, promotion is defined and its policy is tested, but it
|
|
cannot happen.
|
|
|
|
## Evidence
|
|
`tools/ci/dr-check.sh` runs in every PostgreSQL build:
|
|
1. An old host serves world 400 past the tick-20 death. An online backup cuts at
|
|
tick 40, and the host goes on acknowledging, to 110.
|
|
2. The world is restored in place into a replacement database, while the old host
|
|
keeps running (a partition, the worst case).
|
|
3. A new host resumes at exactly `epoch=2 committed=40` and acknowledges from
|
|
tick 41.
|
|
4. The old database is fenced. The still-running old host drops its
|
|
unacknowledged inputs as `fenced` and quarantines (exit 4), and the old head
|
|
never advances again. A host started later on the old database quarantines at
|
|
once.
|
|
5. The restored world's full history audit passes: deaths, the founder and
|
|
descendant lineage, inventory conservation and ancestry. The death is durable
|
|
exactly once and never repeated.
|
|
6. The rewind is measured: 70 acknowledged ticks were undone. That number is the
|
|
reason World0 stays blocked.
|
|
|
|
The deploy policy check is tested negatively: a World0 workload, a resolved World0
|
|
Application path, and a World0 overlay digest not pinned in stage are each
|
|
rejected.
|
|
|
|
## Consequences
|
|
- AEvolution can run dev and stage, and qualify releases there. It cannot launch
|
|
World0 on this hardware.
|
|
- Host-loss recovery for dev and stage is a deliberate, recorded rewind.
|
|
Scheduling bundles (backlog) bounds its RPO.
|
|
- M8.9 (upgrade and growth) can proceed in dev and stage.
|
|
|
|
## Alternatives rejected
|
|
- **Launch World0 with a backup RPO.** Old backups would undo acknowledged deaths
|
|
and births. The founding brief forbids this, and a silent RPO change is not a
|
|
workaround.
|
|
- **Replicas on the same host** (a second PostgreSQL VM or a RAID mirror). These
|
|
are not an independent failure domain.
|
|
- **A schema-level "superseded" flag.** It would add a migration and a new error
|
|
path to the host. The existing barrier already gives the same fence, serialized
|
|
with every acquisition.
|
|
- **Restore in place without `--accept-rewind`.** It would make the rewind
|
|
implicit, which is exactly the named risk.
|