feat(ops): M8.8 World0 acknowledgment gate and host-loss recovery (ADR-0075)

Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked
(policy-enforced) until acknowledged outcomes survive host loss in an
independent failure domain. Adds in-place restore with an explicit rewind,
barrier-based fencing of a superseded database, and promotion of stage-pinned
digests only. dr-check.sh rehearses host loss in every PostgreSQL build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 17:07:07 -04:00
co-authored by Claude Opus 5.5
parent 5512fdec54
commit 518a960864
15 changed files with 486 additions and 15 deletions
+1
View File
@@ -8,6 +8,7 @@ names it. Remove an item only when it is done, and record where it was done.
|---|---|---|---|
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
| **Backup destination for dev/stage stores.** The tooling now exists up to the destination (ADR-0074): `tools/backup/cluster-backup.sh <env> --to DIR` writes a verified bundle to any directory, and `verify-backup.sh` restore-tests it. Still to choose: the permanent off-host destination, and with it (1) bundle encryption and key custody, (2) a scheduled CronJob synced by Argo, (3) retention (proposal: 30 daily, 12 weekly, 12 monthly), (4) incremental object copies. | M8.3, 2026-09-29; M8.7, 2026-09-30 | The maintainer will supply a destination later | Scheduled backups; any World0 data; M8.8 |
| **Independent durable failure domain (World0 launch).** World0 stays blocked until acknowledged outcomes survive host loss (ADR-0075): before acknowledging, the commit rows and every referenced object must be durable on other hardware (for example a synchronous PostgreSQL standby with `remote_apply`, and object replication verified before commit), proven by a host-loss drill that rewinds nothing. Needs a second physical host and storage. | M8.8, 2026-09-30 | No second failure domain exists | World0 launch; lifting the ADR-0075 policy rule |
| **WAL archiving on pg-forge-01.** Bundles are logical backups: the recovery point is the last bundle. Continuous archiving (`archive_command` or `pg_receivewal`) plus periodic base backups would allow point-in-time recovery. It needs the backup destination. | M8.7, 2026-09-30 | Needs the destination and VM configuration | M8.8 host-loss RPO decision |
| **Harbor retention of build images.** A bundle's MANIFEST names every build (region image digest) that wrote history; recovery and replay need those images. Confirm Harbor's retention/tag-immutability policy keeps them (or pin them in a protected project). | M8.7, 2026-09-30 | Not yet checked | M8.8; any upgrade barrier that retires a build |
| **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 |
+8
View File
@@ -660,3 +660,11 @@ M8.7 implemented and locally qualified (2026-09-30), under ADR-0074:
- an operator-run in-cluster backup to a directory off the cluster.
Pending the maintainer's destination: encryption, schedule, retention and WAL
archiving (docs/BACKLOG.md). In-cluster bundle evidence waits on a run.
M8.8 decided (2026-09-30), under ADR-0075: dev/stage objectives ratified from
measurements (worker loss RPO 0 / RTO <= 60 s; host loss = explicit rewind to the
last verified bundle); **World0 launch stays blocked** because no independent
durable failure domain exists, enforced by the deploy policy check; host-loss
restore in place with barrier fencing of the superseded database rehearsed
locally; promotion only of stage-pinned digests. **M8-dev/stage passed;
M8-World0 launch not passed.**
+16
View File
@@ -1125,3 +1125,19 @@ ADR-0074 adds world backup bundles and a verified restore:
The destination, encryption, schedule, retention and WAL archiving wait on the
maintainer (docs/BACKLOG.md).
## M8.8 World0 gate and host-loss recovery (2026-09-30)
ADR-0075:
- **World0 launch is blocked.** Everything runs on one physical host, so an
acknowledged death could be undone by host loss. The deploy policy check now
rejects any World0 workload, a resolved World0 Application path, and World0
overlays; any future World0 pin must equal stage's digest.
- **Dev/stage objectives ratified:** worker loss RPO 0 and RTO <= 60 s (measured
~20 s); host loss rewinds to the last verified bundle, explicitly.
- `restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a world
under its own identity with its grants, fences restored leases, audits runtime
rights and prints the rewind.
- `fence-database.sh` supersedes the old database through the upgrade barrier: a
still-running old host quarantines, and no host can acquire it again.
- `tools/ci/dr-check.sh` rehearses it in every PostgreSQL build.
+5 -3
View File
@@ -37,7 +37,7 @@ facts make a barrier unnecessary for a single database:
| File | Content |
|---|---|
| `MANIFEST` | Format, time and label. The server version. Every schema migration and its checksum. Every retained build (image digest) and whether it is retired. Every region head (world, region, frame, sequence, tick, commit, digest, build). Row counts, and the object count and bytes |
| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners and no grants |
| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners. Grants are kept for an in-place restore (ADR-0075 amendment); every other restore skips them |
| `objects/w<world>/r<region>/<sha256>` | Every object in the store, each checked against its name while it is copied |
| `SHA256SUMS` | Checksums of every other file |
@@ -76,7 +76,7 @@ empty database owned by the restoring login, and fails closed in this order:
A failure after step 2 removes the restored schema and objects, so an unverified
restore cannot be mistaken for a world. The result is a verbatim, owner-only copy
that keeps the source's world identities. It has no runtime grants and is never
served.
served. (ADR-0075 adds `--in-place` for dev/stage host-loss recovery.)
**Isolated rebuild.** `tools/backup/verify-backup.sh BUNDLE --isolate WORLD`:
1. Starts a private, throwaway PostgreSQL cluster on local disk.
@@ -94,7 +94,9 @@ own identity, with the old writers fenced, is host-loss recovery and belongs to
M8.8.
**Credentials and keys.**
- A bundle carries no roles, grants, passwords or connection details.
- A bundle carries no roles, passwords or connection details. Since ADR-0075 its
dump keeps grants (role names only), for an in-place restore; the MANIFEST says
`dump privileges`.
- Restoring needs only a login that owns an empty database, and no source
credential. `backup-check.sh` and `verify-backup.sh` prove this with fresh,
unrelated roles.
+154
View File
@@ -0,0 +1,154 @@
# ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion
## Status
Accepted for M8.8, 2026-09-30. It builds on:
- ADR-0068 (upgrade barrier);
- ADR-0069 (World0 protection);
- ADR-0073 (fault drills);
- ADR-0074 (backup bundles).
It resolves founding open question 38 (DR objectives) for dev and stage. For
World0, it records why launch stays blocked.
## Context
M8.8 asks for three things:
- ratify the measured RPO and RTO, and host-loss causal durability, before any
irreversible acknowledgement;
- an independent durable chain, or else keep launch blocked;
- an explicit promotion of a reviewed digest and state.
The named risk is an old backup that resurrects dead organisms or duplicates
resources.
The facts:
- **One host.** Every AEvolution component runs on one physical host: the four
cluster VMs and the PostgreSQL VM pg-forge-01 (INVENTORY.md). A replica on that
host does not survive the host's loss (PERSISTENCE_ARCHITECTURE.md).
- **Acknowledgement.** The host acknowledges an outcome once it is committed in
that one database, with its payload on that one host's volume (ADR-0061/0064).
- **Backups.** Bundles (ADR-0074) are copies taken at a point in time. A restore
undoes every acknowledgement made after the bundle's cut.
## Decision
**1. Ratified objectives for dev and stage.** They rest on these measurements:
| Failure | Objective | Evidence |
|---|---|---|
| Worker/pod loss, stores healthy | RPO 0 for acknowledged outcomes; RTO ≤ 60 s | dev drill: 3/3 kills lost no acknowledgement; RTO 20.1, 19.8 and 20.2 s (lease-bound). M7.9 locally: 19.6 s |
| Object volume full | Nothing acknowledged while full; lossless resume | dev drill; retries now back off (M8.6 follow-up) |
| Object volume empty or wrong | Never ready, never a fresh world; lossless resume | dev drill (quarantine evidence re-run pending) |
| Database outage | Not ready, nothing acknowledged; a newer epoch after the outage | Local tests only. The dev drill did not partition the database. Stays a proposal until a valid dev run |
| Host loss | RPO = age of the last verified bundle; every later acknowledgement is rewound, explicitly | Local rehearsal (below). There is no schedule yet, so the dev/stage host-loss RPO is currently unbounded |
The 60 s RTO is ratified for dev and stage only. The OPERATIONS.md proposals for
World0 (24 h host-loss RTO, 15-minute RPO) are **rejected as launch criteria**. A
nonzero RPO for acknowledged deaths, births and ancestry contradicts permadeath.
**2. World0 launch is blocked.** World0 must not run a region host or acknowledge
anything until acknowledged outcomes survive host loss. The concrete requirement
is that, before acknowledging, the host has the complete causal recovery chain
durable in a second, independent failure domain:
- the commit row, its decisions and its outbox rows, for example through
synchronous PostgreSQL replication (`synchronous_commit = remote_apply`) to a
standby on other hardware;
- every payload and trace object the commit references, verified on storage in
that same second failure domain.
A host-loss drill must then show that no acknowledged outcome is rewound. None of
this hardware exists. The policy check enforces the block:
- no workload may target World0;
- the World0 Application keeps an unresolved path;
- no `region/world0` or `stores/world0` overlay may exist.
Lifting the block is the launch decision. It needs its own accepted ADR with that
evidence. Dev and stage proceed, as the roadmap allows.
**3. Host-loss recovery for dev and stage (restore in place).**
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
bundle under the world's own identity into a replacement database:
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
- It fences every restored lease, so the next owner acquires a newer epoch.
- It re-runs the migration rights audit for the environment's runtime role.
- It prints a `REWIND` line per region naming the tick it was restored to.
`--accept-rewind` is required: it states that undoing later acknowledgements is
acceptable, which it is only for disposable environments. The restored world does
**not** resurrect anything silently:
- the rewind is explicit and measured;
- history up to the cut is audited;
- the synthetic driver (or a future gateway) resumes after the cut, so a death at
or before the cut is never repeated.
**4. Fencing the old environment.** `fence-database.sh --superseded-by LABEL
--confirm`, run as owner on the old database, uses the ADR-0068 barrier:
- it retains the supersession identity SHA-256("AEVOLUTION-SUPERSEDED-v1"), which
no image digest can equal;
- it retires every other build in one transaction.
The effects:
- every lease is fenced;
- a running old host's next lease check fails and its re-acquisition is refused
(unsupported build), so it quarantines (exit 4);
- no later host can acquire;
- a commit naming a retired build is refused by the trigger.
The script prints the old environment's final heads, which measure the rewind.
Run it before any old host can return. If the old database is gone, run it the
moment it is reachable, and keep its credentials revoked until then. No schema
change is needed: supersession is "no current build except one that no image can
be".
**5. Promotion.** Nothing reaches World0 except by promotion:
- A World0 overlay may pin only the digests that the stage overlay pins for the
same image. The policy check enforces this. The digest is promoted after stage
qualification, never rebuilt.
- World0's opening is a reviewed world-design decision. The provisioning tool
accepts only `development-v1`, which may never provision World0.
- A promotion is one reviewed change that pins the digests and names:
- the stage evidence for that exact digest (drills, a verified backup);
- the reviewed opening;
- the accepted ADR that lifts the gate.
While the gate stands, promotion is defined and its policy is tested, but it
cannot happen.
## Evidence
`tools/ci/dr-check.sh` runs in every PostgreSQL build:
1. An old host serves world 400 past the tick-20 death. An online backup cuts at
tick 40, and the host goes on acknowledging, to 110.
2. The world is restored in place into a replacement database, while the old host
keeps running (a partition, the worst case).
3. A new host resumes at exactly `epoch=2 committed=40` and acknowledges from
tick 41.
4. The old database is fenced. The still-running old host drops its
unacknowledged inputs as `fenced` and quarantines (exit 4), and the old head
never advances again. A host started later on the old database quarantines at
once.
5. The restored world's full history audit passes: deaths, the founder and
descendant lineage, inventory conservation and ancestry. The death is durable
exactly once and never repeated.
6. The rewind is measured: 70 acknowledged ticks were undone. That number is the
reason World0 stays blocked.
The deploy policy check is tested negatively: a World0 workload, a resolved World0
Application path, and a World0 overlay digest not pinned in stage are each
rejected.
## Consequences
- AEvolution can run dev and stage, and qualify releases there. It cannot launch
World0 on this hardware.
- Host-loss recovery for dev and stage is a deliberate, recorded rewind.
Scheduling bundles (backlog) bounds its RPO.
- M8.9 (upgrade and growth) can proceed in dev and stage.
## Alternatives rejected
- **Launch World0 with a backup RPO.** Old backups would undo acknowledged deaths
and births. The founding brief forbids this, and a silent RPO change is not a
workaround.
- **Replicas on the same host** (a second PostgreSQL VM or a RAID mirror). These
are not an independent failure domain.
- **A schema-level "superseded" flag.** It would add a migration and a new error
path to the host. The existing barrier already gives the same fence, serialized
with every acquisition.
- **Restore in place without `--accept-rewind`.** It would make the rewind
implicit, which is exactly the named risk.
+1
View File
@@ -85,3 +85,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
- [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending.
- [ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion](0075-world0-acknowledgment-gate.md) - accepted for M8.8 ratified dev/stage RPO/RTO, World0 launch blocked (policy-enforced) until acknowledgements survive host loss in an independent failure domain, explicit-rewind in-place restore, barrier-based fencing of a superseded database, promotion only of stage-pinned digests.
+1 -1
View File
@@ -43,7 +43,7 @@ person. ADR indexes distinguish scoped proof choices from production proposals.
| 35 | Morphology limits | Development graph validates material, attachment and capability | Biology/design; M13 |
| 36 | Procedural animation | Isolated locomotion prototypes; no authored animation dependence | Engine; M13 |
| 37 | Retention | Permanent genome ancestry; tiered replay/snapshots/backups | Platform/design; M7/M8 |
| 38 | DR objectives | Propose measurable RPO/RTO, validate on actual storage/off-host target | Platform; before World0 |
| 38 | DR objectives | Dev/stage ratified (ADR-0075: worker loss RPO 0, RTO <= 60 s; host loss = explicit rewind to last verified bundle). World0 requires acknowledged outcomes durable in an independent failure domain before ACK; launch blocked until then | Platform; before World0 |
Additional blockers: actual host inventory, domain/TLS, registry and Git provider,
storage class/backup destination, resource budgets, project licensing and World0
+40
View File
@@ -2952,3 +2952,43 @@ Fixes (this segment):
- any restore of in-cluster data.
- The permanent destination, encryption, schedule and WAL archiving are not
implemented; they are pending the maintainer's choice of destination.
## M8.8 World0 gate and host-loss rehearsal - 2026-09-30
- `tools/ci/dr-check.sh` (inside `postgres-check.sh`; real PostgreSQL 18.6 and real
`evo-server` processes) passed in the Release, analysis and ASan/UBSan
PostgreSQL builds. The Release run:
- The old host served world 400 past the tick-20 death.
- An online backup cut at tick 40, and the old host went on acknowledging, to
tick 110.
- An in-place restore into a replacement database, with the old host still
running, audited 9 commits and printed `REWIND ... restored at tick 40`. It
left no lease held and passed the runtime rights audit.
- The new host resumed at `phase running epoch=2 committed=40`, and its first
acknowledgement was tick 41.
- `fence-database.sh` retired build b1…, leaving `current 0` and `held 0`, and
printed the final head (`final 400 1 22 110 1`). The still-running old host
dropped its queued inputs as `fenced` and exited 4 (quarantined). The old head
stayed at 110.
- A later host on the fenced database logged `phase quarantined` and exited 4,
never ready and with no acknowledgement.
- The restored world's audit passed (16 commits to tick 73, one outcome). There
was exactly one durable decision, and the new host printed no `outcome 20`.
- Measured rewind: 70 acknowledged ticks (110 − 40).
- The ASan run measured 115 and 75.
- Mutations, each caught by `dr-check.sh`:
- leases not fenced on restore: "a restored lease is still held";
- fence without the barrier: "fence evidence";
- in-place restore without grants: the runtime rights audit failed, and the
restore was removed.
- Deploy policy negatives, each rejected: a Job in the World0 path; the World0
Application with a resolved path; a `region/world0` overlay pinning a digest
that is not stage's. The unmodified tree: 67 objects, 0 failures.
- Found and fixed during validation: `dr-check.sh`'s EXIT trap ran `kill -9` on
exited processes under `set -e`, which made the script exit 1 after passing
(the kill now ends in `|| true`).
- The backup check still passes with grants kept in the dump.
- Not executed:
- any in-cluster host-loss restore or fence;
- a second failure domain, which does not exist;
- any World0 deployment or promotion, which is blocked by design.
+15 -1
View File
@@ -11,7 +11,7 @@ exercised (M8.8).
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below |
| Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria |
Targets are not observed results. One host cannot promise availability or zero
data loss after its destruction. A proposed 15-minute backup RPO for mutable
@@ -72,3 +72,17 @@ commit-before-ACK crash, stale owner resume, missing object, disk full, broker
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
permanent history and measured recovery. Never substitute namespace deletion for
a worker recovery test.
## Host-loss recovery in dev/stage (M8.8, ADR-0075)
1. Declare the loss and the replacement database (same PostgreSQL major version; the
environment's migrator and runtime roles recreated).
2. Restore the newest verified bundle in place, as the migrator:
`restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`.
Record every `REWIND` line: acknowledgements after those ticks are undone.
3. Point the environment's connection Secret and object volume at the replacement,
then sync the environment's Argo application; the host resumes under a newer epoch.
4. The moment the old database is reachable (or at once, if it never went away):
`fence-database.sh --superseded-by <label> --confirm` as its owner. Record the
printed final heads; the difference from the `REWIND` ticks is the measured loss.
5. Never use this for World0: its launch gate forbids a rewind of acknowledged outcomes.
+4 -2
View File
@@ -18,7 +18,8 @@
# source database or the object volume, and no admission barrier is needed.
#
# Bundle (DIR): MANIFEST, database.dump (pg_dump custom format, schema `evolution`,
# no owners or grants: a restore never needs the source's roles or credentials),
# no owners; grants are kept for an in-place restore, ADR-0075, and skipped by every
# other restore, which therefore never needs the source's roles or credentials),
# objects/w<world>/r<region>/<sha256>, SHA256SUMS over every other file. A source
# object whose bytes differ from its name fails the backup.
set -euo pipefail
@@ -71,7 +72,7 @@ SELECT 'rows', (SELECT count(*) FROM evolution.region_commits),
(SELECT count(*) FROM evolution.region_outbox),
(SELECT count(*) FROM evolution.region_origins);
\setenv EVO_BACKUP_SNAPSHOT :snapshot
\! pg_dump --no-password --format=custom --schema=evolution --no-owner --no-privileges --snapshot="$EVO_BACKUP_SNAPSHOT" --file="$EVO_BACKUP_DUMP" && echo 'dumped' || echo 'dump-failed'
\! pg_dump --no-password --format=custom --schema=evolution --no-owner --snapshot="$EVO_BACKUP_SNAPSHOT" --file="$EVO_BACKUP_DUMP" && echo 'dumped' || echo 'dump-failed'
COMMIT;
SQL
grep -qx 'dumped' "$out/.cut" || { echo "backup failed: pg_dump did not complete" >&2; exit 3; }
@@ -95,6 +96,7 @@ done < <(find "$objects" -mindepth 3 -maxdepth 3 -type f ! -name '.tmp-*' -print
echo "format aevolution-backup-v1"
echo "created $started"
echo "label ${label:--}"
echo "dump privileges"
grep -E '^(database|schema|build|head|rows) ' "$out/.cut"
echo "objects $count $bytes"
} > "$out/MANIFEST"
+60
View File
@@ -0,0 +1,60 @@
#!/usr/bin/env bash
# M8.8 supersession fence (ADR-0075): makes a database whose world was restored
# elsewhere permanently unable to grant authority again.
#
# tools/backup/fence-database.sh --superseded-by LABEL --confirm
#
# Connection: PG* environment naming the OLD database, as its owner (migrator).
# Run it as soon as the old database is reachable, before any of its hosts can
# return (or immediately, when it never went away).
#
# It uses the upgrade barrier (ADR-0068), which serializes with every acquisition,
# lease check and provisioning. It first retains a supersession identity,
# SHA-256("AEVOLUTION-SUPERSEDED-v1"), which no image digest can equal, and then
# retires every other build. Each retirement fences every lease. So:
# - a running host's next lease check or commit fails, and it re-acquires;
# - no retained build remains current, so every acquisition is refused and the
# host quarantines (evo-server exit 4);
# - a commit that names a retired build is refused by the trigger.
# History, objects and heads are untouched. The database stays readable for audits
# and for comparing what was lost. The heads at the moment of fencing are printed:
# they are the old environment's final state, and record what the restore rewound.
# Undoing this means deliberately retaining a new build as owner: that is an
# unfencing decision, not an operation.
set -euo pipefail
usage() { echo "usage: $0 --superseded-by LABEL --confirm" >&2; exit 2; }
label="" confirm=0
while [ $# -gt 0 ]; do
case "$1" in
--superseded-by) label="${2:-}"; shift 2 ;;
--confirm) confirm=1; shift ;;
*) usage ;;
esac
done
[[ "$label" =~ ^[A-Za-z0-9._:-]+$ ]] && [ "$confirm" = 1 ] || usage
if [ -n "${EVO_PGPASS_SOURCE:-}" ]; then
umask 077
export PGPASSFILE
PGPASSFILE="$(mktemp /tmp/evo-pgpass.XXXXXXXX)"
trap 'rm -f -- "$PGPASSFILE"' EXIT
cat "$EVO_PGPASS_SOURCE" > "$PGPASSFILE"
fi
psql -X --no-password -q -A -t -F ' ' -v ON_ERROR_STOP=1 -v "label=$label" <<'SQL'
BEGIN;
INSERT INTO evolution.retained_builds(build_identity)
VALUES (sha256(convert_to('AEVOLUTION-SUPERSEDED-v1', 'UTF8')))
ON CONFLICT (build_identity) DO NOTHING;
SELECT 'retired', encode(b.build_identity, 'hex'), evolution.upgrade_barrier(b.build_identity)
FROM evolution.retained_builds b
WHERE b.retired_at IS NULL
AND b.build_identity <> sha256(convert_to('AEVOLUTION-SUPERSEDED-v1', 'UTF8'))
ORDER BY b.build_identity;
SELECT 'current', count(*) FROM evolution.retained_builds b
WHERE b.retired_at IS NULL
AND b.build_identity <> sha256(convert_to('AEVOLUTION-SUPERSEDED-v1', 'UTF8'));
SELECT 'held', count(*) FROM evolution.region_leases WHERE holder IS NOT NULL;
SELECT 'final', h.world_id, h.region_id, h.commit_sequence, h.end_tick, h.published_epoch
FROM evolution.region_heads h ORDER BY 2, 3;
SELECT 'superseded-by', :'label';
COMMIT;
SQL
+50 -7
View File
@@ -2,11 +2,12 @@
# M8.7 verified restore of a backup bundle (ADR-0074) onto independent storage.
#
# tools/backup/restore-world.sh --bundle DIR --objects DIR
# [--in-place --runtime-role ROLE --accept-rewind]
#
# Target connection: PG* environment naming an EMPTY database that the login owns
# (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE).
# The restore needs no source role, credential or key: the dump carries no owners
# or grants, so the target login owns everything restored. --objects must not
# The restore needs no source role, credential or key: owners and grants are not
# restored, so the target login owns everything restored. --objects must not
# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or
# PATH.
#
@@ -22,18 +23,42 @@
# so an unverified restore can never be mistaken for a world. The result is a
# verbatim, owner-only copy with the source's world identities: it has no runtime
# grants and is never served. To run it, clone it into an isolated world
# (tools/backup/verify-backup.sh --isolate); host-loss recovery in place is M8.8.
# (tools/backup/verify-backup.sh --isolate).
#
# --in-place (M8.8 host-loss recovery for dev/stage, ADR-0075) restores the world
# under its own identity so its hosts can serve it again: the dump's grants are
# restored (the environment's roles must already exist in the target cluster), every
# restored lease is fenced so a new owner acquires under a newer epoch, and
# migrate-postgres.sh audits the runtime role's effective rights. Every acknowledgement
# after the bundle's cut is undone: deaths may reverse and resources may reappear.
# --accept-rewind states that this is acceptable, which it is for dev and stage only;
# World0 launch stays blocked until acknowledgements survive host loss (ADR-0075).
# Fence the old database with fence-database.sh before any of its hosts can return.
set -euo pipefail
usage() { echo "usage: $0 --bundle DIR --objects DIR" >&2; exit 2; }
bundle="" objects=""
usage() {
echo "usage: $0 --bundle DIR --objects DIR [--in-place --runtime-role ROLE --accept-rewind]" >&2
exit 2
}
bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0
while [ $# -gt 0 ]; do
case "$1" in
--bundle) bundle="${2:-}"; shift 2 ;;
--objects) objects="${2:-}"; shift 2 ;;
--in-place) in_place=1; shift ;;
--runtime-role) runtime_role="${2:-}"; shift 2 ;;
--accept-rewind) accept_rewind=1; shift ;;
*) usage ;;
esac
done
[ -n "$bundle" ] && [ -n "$objects" ] || usage
if [ "$in_place" = 1 ]; then
[ -n "$runtime_role" ] || { echo "--in-place needs --runtime-role" >&2; exit 2; }
[ "$accept_rewind" = 1 ] ||
{ echo "--in-place undoes every acknowledgement after the backup; pass --accept-rewind (dev/stage only)" >&2; exit 2; }
elif [ -n "$runtime_role" ] || [ "$accept_rewind" = 1 ]; then
usage
fi
root="$(cd "$(dirname "$0")/../.." && pwd)"
bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")"
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
umask 077
@@ -58,6 +83,11 @@ done <<< "$present"
grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format"
expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)"
[ -n "$expected_heads" ] || refuse "manifest names no region"
privileges=(--no-privileges)
if [ "$in_place" = 1 ]; then
grep -qx 'dump privileges' "$bundle/MANIFEST" || refuse "bundle predates in-place restore (no grants in its dump)"
privileges=()
fi
# 2. Target: empty, same server version.
[ ! -e "$objects" ] || refuse "objects target already exists: $objects"
@@ -79,7 +109,7 @@ trap cleanup EXIT
target="${PGDATABASE:-}"
[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE"
[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; }
pg_restore --no-password --exit-on-error --single-transaction --no-owner --no-privileges \
pg_restore --no-password --exit-on-error --single-transaction --no-owner "${privileges[@]}" \
--dbname="$target" "$bundle/database.dump"
mkdir -m 700 -- "$objects"
(cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' |
@@ -117,5 +147,18 @@ while read -r world region _ _ tick _; do
commits=$((commits + BASH_REMATCH[1]))
echo "region $world/$region $line"
done <<< "$expected_heads"
if [ "$in_place" = 1 ]; then
# No lease from before the loss survives: the next owner acquires a newer epoch.
psql -X --no-password -q -v ON_ERROR_STOP=1 -c \
'UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL' > /dev/null
EVO_PG_RUNTIME_ROLE="$runtime_role" bash "$root/tools/migrate-postgres.sh" > /dev/null ||
{ echo "runtime role $runtime_role failed the rights audit" >&2; exit 3; }
created="$(sed -n 's/^created //p' "$bundle/MANIFEST")"
while read -r world region _ _ tick _; do
echo "REWIND region $world/$region restored at tick $tick (backup $created): every acknowledgement after this tick is undone"
done <<< "$expected_heads"
fi
restored=1
echo "RESTORE VERIFIED regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
what="RESTORE VERIFIED"
[ "$in_place" = 1 ] && what="RESTORE IN PLACE VERIFIED"
echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
+29 -1
View File
@@ -15,7 +15,10 @@ environment-isolation or World0-protection invariant is broken:
cascade-delete finalizer, and each stays in its own environment;
- the live Argo path (deploy/kubernetes/dev) holds no Namespace, Secret, quota or
RBAC objects;
- the cleanup tool refuses World0 before invoking kubectl.
- the cleanup tool refuses World0 before invoking kubectl;
- World0 launch stays blocked (ADR-0075): no workload targets World0 and its
Application keeps an unresolved path; any image a World0 overlay pins must be
the digest already pinned in stage (promotion, never a fresh build).
"""
import os
import shutil
@@ -253,6 +256,31 @@ for path, doc in objects:
if path.startswith('deploy/') and doc['kind'] in ('ClusterRole', 'ClusterRoleBinding', 'Role', 'RoleBinding'):
check(False, f'{path}: AEvolution grants no Kubernetes RBAC')
# M8.8 World0 launch gate (ADR-0075). This host has no independent durable failure
# domain, so an acknowledged death could be undone by host loss: World0 must not run
# or acknowledge. Lifting this rule is itself the launch decision and needs its own
# accepted ADR with host-loss RPO-0 evidence.
workload_kinds = {'Deployment', 'StatefulSet', 'DaemonSet', 'Job', 'CronJob', 'Pod', 'ReplicaSet'}
for path, doc in objects:
in_world0 = doc['metadata'].get('namespace') == 'evolution-world0' or '/world0/' in path
check(not (in_world0 and doc['kind'] in workload_kinds),
f'{path}: World0 launch is blocked (ADR-0075); no {doc["kind"]} may target World0')
if doc['kind'] == 'Application' and doc['metadata'].get('name') == 'aevolution-world0':
check('REPLACE' in doc['spec'].get('source', {}).get('path', ''),
f'{path}: the World0 Application must keep an unresolved path while launch is blocked')
for component in ('region', 'stores'):
check(not os.path.exists(os.path.join(ROOT, 'deploy', 'kubernetes', component, 'world0')),
f'deploy/kubernetes/{component}/world0: World0 launch is blocked (ADR-0075)')
# Promotion: World0 may only ever pin a digest that stage already runs.
for directory, _, files in os.walk(os.path.join(ROOT, 'deploy', 'kubernetes')):
rel = os.path.relpath(directory, ROOT).replace(os.sep, '/')
if rel.endswith('/world0') and 'kustomization.yaml' in files:
stage = rel[:-len('world0')] + 'stage/kustomization.yaml'
staged = {(i.get('name'), i.get('digest')) for i in manifests.get(stage, [{}])[0].get('images', [])}
for image in manifests[rel + '/kustomization.yaml'][0].get('images', []):
check((image.get('name'), image.get('digest')) in staged,
f'{rel}: {image.get("name")} must be promoted from {stage}, not pinned independently')
# The platform bundle: every rendered object is protected from prune and deletion.
bundle = manifests.get(platform + '/cluster/kustomization.yaml', [{}])[0]
check(bundle.get('commonAnnotations', {}).get('argocd.argoproj.io/sync-options') == 'Prune=false,Delete=false',
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env bash
# M8.8 host-loss rehearsal (ADR-0075). Run by postgres-check.sh inside its throwaway
# cluster (PG* set; EVO_SERVER_BIN, EVO_WORLD_CLONE_BIN, runtime roles exported).
#
# The old environment keeps serving world 400 past an online backup, so it has
# acknowledged ticks the backup does not hold. The world is restored in place, under
# the same identity, into a replacement database, and a new host serves it. The old
# database is then fenced while its host is still running. Expected:
# - the new owner resumes exactly at the backup head, under a newer epoch;
# - the tick-20 death stays in history exactly once and is never repeated;
# - the full history audit (deaths, offspring, inventory, ancestry) passes;
# - the old host quarantines and acknowledges nothing after the fence;
# - a host started later against the old database never becomes ready;
# - the rewind (acknowledged by the old host but absent from the restore) is
# measured and reported. It is why World0 launch stays blocked.
set -euo pipefail
root="$(cd "$(dirname "$0")/../.." && pwd)"
work="$(mktemp -d /tmp/evo-dr-check.XXXXXXXX)"
chmod 700 "$work"
pids=()
trap 'for p in "${pids[@]}"; do kill -9 "$p" 2> /dev/null || true; done; rm -rf -- "$work"' EXIT
fail() { echo "host-loss rehearsal FAILED: $*" >&2; exit 1; }
build=b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1
db() { local d="$1"; shift; PGDATABASE="$d" psql -XAt -v ON_ERROR_STOP=1 "$@"; }
serve() { # serve <database> <objects> <log>: evo-server under synthetic load.
"$EVO_SERVER_BIN" --profile lineage --world 400 --region 1 --frame 9 \
--pg-host "$PGHOST" --pg-port "$PGPORT" --pg-database "$1" --pg-user "$EVO_PG_RUNTIME_USER" \
--objects "$2" --build "$build" --flush-ms 200 --lease-profile fast \
--synthetic-inputs development-v1 < /dev/null > "$3" 2>&1 &
pids+=($!)
server_pid=$!
}
acked_through() { awk -v t="$2" '$1 == "ack" && $3 >= t {f = 1} END {exit !f}' "$1"; }
max_ack() { awk '$1 == "ack" && $3 > m {m = $3} END {print m + 0}' "$1"; }
wait_ack() { for _ in $(seq 1 300); do acked_through "$1" "$2" && return 0; sleep 0.1; done; return 1; }
# The old environment: provisioned, serving, backed up online, then serving on.
createdb evo_dr_old
PGDATABASE=evo_dr_old bash "$root/tools/migrate-postgres.sh" > /dev/null
mkdir -m 700 "$work/old-objects"
"$EVO_WORLD_CLONE_BIN" provision --world 400 --region 1 --opening development-v1 \
--pg-host "$PGHOST" --pg-port "$PGPORT" --pg-database evo_dr_old --pg-user "$PGUSER" \
--objects "$work/old-objects" --build "$build" --retain-build true > /dev/null
serve evo_dr_old "$work/old-objects" "$work/old.log"; old_pid=$server_pid
wait_ack "$work/old.log" 40 || fail "the old host acknowledged nothing"
PGDATABASE=evo_dr_old bash "$root/tools/backup/backup-world.sh" --objects "$work/old-objects" \
--out "$work/bundle" --label rehearsal > /dev/null
read -r _ _ _ _ _ cut _ < <(grep '^head 400 1 ' "$work/bundle/MANIFEST")
[ "$cut" -ge 20 ] || fail "the backup precedes the death ($cut)"
wait_ack "$work/old.log" $((cut + 30)) || fail "the old host stopped after the backup"
# Host loss: the world is restored in place into a replacement database, while the
# old host still runs (a partition, the worst case for competing authority).
createdb evo_dr_new
PGDATABASE=evo_dr_new bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$work/new-objects" --in-place --runtime-role "$EVO_PG_RUNTIME_ROLE" --accept-rewind \
> "$work/restore.log" 2>&1 || { cat "$work/restore.log" >&2; fail "in-place restore"; }
grep -q "^REWIND region 400/1 restored at tick $cut " "$work/restore.log" &&
grep -q '^RESTORE IN PLACE VERIFIED regions=1 ' "$work/restore.log" || fail "restore evidence"
[ "$(db evo_dr_new -c 'SELECT count(*) FROM evolution.region_leases WHERE holder IS NOT NULL')" = 0 ] ||
fail "a restored lease is still held"
serve evo_dr_new "$work/new-objects" "$work/new.log"; new_pid=$server_pid
wait_ack "$work/new.log" $((cut + 30)) || { tail "$work/new.log" >&2; fail "the new host did not serve"; }
grep -q "^phase running epoch=2 committed=$cut " "$work/new.log" ||
fail "the new owner did not resume at the backup head under a newer epoch"
first_new="$(awk '$1 == "ack" {print $3; exit}' "$work/new.log")"
[ "$first_new" = $((cut + 1)) ] || fail "the new owner resumed at $first_new, not $((cut + 1))"
# Fence the old database while its host is still running.
PGDATABASE=evo_dr_old bash "$root/tools/backup/fence-database.sh" --superseded-by evo_dr_new \
--confirm > "$work/fence.log" || fail "fence"
grep -qx 'current 0' "$work/fence.log" && grep -q "^retired $build " "$work/fence.log" || fail "fence evidence"
final_old="$(db evo_dr_old -c 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 400')"
fenced_acks="$(max_ack "$work/old.log")"
code=0
for _ in $(seq 1 300); do kill -0 "$old_pid" 2> /dev/null || break; sleep 0.1; done
wait "$old_pid" || code=$?
[ "$code" = 4 ] || { tail "$work/old.log" >&2; fail "the fenced old host exited $code, not 4 (quarantined)"; }
[ "$(max_ack "$work/old.log")" -le "$final_old" ] || fail "the old host acknowledged past its final head"
[ "$(db evo_dr_old -c 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 400')" = "$final_old" ] ||
fail "the old database advanced after the fence"
# A host started later against the old database never serves.
serve evo_dr_old "$work/old-objects" "$work/late.log"; late_pid=$server_pid
code=0; wait "$late_pid" || code=$?
[ "$code" = 4 ] && ! grep -q '^ready 1' "$work/late.log" && ! grep -q '^ack ' "$work/late.log" ||
fail "a late host on the fenced database exited $code or served"
kill -TERM "$new_pid"; code=0; wait "$new_pid" || code=$?
[ "$code" = 0 ] || fail "the new host exited $code"
new_head="$(db evo_dr_new -c 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 400')"
audit="$("$EVO_WORLD_CLONE_BIN" audit --world 400 --region 1 --pg-host "$PGHOST" --pg-port "$PGPORT" \
--pg-database evo_dr_new --pg-user "$PGUSER" --objects "$work/new-objects" --retained "$build")" ||
fail "history audit of the restored world"
[[ "$audit" == *" tick=$new_head "* ]] || fail "audit: $audit"
# The death at tick 20 is durable once, before the cut, and never repeated.
[ "$(db evo_dr_new -c 'SELECT count(*) FROM evolution.region_decisions WHERE world_id = 400')" = 1 ] ||
fail "the restored world lost or repeated the death"
[ "$(grep -c '^outcome 20 ' "$work/new.log" || true)" = 0 ] || fail "the new owner repeated the death"
echo "host-loss rehearsal passed: cut=$cut old_acknowledged=$fenced_acks old_final=$final_old" \
"rewound=$((final_old - cut)) ticks new_head=$new_head ($audit)"
+2
View File
@@ -32,6 +32,8 @@ EVO_WORLD_CLONE_BIN="$(dirname "$1")/evo-world-clone"
"$1"
# M8.7: online backup under load, restore on a separate cluster, isolated rebuild.
bash "$root/tools/ci/backup-check.sh"
# M8.8: host-loss rehearsal: in-place restore, fenced old environment, measured rewind.
bash "$root/tools/ci/dr-check.sh"
v1_sha="$(sha256sum "$root/migrations/0001_region_bootstrap.sql")"
v1_sha="${v1_sha%% *}"
# Migration failure must not publish a partial schema.