From 518a960864308df01c70028531fec3924e3e36a0 Mon Sep 17 00:00:00 2001 From: dingo Date: Wed, 30 Sep 2026 17:07:07 -0400 Subject: [PATCH] feat(ops): M8.8 World0 acknowledgment gate and host-loss recovery (ADR-0075) Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked (policy-enforced) until acknowledged outcomes survive host loss in an independent failure domain. Adds in-place restore with an explicit rewind, barrier-based fencing of a superseded database, and promotion of stage-pinned digests only. dr-check.sh rehearses host loss in every PostgreSQL build. Co-Authored-By: Claude Opus 5.5 --- docs/BACKLOG.md | 1 + docs/ROADMAP.md | 8 + docs/STATUS.md | 16 ++ docs/adr/0074-world-backup-and-restore.md | 8 +- docs/adr/0075-world0-acknowledgment-gate.md | 154 ++++++++++++++++++++ docs/adr/README.md | 1 + docs/architecture/OPEN_QUESTIONS.md | 2 +- docs/engineering/VALIDATION.md | 40 +++++ docs/infrastructure/OPERATIONS.md | 16 +- tools/backup/backup-world.sh | 6 +- tools/backup/fence-database.sh | 60 ++++++++ tools/backup/restore-world.sh | 57 +++++++- tools/ci/deploy-policy-check.py | 30 +++- tools/ci/dr-check.sh | 100 +++++++++++++ tools/ci/postgres-check.sh | 2 + 15 files changed, 486 insertions(+), 15 deletions(-) create mode 100644 docs/adr/0075-world0-acknowledgment-gate.md create mode 100644 tools/backup/fence-database.sh create mode 100644 tools/ci/dr-check.sh diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index fe8946a..24e65d8 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -8,6 +8,7 @@ names it. Remove an item only when it is done, and record where it was done. |---|---|---|---| | **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch | | **Backup destination for dev/stage stores.** The tooling now exists up to the destination (ADR-0074): `tools/backup/cluster-backup.sh --to DIR` writes a verified bundle to any directory, and `verify-backup.sh` restore-tests it. Still to choose: the permanent off-host destination, and with it (1) bundle encryption and key custody, (2) a scheduled CronJob synced by Argo, (3) retention (proposal: 30 daily, 12 weekly, 12 monthly), (4) incremental object copies. | M8.3, 2026-09-29; M8.7, 2026-09-30 | The maintainer will supply a destination later | Scheduled backups; any World0 data; M8.8 | +| **Independent durable failure domain (World0 launch).** World0 stays blocked until acknowledged outcomes survive host loss (ADR-0075): before acknowledging, the commit rows and every referenced object must be durable on other hardware (for example a synchronous PostgreSQL standby with `remote_apply`, and object replication verified before commit), proven by a host-loss drill that rewinds nothing. Needs a second physical host and storage. | M8.8, 2026-09-30 | No second failure domain exists | World0 launch; lifting the ADR-0075 policy rule | | **WAL archiving on pg-forge-01.** Bundles are logical backups: the recovery point is the last bundle. Continuous archiving (`archive_command` or `pg_receivewal`) plus periodic base backups would allow point-in-time recovery. It needs the backup destination. | M8.7, 2026-09-30 | Needs the destination and VM configuration | M8.8 host-loss RPO decision | | **Harbor retention of build images.** A bundle's MANIFEST names every build (region image digest) that wrote history; recovery and replay need those images. Confirm Harbor's retention/tag-immutability policy keeps them (or pin them in a protected project). | M8.7, 2026-09-30 | Not yet checked | M8.8; any upgrade barrier that retires a build | | **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 | diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 8a5e20f..7e39b4c 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -660,3 +660,11 @@ M8.7 implemented and locally qualified (2026-09-30), under ADR-0074: - an operator-run in-cluster backup to a directory off the cluster. Pending the maintainer's destination: encryption, schedule, retention and WAL archiving (docs/BACKLOG.md). In-cluster bundle evidence waits on a run. + +M8.8 decided (2026-09-30), under ADR-0075: dev/stage objectives ratified from +measurements (worker loss RPO 0 / RTO <= 60 s; host loss = explicit rewind to the +last verified bundle); **World0 launch stays blocked** because no independent +durable failure domain exists, enforced by the deploy policy check; host-loss +restore in place with barrier fencing of the superseded database rehearsed +locally; promotion only of stage-pinned digests. **M8-dev/stage passed; +M8-World0 launch not passed.** diff --git a/docs/STATUS.md b/docs/STATUS.md index fd407dd..5225f03 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1125,3 +1125,19 @@ ADR-0074 adds world backup bundles and a verified restore: The destination, encryption, schedule, retention and WAL archiving wait on the maintainer (docs/BACKLOG.md). + +## M8.8 World0 gate and host-loss recovery (2026-09-30) + +ADR-0075: +- **World0 launch is blocked.** Everything runs on one physical host, so an + acknowledged death could be undone by host loss. The deploy policy check now + rejects any World0 workload, a resolved World0 Application path, and World0 + overlays; any future World0 pin must equal stage's digest. +- **Dev/stage objectives ratified:** worker loss RPO 0 and RTO <= 60 s (measured + ~20 s); host loss rewinds to the last verified bundle, explicitly. +- `restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a world + under its own identity with its grants, fences restored leases, audits runtime + rights and prints the rewind. +- `fence-database.sh` supersedes the old database through the upgrade barrier: a + still-running old host quarantines, and no host can acquire it again. +- `tools/ci/dr-check.sh` rehearses it in every PostgreSQL build. diff --git a/docs/adr/0074-world-backup-and-restore.md b/docs/adr/0074-world-backup-and-restore.md index 3ae80ca..298f761 100644 --- a/docs/adr/0074-world-backup-and-restore.md +++ b/docs/adr/0074-world-backup-and-restore.md @@ -37,7 +37,7 @@ facts make a barrier unnecessary for a single database: | File | Content | |---|---| | `MANIFEST` | Format, time and label. The server version. Every schema migration and its checksum. Every retained build (image digest) and whether it is retired. Every region head (world, region, frame, sequence, tick, commit, digest, build). Row counts, and the object count and bytes | -| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners and no grants | +| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners. Grants are kept for an in-place restore (ADR-0075 amendment); every other restore skips them | | `objects/w/r/` | Every object in the store, each checked against its name while it is copied | | `SHA256SUMS` | Checksums of every other file | @@ -76,7 +76,7 @@ empty database owned by the restoring login, and fails closed in this order: A failure after step 2 removes the restored schema and objects, so an unverified restore cannot be mistaken for a world. The result is a verbatim, owner-only copy that keeps the source's world identities. It has no runtime grants and is never -served. +served. (ADR-0075 adds `--in-place` for dev/stage host-loss recovery.) **Isolated rebuild.** `tools/backup/verify-backup.sh BUNDLE --isolate WORLD`: 1. Starts a private, throwaway PostgreSQL cluster on local disk. @@ -94,7 +94,9 @@ own identity, with the old writers fenced, is host-loss recovery and belongs to M8.8. **Credentials and keys.** -- A bundle carries no roles, grants, passwords or connection details. +- A bundle carries no roles, passwords or connection details. Since ADR-0075 its + dump keeps grants (role names only), for an in-place restore; the MANIFEST says + `dump privileges`. - Restoring needs only a login that owns an empty database, and no source credential. `backup-check.sh` and `verify-backup.sh` prove this with fresh, unrelated roles. diff --git a/docs/adr/0075-world0-acknowledgment-gate.md b/docs/adr/0075-world0-acknowledgment-gate.md new file mode 100644 index 0000000..da524df --- /dev/null +++ b/docs/adr/0075-world0-acknowledgment-gate.md @@ -0,0 +1,154 @@ +# ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion + +## Status +Accepted for M8.8, 2026-09-30. It builds on: +- ADR-0068 (upgrade barrier); +- ADR-0069 (World0 protection); +- ADR-0073 (fault drills); +- ADR-0074 (backup bundles). + +It resolves founding open question 38 (DR objectives) for dev and stage. For +World0, it records why launch stays blocked. + +## Context +M8.8 asks for three things: +- ratify the measured RPO and RTO, and host-loss causal durability, before any + irreversible acknowledgement; +- an independent durable chain, or else keep launch blocked; +- an explicit promotion of a reviewed digest and state. + +The named risk is an old backup that resurrects dead organisms or duplicates +resources. + +The facts: +- **One host.** Every AEvolution component runs on one physical host: the four + cluster VMs and the PostgreSQL VM pg-forge-01 (INVENTORY.md). A replica on that + host does not survive the host's loss (PERSISTENCE_ARCHITECTURE.md). +- **Acknowledgement.** The host acknowledges an outcome once it is committed in + that one database, with its payload on that one host's volume (ADR-0061/0064). +- **Backups.** Bundles (ADR-0074) are copies taken at a point in time. A restore + undoes every acknowledgement made after the bundle's cut. + +## Decision +**1. Ratified objectives for dev and stage.** They rest on these measurements: + +| Failure | Objective | Evidence | +|---|---|---| +| Worker/pod loss, stores healthy | RPO 0 for acknowledged outcomes; RTO ≤ 60 s | dev drill: 3/3 kills lost no acknowledgement; RTO 20.1, 19.8 and 20.2 s (lease-bound). M7.9 locally: 19.6 s | +| Object volume full | Nothing acknowledged while full; lossless resume | dev drill; retries now back off (M8.6 follow-up) | +| Object volume empty or wrong | Never ready, never a fresh world; lossless resume | dev drill (quarantine evidence re-run pending) | +| Database outage | Not ready, nothing acknowledged; a newer epoch after the outage | Local tests only. The dev drill did not partition the database. Stays a proposal until a valid dev run | +| Host loss | RPO = age of the last verified bundle; every later acknowledgement is rewound, explicitly | Local rehearsal (below). There is no schedule yet, so the dev/stage host-loss RPO is currently unbounded | + +The 60 s RTO is ratified for dev and stage only. The OPERATIONS.md proposals for +World0 (24 h host-loss RTO, 15-minute RPO) are **rejected as launch criteria**. A +nonzero RPO for acknowledged deaths, births and ancestry contradicts permadeath. + +**2. World0 launch is blocked.** World0 must not run a region host or acknowledge +anything until acknowledged outcomes survive host loss. The concrete requirement +is that, before acknowledging, the host has the complete causal recovery chain +durable in a second, independent failure domain: +- the commit row, its decisions and its outbox rows, for example through + synchronous PostgreSQL replication (`synchronous_commit = remote_apply`) to a + standby on other hardware; +- every payload and trace object the commit references, verified on storage in + that same second failure domain. + +A host-loss drill must then show that no acknowledged outcome is rewound. None of +this hardware exists. The policy check enforces the block: +- no workload may target World0; +- the World0 Application keeps an unresolved path; +- no `region/world0` or `stores/world0` overlay may exist. + +Lifting the block is the launch decision. It needs its own accepted ADR with that +evidence. Dev and stage proceed, as the roadmap allows. + +**3. Host-loss recovery for dev and stage (restore in place).** +`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a +bundle under the world's own identity into a replacement database: +- It restores the dump's grants (bundles now keep them; ADR-0074 is amended). +- It fences every restored lease, so the next owner acquires a newer epoch. +- It re-runs the migration rights audit for the environment's runtime role. +- It prints a `REWIND` line per region naming the tick it was restored to. + +`--accept-rewind` is required: it states that undoing later acknowledgements is +acceptable, which it is only for disposable environments. The restored world does +**not** resurrect anything silently: +- the rewind is explicit and measured; +- history up to the cut is audited; +- the synthetic driver (or a future gateway) resumes after the cut, so a death at + or before the cut is never repeated. + +**4. Fencing the old environment.** `fence-database.sh --superseded-by LABEL +--confirm`, run as owner on the old database, uses the ADR-0068 barrier: +- it retains the supersession identity SHA-256("AEVOLUTION-SUPERSEDED-v1"), which + no image digest can equal; +- it retires every other build in one transaction. + +The effects: +- every lease is fenced; +- a running old host's next lease check fails and its re-acquisition is refused + (unsupported build), so it quarantines (exit 4); +- no later host can acquire; +- a commit naming a retired build is refused by the trigger. + +The script prints the old environment's final heads, which measure the rewind. +Run it before any old host can return. If the old database is gone, run it the +moment it is reachable, and keep its credentials revoked until then. No schema +change is needed: supersession is "no current build except one that no image can +be". + +**5. Promotion.** Nothing reaches World0 except by promotion: +- A World0 overlay may pin only the digests that the stage overlay pins for the + same image. The policy check enforces this. The digest is promoted after stage + qualification, never rebuilt. +- World0's opening is a reviewed world-design decision. The provisioning tool + accepts only `development-v1`, which may never provision World0. +- A promotion is one reviewed change that pins the digests and names: + - the stage evidence for that exact digest (drills, a verified backup); + - the reviewed opening; + - the accepted ADR that lifts the gate. + +While the gate stands, promotion is defined and its policy is tested, but it +cannot happen. + +## Evidence +`tools/ci/dr-check.sh` runs in every PostgreSQL build: +1. An old host serves world 400 past the tick-20 death. An online backup cuts at + tick 40, and the host goes on acknowledging, to 110. +2. The world is restored in place into a replacement database, while the old host + keeps running (a partition, the worst case). +3. A new host resumes at exactly `epoch=2 committed=40` and acknowledges from + tick 41. +4. The old database is fenced. The still-running old host drops its + unacknowledged inputs as `fenced` and quarantines (exit 4), and the old head + never advances again. A host started later on the old database quarantines at + once. +5. The restored world's full history audit passes: deaths, the founder and + descendant lineage, inventory conservation and ancestry. The death is durable + exactly once and never repeated. +6. The rewind is measured: 70 acknowledged ticks were undone. That number is the + reason World0 stays blocked. + +The deploy policy check is tested negatively: a World0 workload, a resolved World0 +Application path, and a World0 overlay digest not pinned in stage are each +rejected. + +## Consequences +- AEvolution can run dev and stage, and qualify releases there. It cannot launch + World0 on this hardware. +- Host-loss recovery for dev and stage is a deliberate, recorded rewind. + Scheduling bundles (backlog) bounds its RPO. +- M8.9 (upgrade and growth) can proceed in dev and stage. + +## Alternatives rejected +- **Launch World0 with a backup RPO.** Old backups would undo acknowledged deaths + and births. The founding brief forbids this, and a silent RPO change is not a + workaround. +- **Replicas on the same host** (a second PostgreSQL VM or a RAID mirror). These + are not an independent failure domain. +- **A schema-level "superseded" flag.** It would add a migration and a new error + path to the host. The existing barrier already gives the same fence, serialized + with every acquisition. +- **Restore in place without `--accept-rewind`.** It would make the rewind + implicit, which is exactly the named risk. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1dc7520..bc95bf3 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -85,3 +85,4 @@ design. No unadopted candidate is a link dependency of the proof. - [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report. - [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks. - [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending. +- [ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion](0075-world0-acknowledgment-gate.md) - accepted for M8.8 ratified dev/stage RPO/RTO, World0 launch blocked (policy-enforced) until acknowledgements survive host loss in an independent failure domain, explicit-rewind in-place restore, barrier-based fencing of a superseded database, promotion only of stage-pinned digests. diff --git a/docs/architecture/OPEN_QUESTIONS.md b/docs/architecture/OPEN_QUESTIONS.md index 64c68ef..f495d93 100644 --- a/docs/architecture/OPEN_QUESTIONS.md +++ b/docs/architecture/OPEN_QUESTIONS.md @@ -43,7 +43,7 @@ person. ADR indexes distinguish scoped proof choices from production proposals. | 35 | Morphology limits | Development graph validates material, attachment and capability | Biology/design; M13 | | 36 | Procedural animation | Isolated locomotion prototypes; no authored animation dependence | Engine; M13 | | 37 | Retention | Permanent genome ancestry; tiered replay/snapshots/backups | Platform/design; M7/M8 | -| 38 | DR objectives | Propose measurable RPO/RTO, validate on actual storage/off-host target | Platform; before World0 | +| 38 | DR objectives | Dev/stage ratified (ADR-0075: worker loss RPO 0, RTO <= 60 s; host loss = explicit rewind to last verified bundle). World0 requires acknowledged outcomes durable in an independent failure domain before ACK; launch blocked until then | Platform; before World0 | Additional blockers: actual host inventory, domain/TLS, registry and Git provider, storage class/backup destination, resource budgets, project licensing and World0 diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 2a532e4..db38053 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2952,3 +2952,43 @@ Fixes (this segment): - any restore of in-cluster data. - The permanent destination, encryption, schedule and WAL archiving are not implemented; they are pending the maintainer's choice of destination. + +## M8.8 World0 gate and host-loss rehearsal - 2026-09-30 + +- `tools/ci/dr-check.sh` (inside `postgres-check.sh`; real PostgreSQL 18.6 and real + `evo-server` processes) passed in the Release, analysis and ASan/UBSan + PostgreSQL builds. The Release run: + - The old host served world 400 past the tick-20 death. + - An online backup cut at tick 40, and the old host went on acknowledging, to + tick 110. + - An in-place restore into a replacement database, with the old host still + running, audited 9 commits and printed `REWIND ... restored at tick 40`. It + left no lease held and passed the runtime rights audit. + - The new host resumed at `phase running epoch=2 committed=40`, and its first + acknowledgement was tick 41. + - `fence-database.sh` retired build b1…, leaving `current 0` and `held 0`, and + printed the final head (`final 400 1 22 110 1`). The still-running old host + dropped its queued inputs as `fenced` and exited 4 (quarantined). The old head + stayed at 110. + - A later host on the fenced database logged `phase quarantined` and exited 4, + never ready and with no acknowledgement. + - The restored world's audit passed (16 commits to tick 73, one outcome). There + was exactly one durable decision, and the new host printed no `outcome 20`. + - Measured rewind: 70 acknowledged ticks (110 − 40). + - The ASan run measured 115 and 75. +- Mutations, each caught by `dr-check.sh`: + - leases not fenced on restore: "a restored lease is still held"; + - fence without the barrier: "fence evidence"; + - in-place restore without grants: the runtime rights audit failed, and the + restore was removed. +- Deploy policy negatives, each rejected: a Job in the World0 path; the World0 + Application with a resolved path; a `region/world0` overlay pinning a digest + that is not stage's. The unmodified tree: 67 objects, 0 failures. +- Found and fixed during validation: `dr-check.sh`'s EXIT trap ran `kill -9` on + exited processes under `set -e`, which made the script exit 1 after passing + (the kill now ends in `|| true`). +- The backup check still passes with grants kept in the dump. +- Not executed: + - any in-cluster host-loss restore or fence; + - a second failure domain, which does not exist; + - any World0 deployment or promotion, which is blocked by design. diff --git a/docs/infrastructure/OPERATIONS.md b/docs/infrastructure/OPERATIONS.md index eb8896f..fdf1411 100644 --- a/docs/infrastructure/OPERATIONS.md +++ b/docs/infrastructure/OPERATIONS.md @@ -11,7 +11,7 @@ exercised (M8.8). | Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds | | Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget | | Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement | -| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below | +| Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria | Targets are not observed results. One host cannot promise availability or zero data loss after its destruction. A proposed 15-minute backup RPO for mutable @@ -72,3 +72,17 @@ commit-before-ACK crash, stale owner resume, missing object, disk full, broker redelivery and each handoff stage. Verify one owner, no duplicated outcomes, permanent history and measured recovery. Never substitute namespace deletion for a worker recovery test. + +## Host-loss recovery in dev/stage (M8.8, ADR-0075) + +1. Declare the loss and the replacement database (same PostgreSQL major version; the + environment's migrator and runtime roles recreated). +2. Restore the newest verified bundle in place, as the migrator: + `restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`. + Record every `REWIND` line: acknowledgements after those ticks are undone. +3. Point the environment's connection Secret and object volume at the replacement, + then sync the environment's Argo application; the host resumes under a newer epoch. +4. The moment the old database is reachable (or at once, if it never went away): + `fence-database.sh --superseded-by