feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)

Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 15:51:04 -04:00
co-authored by Claude Opus 5.5
parent 4a900b0bd3
commit 5512fdec54
19 changed files with 869 additions and 13 deletions
+27 -1
View File
@@ -192,7 +192,7 @@ To delete World0 data or identity on purpose:
1. Record the operation and its approval.
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
example a short-lived client certificate with that organization.
3. Take and verify a backup first (M8.7).
3. Take and verify a backup first (M8.7; see "Backups" below).
Removing the admission policy itself is the break-glass of last resort. It needs
cluster-admin and must be recorded the same way. Never test by deleting World0: the
@@ -206,3 +206,29 @@ verify script uses server-side dry runs.
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
node-local until M8.3.
## Backups (M8.7, ADR-0074)
Dev and stage only; the tools refuse World0. From a machine with kubectl access
that is not the hypervisor host:
```bash
# 1. Sync the environment's application once so the egress policy admits the
# backup component and the store overlay pins a postgres-tools image that
# contains pg_dump (any image built after ADR-0074).
# 2. Back up to a local directory (the destination until one is chosen):
tools/backup/cluster-backup.sh dev --to /path/to/backups
# 3. Restore-test it on independent storage and rebuild it as isolated world 900
# (WSL: the pinned PostgreSQL build and the Linux PostgreSQL build of the tools).
PATH="$PWD/.cache/postgres/install/bin:$PATH" \
LD_LIBRARY_PATH="$PWD/.cache/postgres/install/lib" \
EVO_WORLD_CLONE_BIN="$PWD/build/postgres-linux/evo-world-clone" \
EVO_SERVER_BIN="$PWD/build/postgres-linux/evo-server" \
tools/backup/verify-backup.sh /path/to/backups/dev-<time> --isolate 900
```
Only `verify-backup.sh` output that ends in `PASS restore verified and world ...
rebuilt as isolated world ...` counts as restore evidence. The restore uses fresh
roles on a throwaway cluster; nothing reaches the source database. A restored copy
keeps the source world identity and is never served; restoring in place after host
loss is M8.8.
+9 -4
View File
@@ -1,7 +1,10 @@
# Operations and disaster recovery
Status: proposal requiring measured acceptance before World0. No deployment,
backup or distributed restore has been exercised in this foundation.
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
implements snapshot-consistent world backup bundles and a verified restore on
independent storage with an isolated rebuild; the destination, encryption,
schedule and WAL archiving are pending, and no host-loss restore in place has been
exercised (M8.8).
| Failure | Proposed objective / gate |
|---|---|
@@ -36,8 +39,10 @@ pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
history does not expire with backup retention. GC respects all clone/backup pins
and complete replay chains.
1. First implementation uses a coordinated admission/transfer barrier, avoiding
an unfinished distributed-snapshot algorithm.
1. First implementation (ADR-0074) needs no admission barrier: one exported
database snapshot is the cut, and objects are immutable and verified before
commit, so copying them after the snapshot is consistent. A barrier returns
only if state ever spans more than one database.
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
transfer state, region watermarks and versions into the manifest.
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.