feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)

Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 15:51:04 -04:00
co-authored by Claude Opus 5.5
parent 4a900b0bd3
commit 5512fdec54
19 changed files with 869 additions and 13 deletions
+36
View File
@@ -2916,3 +2916,39 @@ Fixes (this segment):
MSVC CTest 59/59 (the server is not part of the Windows build).
`fault-drill.sh` passed `bash -n`.
- Pending: a new image, then re-running the full, empty and db-outage scenarios.
## M8.7 backup and restore - 2026-09-30
- `tools/ci/backup-check.sh` (inside `postgres-check.sh`; real PostgreSQL 18.6, a
real `evo-server` under synthetic load) passed in the Release, analysis and
ASan/UBSan PostgreSQL builds:
- An online backup taken while the server committed recorded head tick 60, at or
past every acknowledgement before it (60). The source then moved on to tick 101.
- `verify-backup.sh --isolate 301` on a second, throwaway cluster, as fresh
roles: the restore audited 13 commits to tick 60 and matched the MANIFEST. The
clone into world 301 resumed at `epoch=1 committed=60`, acknowledged to tick
82, and drained with exit 0.
- Refused bundles, each leaving no schema and no objects behind:
- corrupt object, unlisted file, damaged dump: exit 4 at the checksum step;
- missing payload object with consistent checksums: exit 3,
`audit failed missing_payload sequence=0`;
- MANIFEST head changed: exit 3, heads differ.
- A second restore into a database that already holds a world was refused
(exit 4), and the first restore was left unchanged.
- The MANIFEST names no credentials.
- Mutations, each caught by `backup-check.sh`:
- restore without the history audit: the missing object was accepted (exit 0);
- a failed restore that keeps its files: restored objects were left behind;
- `pg_dump` outside the exported snapshot (2 s later): the restored heads
differed from the MANIFEST, and verification failed.
- The PostgreSQL migration, rollback and permission suites passed in all three
builds. `deploy-policy-check.py`: 67 objects, 0 failures.
`cluster-backup.sh` refuses World0 before kubectl (policy rule).
- `bash -n` passed on every new script. shellcheck is not installed in WSL, so it
did not run.
- Not executed:
- the postgres-tools image build with `pg_dump` (CI builds it on push);
- `cluster-backup.sh` against dev or stage;
- any restore of in-cluster data.
- The permanent destination, encryption, schedule and WAL archiving are not
implemented; they are pending the maintainer's choice of destination.