feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)

Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 15:51:04 -04:00
co-authored by Claude Opus 5.5
parent 4a900b0bd3
commit 5512fdec54
19 changed files with 869 additions and 13 deletions
+3 -1
View File
@@ -7,7 +7,9 @@ names it. Remove an item only when it is done, and record where it was done.
| Item | Raised | Why deferred | Needed before |
|---|---|---|---|
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
| **Backup destination for dev/stage stores.** There are no backups yet (maintainer, 2026-09-29). The plan is to build backup tooling for the AEvolution PostgreSQL VM and each `aevolution-objects` volume up to a pluggable destination, then add the destination. It must be off the host for anything beyond disposable environments. | M8.3, 2026-09-29 | The maintainer will supply a destination later | M8.7 backup and restore; any World0 data |
| **Backup destination for dev/stage stores.** The tooling now exists up to the destination (ADR-0074): `tools/backup/cluster-backup.sh <env> --to DIR` writes a verified bundle to any directory, and `verify-backup.sh` restore-tests it. Still to choose: the permanent off-host destination, and with it (1) bundle encryption and key custody, (2) a scheduled CronJob synced by Argo, (3) retention (proposal: 30 daily, 12 weekly, 12 monthly), (4) incremental object copies. | M8.3, 2026-09-29; M8.7, 2026-09-30 | The maintainer will supply a destination later | Scheduled backups; any World0 data; M8.8 |
| **WAL archiving on pg-forge-01.** Bundles are logical backups: the recovery point is the last bundle. Continuous archiving (`archive_command` or `pg_receivewal`) plus periodic base backups would allow point-in-time recovery. It needs the backup destination. | M8.7, 2026-09-30 | Needs the destination and VM configuration | M8.8 host-loss RPO decision |
| **Harbor retention of build images.** A bundle's MANIFEST names every build (region image digest) that wrote history; recovery and replay need those images. Confirm Harbor's retention/tag-immutability policy keeps them (or pin them in a protected project). | M8.7, 2026-09-30 | Not yet checked | M8.8; any upgrade barrier that retires a build |
| **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 |
| **Store Job values.** Supply the published registry digests for `aevolution-postgres-tools` and `aevolution-server` (from each CI Publish step's `Digest:` line), and the NOLOGIN runtime group role for dev (`aevolution`) and for stage (`aevolution_stage`). Then include `stores/<env>` in each environment's path. | M8.3, 2026-09-29 | Values pending | M8.3 in-cluster evidence |
| **Alert destination.** Alertmanager routes every alert to a null receiver (ADR-0072). Choose where alerts go (Discord, email, a webhook), and optionally a heartbeat receiver for the always-firing Watchdog. Receiver credentials go in a Secret. | M8.5, 2026-09-30 | The maintainer will decide later | Any claim that alerts reach a person; M8.6 alert evidence; World0 |