Files
aevolution/docs/infrastructure/OPERATIONS.md
T
dingoandClaude Opus 5.5 5512fdec54 feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)
Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:51:04 -04:00

75 lines
4.5 KiB
Markdown

# Operations and disaster recovery
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
implements snapshot-consistent world backup bundles and a verified restore on
independent storage with an isolated rebuild; the destination, encryption,
schedule and WAL archiving are pending, and no host-loss restore in place has been
exercised (M8.8).
| Failure | Proposed objective / gate |
|---|---|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below |
Targets are not observed results. One host cannot promise availability or zero
data loss after its destruction. A proposed 15-minute backup RPO for mutable
world state cannot guarantee every acknowledged death/birth/ancestry record.
Permanent lifecycle/history requires an independent durable copy of the complete
causal recovery chain and referenced payloads before ACK for
host-loss RPO0. That replication/acknowledgment contract and its latency are a
World0 launch gate. Overlaying isolated lifecycle records on older resource state
can duplicate biomass or lose offspring. Restoring an older world must not
resurrect organisms or erase history; if the required causal state cannot be
reconciled, keep the world unavailable.
## Backup and clone protocol
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
referenced by each recovery point. A database backup without those objects is not
a world backup. Manifest fields: world identity, simulation/generator/world/codec
versions, DB recovery target, region watermarks, object inventory/strong digests,
and binaries/container artifacts needed for replay. Encrypt off-host copies,
separate credentials and test recovery-key access.
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
history does not expire with backup retention. GC respects all clone/backup pins
and complete replay chains.
1. First implementation (ADR-0074) needs no admission barrier: one exported
database snapshot is the cut, and objects are immutable and verified before
commit, so copying them after the snapshot is consistent. A barrier returns
only if state ever spans more than one database.
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
transfer state, region watermarks and versions into the manifest.
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
4. Assign isolated world/runtime identities while preserving source provenance;
replace credentials, buckets and subjects; disable production integrations.
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
and hashes before readiness. Measure duration and missing-object behavior.
6. Record actual restore evidence. Upload success or a scheduled job is not a
restore test; alerts and recovery credentials must also work.
Actual recovery fences old database/cluster writers before routing cutover and
prevents the old environment returning as competing authority. Resolve ambiguous
commits by command ID/request digest; timeout is not rollback. Retain old
simulation binaries and generators needed by historical state.
## World0 protection and drills
Separate Argo permissions/environment credentials; Retain reclaim policy where
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
rejects World0. Drill worker termination in clones first: upload failure,
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
permanent history and measured recovery. Never substitute namespace deletion for
a worker recovery test.