Files
aevolution/docs/infrastructure/OPERATIONS.md
T
dingoandClaude Opus 5.5 518a960864 feat(ops): M8.8 World0 acknowledgment gate and host-loss recovery (ADR-0075)
Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked
(policy-enforced) until acknowledged outcomes survive host loss in an
independent failure domain. Adds in-place restore with an explicit rewind,
barrier-based fencing of a superseded database, and promotion of stage-pinned
digests only. dr-check.sh rehearses host loss in every PostgreSQL build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:07:07 -04:00

89 lines
5.5 KiB
Markdown

# Operations and disaster recovery
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
implements snapshot-consistent world backup bundles and a verified restore on
independent storage with an isolated rebuild; the destination, encryption,
schedule and WAL archiving are pending, and no host-loss restore in place has been
exercised (M8.8).
| Failure | Proposed objective / gate |
|---|---|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
| Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria |
Targets are not observed results. One host cannot promise availability or zero
data loss after its destruction. A proposed 15-minute backup RPO for mutable
world state cannot guarantee every acknowledged death/birth/ancestry record.
Permanent lifecycle/history requires an independent durable copy of the complete
causal recovery chain and referenced payloads before ACK for
host-loss RPO0. That replication/acknowledgment contract and its latency are a
World0 launch gate. Overlaying isolated lifecycle records on older resource state
can duplicate biomass or lose offspring. Restoring an older world must not
resurrect organisms or erase history; if the required causal state cannot be
reconciled, keep the world unavailable.
## Backup and clone protocol
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
referenced by each recovery point. A database backup without those objects is not
a world backup. Manifest fields: world identity, simulation/generator/world/codec
versions, DB recovery target, region watermarks, object inventory/strong digests,
and binaries/container artifacts needed for replay. Encrypt off-host copies,
separate credentials and test recovery-key access.
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
history does not expire with backup retention. GC respects all clone/backup pins
and complete replay chains.
1. First implementation (ADR-0074) needs no admission barrier: one exported
database snapshot is the cut, and objects are immutable and verified before
commit, so copying them after the snapshot is consistent. A barrier returns
only if state ever spans more than one database.
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
transfer state, region watermarks and versions into the manifest.
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
4. Assign isolated world/runtime identities while preserving source provenance;
replace credentials, buckets and subjects; disable production integrations.
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
and hashes before readiness. Measure duration and missing-object behavior.
6. Record actual restore evidence. Upload success or a scheduled job is not a
restore test; alerts and recovery credentials must also work.
Actual recovery fences old database/cluster writers before routing cutover and
prevents the old environment returning as competing authority. Resolve ambiguous
commits by command ID/request digest; timeout is not rollback. Retain old
simulation binaries and generators needed by historical state.
## World0 protection and drills
Separate Argo permissions/environment credentials; Retain reclaim policy where
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
rejects World0. Drill worker termination in clones first: upload failure,
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
permanent history and measured recovery. Never substitute namespace deletion for
a worker recovery test.
## Host-loss recovery in dev/stage (M8.8, ADR-0075)
1. Declare the loss and the replacement database (same PostgreSQL major version; the
environment's migrator and runtime roles recreated).
2. Restore the newest verified bundle in place, as the migrator:
`restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`.
Record every `REWIND` line: acknowledgements after those ticks are undone.
3. Point the environment's connection Secret and object volume at the replacement,
then sync the environment's Argo application; the host resumes under a newer epoch.
4. The moment the old database is reachable (or at once, if it never went away):
`fence-database.sh --superseded-by <label> --confirm` as its owner. Record the
printed final heads; the difference from the `REWIND` ticks is the measured loss.
5. Never use this for World0: its launch gate forbids a rewind of acknowledged outcomes.