# Operations and disaster recovery Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074) implements snapshot-consistent world backup bundles and a verified restore on independent storage with an isolated rebuild; the destination, encryption, schedule and WAL archiving are pending, and no host-loss restore in place has been exercised (M8.8). | Failure | Proposed objective / gate | |---|---| | Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds | | Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget | | Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement | | Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria | Targets are not observed results. One host cannot promise availability or zero data loss after its destruction. A proposed 15-minute backup RPO for mutable world state cannot guarantee every acknowledged death/birth/ancestry record. Permanent lifecycle/history requires an independent durable copy of the complete causal recovery chain and referenced payloads before ACK for host-loss RPO0. That replication/acknowledgment contract and its latency are a World0 launch gate. Overlaying isolated lifecycle records on older resource state can duplicate biomass or lose offspring. Restoring an older world must not resurrect organisms or erase history; if the required causal state cannot be reconciled, keep the world unavailable. ## Backup and clone protocol Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object referenced by each recovery point. A database backup without those objects is not a world backup. Manifest fields: world identity, simulation/generator/world/codec versions, DB recovery target, region watermarks, object inventory/strong digests, and binaries/container artifacts needed for replay. Encrypt off-host copies, separate credentials and test recovery-key access. [PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html). Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle history does not expire with backup retention. GC respects all clone/backup pins and complete replay chains. 1. First implementation (ADR-0074) needs no admission barrier: one exported database snapshot is the cut, and objects are immutable and verified before commit, so copying them after the snapshot is consistent. A barrier returns only if state ever spans more than one database. 2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing transfer state, region watermarks and versions into the manifest. 3. Verify/copy objects, restore isolated PostgreSQL at the compatible point. 4. Assign isolated world/runtime identities while preserving source provenance; replace credentials, buckets and subjects; disable production integrations. 5. Replay and verify unique ownership, conservation, terminal deaths, ancestry and hashes before readiness. Measure duration and missing-object behavior. 6. Record actual restore evidence. Upload success or a scheduled job is not a restore test; alerts and recovery credentials must also work. Actual recovery fences old database/cluster writers before routing cutover and prevents the old environment returning as competing authority. Resolve ambiguous commits by command ID/request digest; timeout is not rollback. Retain old simulation binaries and generators needed by historical state. ## World0 protection and drills Separate Argo permissions/environment credentials; Retain reclaim policy where supported; protected PVC and namespace/data deletion via RBAC/admission. Test pruning and parent-app deletion separately. One prune flag is not a full safeguard. [Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/), [Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/). CI has no World0 data credentials. Cleanup requires exact dev/clone identity and rejects World0. Drill worker termination in clones first: upload failure, commit-before-ACK crash, stale owner resume, missing object, disk full, broker redelivery and each handoff stage. Verify one owner, no duplicated outcomes, permanent history and measured recovery. Never substitute namespace deletion for a worker recovery test. ## Host-loss recovery in dev/stage (M8.8, ADR-0075) 1. Declare the loss and the replacement database (same PostgreSQL major version; the environment's migrator and runtime roles recreated). 2. Restore the newest verified bundle in place, as the migrator: `restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`. Record every `REWIND` line: acknowledgements after those ticks are undone. 3. Point the environment's connection Secret and object volume at the replacement, then sync the environment's Argo application; the host resumes under a newer epoch. 4. The moment the old database is reachable (or at once, if it never went away): `fence-database.sh --superseded-by