Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked (policy-enforced) until acknowledged outcomes survive host loss in an independent failure domain. Adds in-place restore with an explicit rewind, barrier-based fencing of a superseded database, and promotion of stage-pinned digests only. dr-check.sh rehearses host loss in every PostgreSQL build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
89 lines
5.5 KiB
Markdown
89 lines
5.5 KiB
Markdown
# Operations and disaster recovery
|
|
|
|
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
|
|
implements snapshot-consistent world backup bundles and a verified restore on
|
|
independent storage with an isolated rebuild; the destination, encryption,
|
|
schedule and WAL archiving are pending, and no host-loss restore in place has been
|
|
exercised (M8.8).
|
|
|
|
| Failure | Proposed objective / gate |
|
|
|---|---|
|
|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
|
|
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
|
|
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
|
|
| Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria |
|
|
|
|
Targets are not observed results. One host cannot promise availability or zero
|
|
data loss after its destruction. A proposed 15-minute backup RPO for mutable
|
|
world state cannot guarantee every acknowledged death/birth/ancestry record.
|
|
Permanent lifecycle/history requires an independent durable copy of the complete
|
|
causal recovery chain and referenced payloads before ACK for
|
|
host-loss RPO0. That replication/acknowledgment contract and its latency are a
|
|
World0 launch gate. Overlaying isolated lifecycle records on older resource state
|
|
can duplicate biomass or lose offspring. Restoring an older world must not
|
|
resurrect organisms or erase history; if the required causal state cannot be
|
|
reconciled, keep the world unavailable.
|
|
|
|
## Backup and clone protocol
|
|
|
|
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
|
|
referenced by each recovery point. A database backup without those objects is not
|
|
a world backup. Manifest fields: world identity, simulation/generator/world/codec
|
|
versions, DB recovery target, region watermarks, object inventory/strong digests,
|
|
and binaries/container artifacts needed for replay. Encrypt off-host copies,
|
|
separate credentials and test recovery-key access.
|
|
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
|
|
|
|
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
|
|
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
|
|
history does not expire with backup retention. GC respects all clone/backup pins
|
|
and complete replay chains.
|
|
|
|
1. First implementation (ADR-0074) needs no admission barrier: one exported
|
|
database snapshot is the cut, and objects are immutable and verified before
|
|
commit, so copying them after the snapshot is consistent. A barrier returns
|
|
only if state ever spans more than one database.
|
|
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
|
|
transfer state, region watermarks and versions into the manifest.
|
|
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
|
|
4. Assign isolated world/runtime identities while preserving source provenance;
|
|
replace credentials, buckets and subjects; disable production integrations.
|
|
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
|
|
and hashes before readiness. Measure duration and missing-object behavior.
|
|
6. Record actual restore evidence. Upload success or a scheduled job is not a
|
|
restore test; alerts and recovery credentials must also work.
|
|
|
|
Actual recovery fences old database/cluster writers before routing cutover and
|
|
prevents the old environment returning as competing authority. Resolve ambiguous
|
|
commits by command ID/request digest; timeout is not rollback. Retain old
|
|
simulation binaries and generators needed by historical state.
|
|
|
|
## World0 protection and drills
|
|
|
|
Separate Argo permissions/environment credentials; Retain reclaim policy where
|
|
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
|
|
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
|
|
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
|
|
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
|
|
|
|
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
|
|
rejects World0. Drill worker termination in clones first: upload failure,
|
|
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
|
|
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
|
|
permanent history and measured recovery. Never substitute namespace deletion for
|
|
a worker recovery test.
|
|
|
|
## Host-loss recovery in dev/stage (M8.8, ADR-0075)
|
|
|
|
1. Declare the loss and the replacement database (same PostgreSQL major version; the
|
|
environment's migrator and runtime roles recreated).
|
|
2. Restore the newest verified bundle in place, as the migrator:
|
|
`restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`.
|
|
Record every `REWIND` line: acknowledgements after those ticks are undone.
|
|
3. Point the environment's connection Secret and object volume at the replacement,
|
|
then sync the environment's Argo application; the host resumes under a newer epoch.
|
|
4. The moment the old database is reachable (or at once, if it never went away):
|
|
`fence-database.sh --superseded-by <label> --confirm` as its owner. Record the
|
|
printed final heads; the difference from the `REWIND` ticks is the measured loss.
|
|
5. Never use this for World0: its launch gate forbids a rewind of acknowledged outcomes.
|