Foundation / checks (ecs) (push) Successful in 2m23s
Foundation / checks (deploy) (push) Successful in 4m10s
Foundation / checks (network) (push) Successful in 21m32s
Foundation / checks (client) (push) Successful in 22m14s
Foundation / checks (tsan) (push) Successful in 24m30s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 41m43s
Foundation / checks (postgres) (push) Canceled after 41m39s
95 lines
6.0 KiB
Markdown
95 lines
6.0 KiB
Markdown
# Operations and disaster recovery
|
|
|
|
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
|
|
implements snapshot-consistent world backup bundles and a verified restore on
|
|
independent storage with an isolated rebuild; the destination, encryption,
|
|
schedule and WAL archiving are pending, and no host-loss restore in place has been
|
|
exercised (M8.8).
|
|
|
|
| Failure | Proposed objective / gate |
|
|
|---|---|
|
|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
|
|
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
|
|
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
|
|
| Host destruction | Dev/stage: explicit rewind to the last verified bundle (ADR-0075). World0: launch blocked until acknowledged outcomes are durable in an independent failure domain before ACK; the 24-hour/15-minute proposals are rejected as launch criteria |
|
|
|
|
Targets are not observed results. One host cannot promise availability or zero
|
|
data loss after its destruction. A proposed 15-minute backup RPO for mutable
|
|
world state cannot guarantee every acknowledged death/birth/ancestry record.
|
|
Permanent lifecycle/history requires an independent durable copy of the complete
|
|
causal recovery chain and referenced payloads before ACK for
|
|
host-loss RPO0. That replication/acknowledgment contract and its latency are a
|
|
World0 launch gate. Overlaying isolated lifecycle records on older resource state
|
|
can duplicate biomass or lose offspring. Restoring an older world must not
|
|
resurrect organisms or erase history; if the required causal state cannot be
|
|
reconciled, keep the world unavailable.
|
|
|
|
## Backup and clone protocol
|
|
|
|
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
|
|
referenced by each recovery point. A database backup without those objects is not
|
|
a world backup. Manifest fields: world identity, simulation/generator/world/codec
|
|
versions, DB recovery target, region watermarks, object inventory/strong digests,
|
|
and binaries/container artifacts needed for replay. Encrypt off-host copies,
|
|
separate credentials and test recovery-key access.
|
|
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
|
|
|
|
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
|
|
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
|
|
history does not expire with backup retention. GC respects all clone/backup pins
|
|
and complete replay chains.
|
|
|
|
1. First implementation (ADR-0074) needs no admission barrier: one exported
|
|
database snapshot is the cut, and objects are immutable and verified before
|
|
commit, so copying them after the snapshot is consistent. A barrier returns
|
|
only if state ever spans more than one database.
|
|
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
|
|
transfer state, region watermarks and versions into the manifest.
|
|
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
|
|
4. Assign isolated world/runtime identities while preserving source provenance;
|
|
replace credentials, buckets and subjects; disable production integrations.
|
|
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
|
|
and hashes before readiness. Measure duration and missing-object behavior.
|
|
6. Record actual restore evidence. Upload success or a scheduled job is not a
|
|
restore test; alerts and recovery credentials must also work.
|
|
|
|
Actual recovery fences old database/cluster writers before routing cutover and
|
|
prevents the old environment returning as competing authority. Resolve ambiguous
|
|
commits by command ID/request digest; timeout is not rollback. Retain old
|
|
simulation binaries and generators needed by historical state.
|
|
|
|
## World0 protection and drills
|
|
|
|
Separate Argo permissions/environment credentials; Retain reclaim policy where
|
|
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
|
|
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
|
|
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
|
|
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
|
|
|
|
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
|
|
rejects World0. Drill worker termination in clones first: upload failure,
|
|
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
|
|
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
|
|
permanent history and measured recovery. Never substitute namespace deletion for
|
|
a worker recovery test.
|
|
|
|
## Host-loss recovery in dev/stage (M8.8, ADR-0075)
|
|
|
|
1. Declare the loss and the replacement database (same PostgreSQL major version; the
|
|
environment's migrator and runtime roles recreated).
|
|
2. Restore the newest verified bundle in place, as the migrator:
|
|
`restore-world.sh --bundle B --objects NEW_VOLUME --in-place --runtime-role ROLE --accept-rewind`.
|
|
Record every `REWIND` line: acknowledgements after those ticks are undone.
|
|
Schema-51 bundles also fork the database history domain before readmission;
|
|
retain the new/parent domains and exact `history_domain_cuts` with the restore
|
|
claim. External evidence keys worlds by `(history_domain, world_id)`.
|
|
Older bundles have no original domain: later migration records adoption only.
|
|
Direct database copies outside this workflow can copy a domain; see
|
|
[ADR-0155](../adr/0155-database-history-domain.md).
|
|
3. Point the environment's connection Secret and object volume at the replacement,
|
|
then sync the environment's Argo application; the host resumes under a newer epoch.
|
|
4. The moment the old database is reachable (or at once, if it never went away):
|
|
`fence-database.sh --superseded-by <label> --confirm` as its owner. Record the
|
|
printed final heads; the difference from the `REWIND` ticks is the measured loss.
|
|
5. Never use this for World0: its launch gate forbids a rewind of acknowledged outcomes.
|