Files
aevolution/docs/infrastructure/OPERATIONS.md
T
dingoandClaude Opus 5 d5b4bd068e Initial commit: architecture docs and headless engineering proof
Aeternalis Evolution, a custom-engine persistent evolutionary MMO.

This first commit establishes the project skeleton:

- Architecture and design docs: project vision, system overview,
  engineering constitution, roadmap, 19 ADRs, and the simulation,
  world, network, persistence and infrastructure architectures.
- Headless engineering proof: deterministic synthetic particles,
  component lifecycle, explicit PRNG streams, canonical snapshots
  and restored execution.
- Build and tooling: CMake 3.28+ / Ninja presets, C++23 feature
  check, clang-format and clang-tidy config, format check script,
  Windows check script, and a Docker build context.
- Tests: unit tests, benchmarks, a snapshot fuzz target and CLI
  integration wiring.

No gameplay, renderer, network service or running deployment yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 15:44:41 -04:00

70 lines
4.1 KiB
Markdown

# Operations and disaster recovery
Status: proposal requiring measured acceptance before World0. No deployment,
backup or distributed restore has been exercised in this foundation.
| Failure | Proposed objective / gate |
|---|---|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below |
Targets are not observed results. One host cannot promise availability or zero
data loss after its destruction. A proposed 15-minute backup RPO for mutable
world state cannot guarantee every acknowledged death/birth/ancestry record.
Permanent lifecycle/history requires an independent durable copy of the complete
causal recovery chain and referenced payloads before ACK for
host-loss RPO0. That replication/acknowledgment contract and its latency are a
World0 launch gate. Overlaying isolated lifecycle records on older resource state
can duplicate biomass or lose offspring. Restoring an older world must not
resurrect organisms or erase history; if the required causal state cannot be
reconciled, keep the world unavailable.
## Backup and clone protocol
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
referenced by each recovery point. A database backup without those objects is not
a world backup. Manifest fields: world identity, simulation/generator/world/codec
versions, DB recovery target, region watermarks, object inventory/strong digests,
and binaries/container artifacts needed for replay. Encrypt off-host copies,
separate credentials and test recovery-key access.
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
history does not expire with backup retention. GC respects all clone/backup pins
and complete replay chains.
1. First implementation uses a coordinated admission/transfer barrier, avoiding
an unfinished distributed-snapshot algorithm.
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
transfer state, region watermarks and versions into the manifest.
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
4. Assign isolated world/runtime identities while preserving source provenance;
replace credentials, buckets and subjects; disable production integrations.
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
and hashes before readiness. Measure duration and missing-object behavior.
6. Record actual restore evidence. Upload success or a scheduled job is not a
restore test; alerts and recovery credentials must also work.
Actual recovery fences old database/cluster writers before routing cutover and
prevents the old environment returning as competing authority. Resolve ambiguous
commits by command ID/request digest; timeout is not rollback. Retain old
simulation binaries and generators needed by historical state.
## World0 protection and drills
Separate Argo permissions/environment credentials; Retain reclaim policy where
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
rejects World0. Drill worker termination in clones first: upload failure,
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
permanent history and measured recovery. Never substitute namespace deletion for
a worker recovery test.