Aeternalis Evolution, a custom-engine persistent evolutionary MMO. This first commit establishes the project skeleton: - Architecture and design docs: project vision, system overview, engineering constitution, roadmap, 19 ADRs, and the simulation, world, network, persistence and infrastructure architectures. - Headless engineering proof: deterministic synthetic particles, component lifecycle, explicit PRNG streams, canonical snapshots and restored execution. - Build and tooling: CMake 3.28+ / Ninja presets, C++23 feature check, clang-format and clang-tidy config, format check script, Windows check script, and a Docker build context. - Tests: unit tests, benchmarks, a snapshot fuzz target and CLI integration wiring. No gameplay, renderer, network service or running deployment yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
70 lines
4.1 KiB
Markdown
70 lines
4.1 KiB
Markdown
# Operations and disaster recovery
|
|
|
|
Status: proposal requiring measured acceptance before World0. No deployment,
|
|
backup or distributed restore has been exercised in this foundation.
|
|
|
|
| Failure | Proposed objective / gate |
|
|
|---|---|
|
|
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
|
|
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
|
|
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
|
|
| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below |
|
|
|
|
Targets are not observed results. One host cannot promise availability or zero
|
|
data loss after its destruction. A proposed 15-minute backup RPO for mutable
|
|
world state cannot guarantee every acknowledged death/birth/ancestry record.
|
|
Permanent lifecycle/history requires an independent durable copy of the complete
|
|
causal recovery chain and referenced payloads before ACK for
|
|
host-loss RPO0. That replication/acknowledgment contract and its latency are a
|
|
World0 launch gate. Overlaying isolated lifecycle records on older resource state
|
|
can duplicate biomass or lose offspring. Restoring an older world must not
|
|
resurrect organisms or erase history; if the required causal state cannot be
|
|
reconciled, keep the world unavailable.
|
|
|
|
## Backup and clone protocol
|
|
|
|
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
|
|
referenced by each recovery point. A database backup without those objects is not
|
|
a world backup. Manifest fields: world identity, simulation/generator/world/codec
|
|
versions, DB recovery target, region watermarks, object inventory/strong digests,
|
|
and binaries/container artifacts needed for replay. Encrypt off-host copies,
|
|
separate credentials and test recovery-key access.
|
|
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
|
|
|
|
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
|
|
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
|
|
history does not expire with backup retention. GC respects all clone/backup pins
|
|
and complete replay chains.
|
|
|
|
1. First implementation uses a coordinated admission/transfer barrier, avoiding
|
|
an unfinished distributed-snapshot algorithm.
|
|
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
|
|
transfer state, region watermarks and versions into the manifest.
|
|
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
|
|
4. Assign isolated world/runtime identities while preserving source provenance;
|
|
replace credentials, buckets and subjects; disable production integrations.
|
|
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
|
|
and hashes before readiness. Measure duration and missing-object behavior.
|
|
6. Record actual restore evidence. Upload success or a scheduled job is not a
|
|
restore test; alerts and recovery credentials must also work.
|
|
|
|
Actual recovery fences old database/cluster writers before routing cutover and
|
|
prevents the old environment returning as competing authority. Resolve ambiguous
|
|
commits by command ID/request digest; timeout is not rollback. Retain old
|
|
simulation binaries and generators needed by historical state.
|
|
|
|
## World0 protection and drills
|
|
|
|
Separate Argo permissions/environment credentials; Retain reclaim policy where
|
|
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
|
|
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
|
|
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
|
|
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
|
|
|
|
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
|
|
rejects World0. Drill worker termination in clones first: upload failure,
|
|
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
|
|
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
|
|
permanent history and measured recovery. Never substitute namespace deletion for
|
|
a worker recovery test.
|