Initial commit: architecture docs and headless engineering proof
Aeternalis Evolution, a custom-engine persistent evolutionary MMO. This first commit establishes the project skeleton: - Architecture and design docs: project vision, system overview, engineering constitution, roadmap, 19 ADRs, and the simulation, world, network, persistence and infrastructure architectures. - Headless engineering proof: deterministic synthetic particles, component lifecycle, explicit PRNG streams, canonical snapshots and restored execution. - Build and tooling: CMake 3.28+ / Ninja presets, C++23 feature check, clang-format and clang-tidy config, format check script, Windows check script, and a Docker build context. - Tests: unit tests, benchmarks, a snapshot fuzz target and CLI integration wiring. No gameplay, renderer, network service or running deployment yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,69 @@
|
||||
# Operations and disaster recovery
|
||||
|
||||
Status: proposal requiring measured acceptance before World0. No deployment,
|
||||
backup or distributed restore has been exercised in this foundation.
|
||||
|
||||
| Failure | Proposed objective / gate |
|
||||
|---|---|
|
||||
| Worker lost, stores healthy | Zero lost committed decisions; fixture ready within 60 seconds |
|
||||
| Uncommitted motion | Experiment with at most one second rollback; stop admission before backlog exceeds budget |
|
||||
| Corrupt/missing state | Fail readiness; restore verified chain or quarantine, never initialize an empty replacement |
|
||||
| Host destruction | RTO experiment 24 hours once independent storage/hardware exists; RPO gate below |
|
||||
|
||||
Targets are not observed results. One host cannot promise availability or zero
|
||||
data loss after its destruction. A proposed 15-minute backup RPO for mutable
|
||||
world state cannot guarantee every acknowledged death/birth/ancestry record.
|
||||
Permanent lifecycle/history requires an independent durable copy of the complete
|
||||
causal recovery chain and referenced payloads before ACK for
|
||||
host-loss RPO0. That replication/acknowledgment contract and its latency are a
|
||||
World0 launch gate. Overlaying isolated lifecycle records on older resource state
|
||||
can duplicate biomass or lose offspring. Restoring an older world must not
|
||||
resurrect organisms or erase history; if the required causal state cannot be
|
||||
reconciled, keep the world unavailable.
|
||||
|
||||
## Backup and clone protocol
|
||||
|
||||
Back up PostgreSQL base data plus continuous WAL. Preserve every immutable object
|
||||
referenced by each recovery point. A database backup without those objects is not
|
||||
a world backup. Manifest fields: world identity, simulation/generator/world/codec
|
||||
versions, DB recovery target, region watermarks, object inventory/strong digests,
|
||||
and binaries/container artifacts needed for replay. Encrypt off-host copies,
|
||||
separate credentials and test recovery-key access.
|
||||
[PostgreSQL continuous archiving](https://www.postgresql.org/docs/17/continuous-archiving.html).
|
||||
|
||||
Retention experiment: 30 daily, 12 weekly, 12 monthly verified manifests plus
|
||||
pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
|
||||
history does not expire with backup retention. GC respects all clone/backup pins
|
||||
and complete replay chains.
|
||||
|
||||
1. First implementation uses a coordinated admission/transfer barrier, avoiding
|
||||
an unfinished distributed-snapshot algorithm.
|
||||
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
|
||||
transfer state, region watermarks and versions into the manifest.
|
||||
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
|
||||
4. Assign isolated world/runtime identities while preserving source provenance;
|
||||
replace credentials, buckets and subjects; disable production integrations.
|
||||
5. Replay and verify unique ownership, conservation, terminal deaths, ancestry
|
||||
and hashes before readiness. Measure duration and missing-object behavior.
|
||||
6. Record actual restore evidence. Upload success or a scheduled job is not a
|
||||
restore test; alerts and recovery credentials must also work.
|
||||
|
||||
Actual recovery fences old database/cluster writers before routing cutover and
|
||||
prevents the old environment returning as competing authority. Resolve ambiguous
|
||||
commits by command ID/request digest; timeout is not rollback. Retain old
|
||||
simulation binaries and generators needed by historical state.
|
||||
|
||||
## World0 protection and drills
|
||||
|
||||
Separate Argo permissions/environment credentials; Retain reclaim policy where
|
||||
supported; protected PVC and namespace/data deletion via RBAC/admission. Test
|
||||
pruning and parent-app deletion separately. One prune flag is not a full safeguard.
|
||||
[Kubernetes volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/),
|
||||
[Argo sync options](https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/).
|
||||
|
||||
CI has no World0 data credentials. Cleanup requires exact dev/clone identity and
|
||||
rejects World0. Drill worker termination in clones first: upload failure,
|
||||
commit-before-ACK crash, stale owner resume, missing object, disk full, broker
|
||||
redelivery and each handoff stage. Verify one owner, no duplicated outcomes,
|
||||
permanent history and measured recovery. Never substitute namespace deletion for
|
||||
a worker recovery test.
|
||||
Reference in New Issue
Block a user