Files
aevolution/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md
T
dingoandClaude Opus 5 d5b4bd068e Initial commit: architecture docs and headless engineering proof
Aeternalis Evolution, a custom-engine persistent evolutionary MMO.

This first commit establishes the project skeleton:

- Architecture and design docs: project vision, system overview,
  engineering constitution, roadmap, 19 ADRs, and the simulation,
  world, network, persistence and infrastructure architectures.
- Headless engineering proof: deterministic synthetic particles,
  component lifecycle, explicit PRNG streams, canonical snapshots
  and restored execution.
- Build and tooling: CMake 3.28+ / Ninja presets, C++23 feature
  check, clang-format and clang-tidy config, format check script,
  Windows check script, and a Docker build context.
- Tests: unit tests, benchmarks, a snapshot fuzz target and CLI
  integration wiring.

No gameplay, renderer, network service or running deployment yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-14 15:44:41 -04:00

5.8 KiB

Infrastructure and Kubernetes topology

Status: production-shaped proposal, not deployed workloads. Initial hardware is one R730xd with approximately 128GB RAM. Kubernetes is available per brief; actual CPU/storage/CNI/CSI/backup/network details remain uninspected. Replicas on one physical host do not create an independent failure domain.

flowchart TB
  Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision]
  Git --> Argo[Argo CD environment applications]
  CI[Build / tests / analysis / sanitizers] --> Registry
  Workflows[Argo Workflows soak / clone / migration] --> Git
  subgraph Cluster[Kubernetes cluster initially one physical node]
    subgraph Dev[evolution-dev]
      D[Isolated development workloads / quotas]
    end
    subgraph Stage[evolution-stage]
      S[Cloned world / validation / quotas]
    end
    subgraph World[evolution-world0]
      API[Modular backend and director]
      R[Disposable region workers]
      B[Background ecology / terrain jobs when needed]
      PG[(PostgreSQL and durable volumes)]
      OBJ[(S3-compatible payload storage)]
      N[JetStream if adopted]
      API --> PG
      R --> PG
      R --> OBJ
      B --> PG
      PG --> N
    end
    Obs[Prometheus / Grafana / Loki; Otel where justified]
  end
  PG --> Off[Independent backup/replication failure domain]
  OBJ --> Off

Deploy one modular backend initially: accounts/identity adapters, history/genome queries and director modules. Region processes scale separately for latency and failure isolation. Background workers split only for measurable workload/lifecycle needs. No Redis unless transient caching/session/routing measurements justify it; no Kafka by default. Evaluate JetStream before adding a bus.

Scheduling, storage and security

Concern Required shape
Region pods Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes
Stateful stores Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests
Future nodes Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable
Network Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement
Identity Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants
Pod security Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas
Secrets External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs
Termination Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary

CPU throttling can miss ticks; measure requests/limits rather than copying web service settings. Autoscaling uses tick debt/latency and safe region moves, not unbounded replica creation. Pod disruption budgets help planned maintenance but cannot protect against chassis loss. Local volumes constrain scheduling; adding compute nodes does not automatically make stateful storage portable.

128GB is a capacity envelope, not all available simulation memory. Inventory OS/ Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery headroom before region budgets. Start measured namespace quotas and leave explicit recovery/compaction headroom. No unverified entity/player-per-node promises.

Delivery and upgrades

Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks → measured benchmark report → container build/scan/SBOM/signature → Harbor digest → reviewed dev GitOps change → soak → staging clone/migration → manual World0 promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks or deployments already ran.

World0 uses separate application/project permissions, disabled automatic prune/ self-heal for destructive state transitions, retained PVCs and explicit promotion. Routine developer jobs cannot delete namespace/data or change world versions. World-law upgrades use a consistent world barrier, not adjacent different-rule canaries. Keep infra/environment values separable so later repository extraction does not require rewriting simulation.

Observability and adoption gates

Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth, queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients, birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels. Alert on unavailable recovery paths, disk capacity/latency, missing backups, restarts, packet loss and prolonged tick debt; test alerts through fault exercises.

S3 is the storage API boundary; backend selection remains open. MinIO's official repository is archived/no longer maintained, so do not assume it is an acceptable new production default. Evaluate maintained alternatives for license, operations, S3 semantics, durability and independent restore. Official MinIO repository.

Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress, Harbor/Argo integration, least privilege, resource budgets and off-host recovery. Then kill a region pod and demonstrate automatic fenced restore. No manifests with fabricated endpoints, secrets or storage classes are deployed in M0.