5.9 KiB
Infrastructure and Kubernetes topology
Status: production-shaped proposal, not deployed workloads. Initial hardware is one R730xd with approximately 128GB RAM. Kubernetes is available per brief; actual CPU/storage/CNI/CSI/backup/network details remain uninspected. Replicas on one physical host do not create an independent failure domain.
flowchart TB
Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision]
Git --> Argo[Argo CD environment applications]
CI[Build / tests / analysis / sanitizers] --> Registry
Workflows[Argo Workflows soak / clone / migration] --> Git
subgraph Cluster[Kubernetes cluster initially one physical node]
subgraph Dev[evolution-dev]
D[Isolated development workloads / quotas]
end
subgraph Stage[evolution-stage]
S[Cloned world / validation / quotas]
end
subgraph World[evolution-world0]
API[Modular backend and director]
R[Disposable region workers]
B[Background ecology / terrain jobs when needed]
PG[(PostgreSQL and durable volumes)]
OBJ[(S3-compatible payload storage)]
N[JetStream if adopted]
API --> PG
R --> PG
R --> OBJ
B --> PG
PG --> N
end
Obs[Prometheus / Grafana / Loki; Otel where justified]
end
PG --> Off[Independent backup/replication failure domain]
OBJ --> Off
Deploy one modular backend initially: accounts/identity adapters, history/genome queries and director modules. Region processes scale separately for latency and failure isolation. Background workers split only for measurable workload/lifecycle needs. No Redis unless transient caching/session/routing measurements justify it; no Kafka by default. Evaluate JetStream before adding a bus.
Scheduling, storage and security
| Concern | Required shape |
|---|---|
| Region pods | Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes |
| Stateful stores | Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests |
| Future nodes | Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable |
| Network | Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement |
| Identity | Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants |
| Pod security | Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas |
| Secrets | External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs |
| Termination | Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary |
CPU throttling can miss ticks; measure requests/limits rather than copying web service settings. Autoscaling uses tick debt/latency and safe region moves, not unbounded replica creation. Pod disruption budgets help planned maintenance but cannot protect against chassis loss. Local volumes constrain scheduling; adding compute nodes does not automatically make stateful storage portable.
128GB is a capacity envelope, not all available simulation memory. Inventory OS/ Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery headroom before region budgets. Start measured namespace quotas and leave explicit recovery/compaction headroom. No unverified entity/player-per-node promises.
Delivery and upgrades
Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks → measured benchmark report → container build/scan/SBOM/signature → Harbor digest → reviewed dev GitOps change → soak → staging clone/migration → manual World0 promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks or deployments already ran.
World0 uses separate application/project permissions, disabled automatic prune/ self-heal for destructive state transitions, retained PVCs and explicit promotion. Routine developer jobs cannot delete namespace/data or change world versions. World-law upgrades use a consistent world barrier, not adjacent different-rule canaries. Keep infra/environment values separable so later repository extraction does not require rewriting simulation.
Observability and adoption gates
Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth, queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients, birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels. Alert on unavailable recovery paths, disk capacity/latency, missing backups, restarts, packet loss and prolonged tick debt; test alerts through fault exercises.
S3 is the storage API boundary; backend selection remains open. MinIO's official repository is archived/no longer maintained, so do not assume it is an acceptable new production default. Evaluate maintained alternatives for license, operations, S3 semantics, durability and independent restore. Official MinIO repository.
Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress, Harbor/Argo integration, least privilege, resource budgets and off-host recovery. Then kill a region pod and demonstrate automatic fenced restore. ADR-0020 adds a stateless M0 verification Job in evolution-dev through the existing Gitea/Harbor setup. It is not a persistent region or evidence of the M8 recovery gate. Templates with unresolved destination/image markers must not be synced.