# Infrastructure and Kubernetes topology Status: production-shaped proposal, not deployed workloads. Initial hardware is one R730xd, with about 128GB RAM in the brief and about 500GB added since. The cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a separate PostgreSQL VM were inventoried in M8.1; see [INVENTORY.md](../infrastructure/INVENTORY.md). Replicas on one physical host do not create an independent failure domain. ```mermaid flowchart TB Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision] Git --> Argo[Argo CD environment applications] CI[Build / tests / analysis / sanitizers] --> Registry Workflows[Argo Workflows soak / clone / migration] --> Git subgraph Cluster[Kubernetes cluster initially one physical node] subgraph Dev[evolution-dev] D[Isolated development workloads / quotas] end subgraph Stage[evolution-stage] S[Cloned world / validation / quotas] end subgraph World[evolution-world0] API[Modular backend and director] R[Disposable region workers] B[Background ecology / terrain jobs when needed] PG[(PostgreSQL and durable volumes)] OBJ[(S3-compatible payload storage)] N[JetStream if adopted] API --> PG R --> PG R --> OBJ B --> PG PG --> N end Obs[Prometheus / Grafana / Loki; Otel where justified] end PG --> Off[Independent backup/replication failure domain] OBJ --> Off ``` Deploy one modular backend initially: accounts/identity adapters, history/genome queries and director modules. Region processes scale separately for latency and failure isolation. Background workers split only for measurable workload/lifecycle needs. No Redis unless transient caching/session/routing measurements justify it; no Kafka by default. Evaluate JetStream before adding a bus. ## Scheduling, storage and security | Concern | Required shape | |---|---| | Region pods | Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes | | Stateful stores | Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests | | Future nodes | Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable | | Network | Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement | | Identity | Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants | | Pod security | Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas | | Secrets | External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs | | Termination | Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary | CPU throttling can miss ticks; measure requests/limits rather than copying web service settings. Autoscaling uses tick debt/latency and safe region moves, not unbounded replica creation. Pod disruption budgets help planned maintenance but cannot protect against chassis loss. Local volumes constrain scheduling; adding compute nodes does not automatically make stateful storage portable. Host memory is a capacity envelope, not all available simulation memory. Inventory OS/ Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery headroom before region budgets. Start measured namespace quotas and leave explicit recovery/compaction headroom. No unverified entity/player-per-node promises. ## Delivery and upgrades Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks → measured benchmark report → container build/scan/SBOM/signature → Harbor digest → reviewed dev GitOps change → soak → staging clone/migration → manual World0 promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks or deployments already ran. World0 uses separate application/project permissions, disabled automatic prune/ self-heal for destructive state transitions, retained PVCs and explicit promotion. Routine developer jobs cannot delete namespace/data or change world versions. World-law upgrades use a consistent world barrier, not adjacent different-rule canaries. Keep infra/environment values separable so later repository extraction does not require rewriting simulation. ## Observability and adoption gates Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth, queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients, birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels. Alert on unavailable recovery paths, disk capacity/latency, missing backups, restarts, packet loss and prolonged tick debt; test alerts through fault exercises. S3 is the storage API boundary; backend selection remains open. MinIO's official repository is archived/no longer maintained, so do not assume it is an acceptable new production default. Evaluate maintained alternatives for license, operations, S3 semantics, durability and independent restore. [Official MinIO repository](https://github.com/minio/minio). Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress, Harbor/Argo integration, least privilege, resource budgets and off-host recovery. Then kill a region pod and demonstrate automatic fenced restore. ADR-0020 adds a stateless M0 verification Job in evolution-dev through the existing Gitea/Harbor setup. It is not a persistent region or evidence of the M8 recovery gate. Templates with unresolved destination/image markers must not be synced.