Files
aevolution/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md
dingoandClaude Opus 5.5 18708340c3 Complete M8.1 with per-worker storage probe results
Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:33:04 -04:00

6.1 KiB

Infrastructure and Kubernetes topology

Status: production-shaped proposal, not deployed workloads. Initial hardware is one R730xd, with about 128GB RAM in the brief and about 500GB added since. The cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a separate PostgreSQL VM were inventoried in M8.1; see INVENTORY.md. Replicas on one physical host do not create an independent failure domain.

flowchart TB
  Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision]
  Git --> Argo[Argo CD environment applications]
  CI[Build / tests / analysis / sanitizers] --> Registry
  Workflows[Argo Workflows soak / clone / migration] --> Git
  subgraph Cluster[Kubernetes cluster initially one physical node]
    subgraph Dev[evolution-dev]
      D[Isolated development workloads / quotas]
    end
    subgraph Stage[evolution-stage]
      S[Cloned world / validation / quotas]
    end
    subgraph World[evolution-world0]
      API[Modular backend and director]
      R[Disposable region workers]
      B[Background ecology / terrain jobs when needed]
      PG[(PostgreSQL and durable volumes)]
      OBJ[(S3-compatible payload storage)]
      N[JetStream if adopted]
      API --> PG
      R --> PG
      R --> OBJ
      B --> PG
      PG --> N
    end
    Obs[Prometheus / Grafana / Loki; Otel where justified]
  end
  PG --> Off[Independent backup/replication failure domain]
  OBJ --> Off

Deploy one modular backend initially: accounts/identity adapters, history/genome queries and director modules. Region processes scale separately for latency and failure isolation. Background workers split only for measurable workload/lifecycle needs. No Redis unless transient caching/session/routing measurements justify it; no Kafka by default. Evaluate JetStream before adding a bus.

Scheduling, storage and security

Concern Required shape
Region pods Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes
Stateful stores Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests
Future nodes Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable
Network Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement
Identity Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants
Pod security Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas
Secrets External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs
Termination Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary

CPU throttling can miss ticks; measure requests/limits rather than copying web service settings. Autoscaling uses tick debt/latency and safe region moves, not unbounded replica creation. Pod disruption budgets help planned maintenance but cannot protect against chassis loss. Local volumes constrain scheduling; adding compute nodes does not automatically make stateful storage portable.

Host memory is a capacity envelope, not all available simulation memory. Inventory OS/ Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery headroom before region budgets. Start measured namespace quotas and leave explicit recovery/compaction headroom. No unverified entity/player-per-node promises.

Delivery and upgrades

Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks → measured benchmark report → container build/scan/SBOM/signature → Harbor digest → reviewed dev GitOps change → soak → staging clone/migration → manual World0 promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks or deployments already ran.

World0 uses separate application/project permissions, disabled automatic prune/ self-heal for destructive state transitions, retained PVCs and explicit promotion. Routine developer jobs cannot delete namespace/data or change world versions. World-law upgrades use a consistent world barrier, not adjacent different-rule canaries. Keep infra/environment values separable so later repository extraction does not require rewriting simulation.

Observability and adoption gates

Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth, queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients, birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels. Alert on unavailable recovery paths, disk capacity/latency, missing backups, restarts, packet loss and prolonged tick debt; test alerts through fault exercises.

S3 is the storage API boundary; backend selection remains open. MinIO's official repository is archived/no longer maintained, so do not assume it is an acceptable new production default. Evaluate maintained alternatives for license, operations, S3 semantics, durability and independent restore. Official MinIO repository.

Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress, Harbor/Argo integration, least privilege, resource budgets and off-host recovery. Then kill a region pod and demonstrate automatic fenced restore. ADR-0020 adds a stateless M0 verification Job in evolution-dev through the existing Gitea/Harbor setup. It is not a persistent region or evidence of the M8 recovery gate. Templates with unresolved destination/image markers must not be synced.