Files
aevolution/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md
T
dingoandClaude Opus 5.5 18708340c3 Complete M8.1 with per-worker storage probe results
Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:33:04 -04:00

109 lines
6.1 KiB
Markdown

# Infrastructure and Kubernetes topology
Status: production-shaped proposal, not deployed workloads. Initial hardware is
one R730xd, with about 128GB RAM in the brief and about 500GB added since. The
cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a
separate PostgreSQL VM were inventoried in M8.1; see
[INVENTORY.md](../infrastructure/INVENTORY.md). Replicas on one
physical host do not create an independent failure domain.
```mermaid
flowchart TB
Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision]
Git --> Argo[Argo CD environment applications]
CI[Build / tests / analysis / sanitizers] --> Registry
Workflows[Argo Workflows soak / clone / migration] --> Git
subgraph Cluster[Kubernetes cluster initially one physical node]
subgraph Dev[evolution-dev]
D[Isolated development workloads / quotas]
end
subgraph Stage[evolution-stage]
S[Cloned world / validation / quotas]
end
subgraph World[evolution-world0]
API[Modular backend and director]
R[Disposable region workers]
B[Background ecology / terrain jobs when needed]
PG[(PostgreSQL and durable volumes)]
OBJ[(S3-compatible payload storage)]
N[JetStream if adopted]
API --> PG
R --> PG
R --> OBJ
B --> PG
PG --> N
end
Obs[Prometheus / Grafana / Loki; Otel where justified]
end
PG --> Off[Independent backup/replication failure domain]
OBJ --> Off
```
Deploy one modular backend initially: accounts/identity adapters, history/genome
queries and director modules. Region processes scale separately for latency and
failure isolation. Background workers split only for measurable workload/lifecycle
needs. No Redis unless transient caching/session/routing measurements justify it;
no Kafka by default. Evaluate JetStream before adding a bus.
## Scheduling, storage and security
| Concern | Required shape |
|---|---|
| Region pods | Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes |
| Stateful stores | Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests |
| Future nodes | Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable |
| Network | Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement |
| Identity | Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants |
| Pod security | Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas |
| Secrets | External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs |
| Termination | Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary |
CPU throttling can miss ticks; measure requests/limits rather than copying web
service settings. Autoscaling uses tick debt/latency and safe region moves, not
unbounded replica creation. Pod disruption budgets help planned maintenance but
cannot protect against chassis loss. Local volumes constrain scheduling; adding
compute nodes does not automatically make stateful storage portable.
Host memory is a capacity envelope, not all available simulation memory. Inventory OS/
Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery
headroom before region budgets. Start measured namespace quotas and leave explicit
recovery/compaction headroom. No unverified entity/player-per-node promises.
## Delivery and upgrades
Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec
checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks
→ measured benchmark report → container build/scan/SBOM/signature → Harbor digest
→ reviewed dev GitOps change → soak → staging clone/migration → manual World0
promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak
jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks
or deployments already ran.
World0 uses separate application/project permissions, disabled automatic prune/
self-heal for destructive state transitions, retained PVCs and explicit promotion.
Routine developer jobs cannot delete namespace/data or change world versions.
World-law upgrades use a consistent world barrier, not adjacent different-rule
canaries. Keep infra/environment values separable so later repository extraction
does not require rewriting simulation.
## Observability and adoption gates
Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth,
queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients,
birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel
traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels.
Alert on unavailable recovery paths, disk capacity/latency, missing backups,
restarts, packet loss and prolonged tick debt; test alerts through fault exercises.
S3 is the storage API boundary; backend selection remains open. MinIO's official
repository is archived/no longer maintained, so do not assume it is an acceptable
new production default. Evaluate maintained alternatives for license, operations,
S3 semantics, durability and independent restore. [Official MinIO repository](https://github.com/minio/minio).
Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress,
Harbor/Argo integration, least privilege, resource budgets and off-host recovery.
Then kill a region pod and demonstrate automatic fenced restore. ADR-0020 adds
a stateless M0 verification Job in evolution-dev through the existing Gitea/Harbor
setup. It is not a persistent region or evidence of the M8 recovery gate.
Templates with unresolved destination/image markers must not be synced.