Record the maintainer-run local-path probe on all three workers (1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication, 110-255 MB/s sequential), well inside the lease deadlines, and flag worker-3's 87% full root filesystem. Apply the maintainer's corrections: about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM outside Kubernetes, and manual-only inventory collection. Storage figures stay provisional until the planned RAID change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
109 lines
6.1 KiB
Markdown
109 lines
6.1 KiB
Markdown
# Infrastructure and Kubernetes topology
|
|
|
|
Status: production-shaped proposal, not deployed workloads. Initial hardware is
|
|
one R730xd, with about 128GB RAM in the brief and about 500GB added since. The
|
|
cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a
|
|
separate PostgreSQL VM were inventoried in M8.1; see
|
|
[INVENTORY.md](../infrastructure/INVENTORY.md). Replicas on one
|
|
physical host do not create an independent failure domain.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
Registry[Harbor immutable images] --> Git[Reviewed deployment Git revision]
|
|
Git --> Argo[Argo CD environment applications]
|
|
CI[Build / tests / analysis / sanitizers] --> Registry
|
|
Workflows[Argo Workflows soak / clone / migration] --> Git
|
|
subgraph Cluster[Kubernetes cluster initially one physical node]
|
|
subgraph Dev[evolution-dev]
|
|
D[Isolated development workloads / quotas]
|
|
end
|
|
subgraph Stage[evolution-stage]
|
|
S[Cloned world / validation / quotas]
|
|
end
|
|
subgraph World[evolution-world0]
|
|
API[Modular backend and director]
|
|
R[Disposable region workers]
|
|
B[Background ecology / terrain jobs when needed]
|
|
PG[(PostgreSQL and durable volumes)]
|
|
OBJ[(S3-compatible payload storage)]
|
|
N[JetStream if adopted]
|
|
API --> PG
|
|
R --> PG
|
|
R --> OBJ
|
|
B --> PG
|
|
PG --> N
|
|
end
|
|
Obs[Prometheus / Grafana / Loki; Otel where justified]
|
|
end
|
|
PG --> Off[Independent backup/replication failure domain]
|
|
OBJ --> Off
|
|
```
|
|
|
|
Deploy one modular backend initially: accounts/identity adapters, history/genome
|
|
queries and director modules. Region processes scale separately for latency and
|
|
failure isolation. Background workers split only for measurable workload/lifecycle
|
|
needs. No Redis unless transient caching/session/routing measurements justify it;
|
|
no Kafka by default. Evaluate JetStream before adding a bus.
|
|
|
|
## Scheduling, storage and security
|
|
|
|
| Concern | Required shape |
|
|
|---|---|
|
|
| Region pods | Stateless image; memory state recovered through fenced ownership; requests/limits, bounded queues, startup/readiness/liveness probes |
|
|
| Stateful stores | Supported operator or reviewed StatefulSet strategy; persistent volumes, retention/reclaim safeguards, backup/restore tests |
|
|
| Future nodes | Labels for realtime, biosimulation, storage/database and optional GPU; affinity/taints and topology spread; no required separation that makes one-node dev unschedulable |
|
|
| Network | Default-deny namespace policies with explicit DNS/service flows; protected DB/storage/metrics; external realtime UDP exposure chosen with CNI/LB measurement |
|
|
| Identity | Dedicated service accounts; disable unnecessary token mounts; narrowly scoped RBAC and storage/DB grants |
|
|
| Pod security | Non-root, read-only rootfs, dropped capabilities, seccomp, bounded writable scratch and resource quotas |
|
|
| Secrets | External/cluster secret management chosen by operator; no plaintext repo secrets or credentials in images/logs |
|
|
| Termination | Stop admission, freeze/drain, commit recoverable state and release lease; grace budget measured; crash recovery remains necessary |
|
|
|
|
CPU throttling can miss ticks; measure requests/limits rather than copying web
|
|
service settings. Autoscaling uses tick debt/latency and safe region moves, not
|
|
unbounded replica creation. Pod disruption budgets help planned maintenance but
|
|
cannot protect against chassis loss. Local volumes constrain scheduling; adding
|
|
compute nodes does not automatically make stateful storage portable.
|
|
|
|
Host memory is a capacity envelope, not all available simulation memory. Inventory OS/
|
|
Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery
|
|
headroom before region budgets. Start measured namespace quotas and leave explicit
|
|
recovery/compaction headroom. No unverified entity/player-per-node promises.
|
|
|
|
## Delivery and upgrades
|
|
|
|
Commit → configure/build/warnings → format/tidy → unit/property/determinism/codec
|
|
checks → ASan/UBSan and separate TSan → applicable integration/ecology/load checks
|
|
→ measured benchmark report → container build/scan/SBOM/signature → Harbor digest
|
|
→ reviewed dev GitOps change → soak → staging clone/migration → manual World0
|
|
promotion. Argo Workflows is the planned orchestrator for clone/accelerated/soak
|
|
jobs; Argo CD reconciles reviewed manifests. CI skeleton does not imply checks
|
|
or deployments already ran.
|
|
|
|
World0 uses separate application/project permissions, disabled automatic prune/
|
|
self-heal for destructive state transitions, retained PVCs and explicit promotion.
|
|
Routine developer jobs cannot delete namespace/data or change world versions.
|
|
World-law upgrades use a consistent world barrier, not adjacent different-rule
|
|
canaries. Keep infra/environment values separable so later repository extraction
|
|
does not require rewriting simulation.
|
|
|
|
## Observability and adoption gates
|
|
|
|
Prometheus/Grafana: tick histograms, worst-region debt, entities/players, bandwidth,
|
|
queues, checkpoint age, fencing/restore failures; add biomass/energy/nutrients,
|
|
birth/death/diversity metrics with biology. Loki: bounded structured logs. Otel
|
|
traces at request/I/O boundaries when useful. Avoid IDs as unbounded metric labels.
|
|
Alert on unavailable recovery paths, disk capacity/latency, missing backups,
|
|
restarts, packet loss and prolonged tick debt; test alerts through fault exercises.
|
|
|
|
S3 is the storage API boundary; backend selection remains open. MinIO's official
|
|
repository is archived/no longer maintained, so do not assume it is an acceptable
|
|
new production default. Evaluate maintained alternatives for license, operations,
|
|
S3 semantics, durability and independent restore. [Official MinIO repository](https://github.com/minio/minio).
|
|
|
|
Before M8: validate inventory, storage class/reclaim behavior, DNS/TLS/ingress,
|
|
Harbor/Argo integration, least privilege, resource budgets and off-host recovery.
|
|
Then kill a region pod and demonstrate automatic fenced restore. ADR-0020 adds
|
|
a stateless M0 verification Job in evolution-dev through the existing Gitea/Harbor
|
|
setup. It is not a persistent region or evidence of the M8 recovery gate.
|
|
Templates with unresolved destination/image markers must not be synced.
|