Files
aevolution/docs/infrastructure/INVENTORY.md
T
dingoandClaude Opus 5.5 18708340c3 Complete M8.1 with per-worker storage probe results
Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:33:04 -04:00

14 KiB
Raw Blame History

Host and cluster inventory (M8.1)

Status: complete for the current layout, 2026-09-29 (cluster inspected and storage probed on every worker). Sources, kept separate below:

  • maintainer statements;
  • the output of tools/inventory/collect-cluster.sh, which the maintainer ran by hand (read requests only; Secrets listed by name, type and key count). No automated collection is planned;
  • local measurements.

The disks and RAID layout will change, so every storage number here describes the current layout only.

Maintainer-stated facts

Fact Source Implication
One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB Founding brief; maintainer, 2026-09-29 Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here.
One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added Maintainer, 2026-09-29; confirmed by node capacities All four nodes share one physical failure domain. Adding memory does not add fault tolerance.
Disks and RAID will be reorganized later Maintainer, 2026-09-29 Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards.
Every Git push builds and publishes to Harbor Maintainer, 2026-09-29; Gitea runs 1399/1402 Publication is automatic. It is not promotion: nothing reaches a cluster except through Git.
Argo CD pulls Git and images and deploys to Kubernetes on each change Maintainer, 2026-09-29 True for the automated applications (sor-*). The live aevolution-dev application has no automated sync (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays .yaml.example or outside deploy/kubernetes/dev. The dev Job image is pinned by digest in Git.
The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret aevolution-db in evolution-dev holds everything needed to connect Maintainer, 2026-09-29 See below. The Secret's values were not read.

The AEvolution database. PostgreSQL runs on its own VM, outside the cluster. aevolution-db (13 keys: PG* and POSTGRES_*) holds its connection details. Its values were not read. Consequences:

  • Stores. The database is not a Kubernetes workload, so its memory and disk are not part of the worker budget. Region hosts reach it across the VM network. Their NetworkPolicy egress must allow exactly that address and port (PGHOSTADDR / PGPORT) and nothing else.
  • Failure domain. The database VM is a separate VM, but on the same single hypervisor unless the maintainer says otherwise. It is separate from worker failures, not from host loss. Backups must leave the host (M8.7).
  • Checks before use (M8.3). These are enforced by the code, not by inventory:
    • the adapter refuses any server other than 18.6 (ADR-0059);
    • the migration audit refuses a privileged or owner-identity runtime role (ADR-0060);
    • the adapter reads a mounted password file, never PGPASSWORD;
    • PGSSLROOTCERT indicates TLS with a CA, so connect with verify-full. The separate evolution-postgres-migrator and evolution-postgres-runtime Secrets (3 keys each) match the required role split.

Cluster (collected 2026-09-29)

Hypervisor and nodes. One hypervisor on the R730xd runs the four cluster VMs and the separate PostgreSQL VM. The maintainer states that memory is not overcommitted and that the host now has about 500 GB more RAM than the brief's 128 GB. Everything shares one physical failure domain.

The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or quotas beyond the defaults:

  • Memory allocatable equals capacity on every node, so no kube-reserved or system-reserved memory is configured, and only the default eviction threshold protects the node.
  • The only priority classes are the two system ones.
  • There are no ResourceQuotas or LimitRanges anywhere.
  • PodDisruptionBudgets exist only for Gitea and Keycloak.
Node vCPU Memory Ephemeral Taints Requests CPU / mem Limits CPU / mem Used CPU / mem
rke2-master-1 (control plane) 6 11.7 GiB 48 GiB none, so workloads schedule here 2675m / 3890Mi 3950m / 4032Mi 548m / 4549Mi (38 %)
rke2-worker-1 8 31.3 GiB 97 GiB none 2200m / 1882Mi 6150m / 4288Mi 2416m / 2466Mi
rke2-worker-2 8 31.3 GiB 97 GiB none 2300m / 2010Mi 6300m / 4480Mi 382m / 2517Mi
rke2-worker-3 8 31.3 GiB 97 GiB none 3200m / 4058Mi 8550m (106 %) / 8896Mi 345m / 5955Mi

Storage.

  • local-path (rancher.io/local-path) is the only StorageClass. It is the default, reclaims with Delete, and binds WaitForFirstConsumer.
  • There are no CSI drivers and no VolumeSnapshotClasses.
  • Every existing PV is a directory pinned to one node. They are spread across all four nodes, including the control plane.

Consequences for M8.3:

  • A durable store on local-path lives and dies with one VM's disk. Its pod can run only on that node.
  • Deleting its PVC destroys the data, and so would an Argo prune of the PVC.
  • Before any durable world data, M8.3 needs:
    • a Retain-policy class (for example, a second local-path class with reclaimPolicy: Retain) or a replicated CSI;
    • off-node backups (M8.7).

Network.

  • CNI: rke2-canal (Calico v3.30 with flannel). NetworkPolicy is enforced through Calico, and the existing deny-all policies rely on that.
  • Ingress: nginx (ingress-nginx 1.12.1) as a DaemonSet on all four nodes.
  • Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer (Gitea SSH, 172.10.1.201).
  • DNS: CoreDNS 1.12.1, two replicas.

TLS. cert-manager, with ClusterIssuers letsencrypt-godaddy and letsencrypt-godaddy-wildcard. The wildcard certificate is in kube-system. Every certificate is Ready except stone-server/webclient-tls, which belongs to another project.

Delivery (Argo CD).

Application Path Sync policy
aevolution-dev (project aevolution-dev) deploy/kubernetes/dev on main No automated sync. Currently Synced/Healthy; deploys only on a manual sync
sor-server, sor-webclient other repositories Automated, with prune and self-heal
aeternalis other repository Manual

The maintainer's statement that Argo deploys on every change applies to the automated applications, not to AEvolution today. The deploy/README.md guidance (manual full sync, no automated prune) matches the live application. Keep it that way for anything durable. Harbor publication on push is confirmed by the CI runs.

evolution-dev contents.

  • The completed aevolution-dev-proof Job (run 1399 digest) and its deny-all NetworkPolicy.
  • ServiceAccounts aevolution-proof, plus three created by the maintainer shortly before collection: aevolution-migrator, aevolution-region and aevolution-backup.
  • Secrets harbor-pull, aevolution-db (13 keys), and evolution-postgres-migrator and evolution-postgres-runtime (3 keys each, also new). Their values were not read. The separate migrator and runtime Secrets match the ADR-0060 role split.

Existing data services.

  • gitea-postgresql runs 17.6 (bitnamilegacy) and harbor-database runs harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution, whose database is the separate VM above.
  • cdn/minio runs quay.io/minio/minio:latest, which is unpinned. INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default. The region host's object store is a filesystem PVC (ADR-0061), so no object server is required yet.

Collector identity. The maintainer ran the collector by hand as system:admin (system:masters). It made only read requests. No automated collection or dedicated collection identity is planned.

Measured workload profile (local fixture)

The measurements come from durability_qualification (engineering/DURABILITY_QUALIFICATION.md): one lineage region under the production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.

Measure (Release, three successive owners) Value
evo-server peak resident memory (VmHWM) 9,940 KiB (about 9.7 MiB)
CPU while serving, including start and recovery 60 ms over 16.3 s of wall time, about 0.3 % of one core
Committed payload 37,756 bytes
Head verification p50 / max 1.6 / 12 ms
Acknowledgement latency p50 / p99 (1 s flush) 549 / 1048 ms
SIGKILL to successor ready 19.6 s (lease-bound)

Local storage baseline: the same script as the disk probe, run in WSL2 on ext4 over the workstation's NVMe (VHD):

Pattern Result
4 KiB O_DSYNC writes mean 1169 µs
40 KiB object publication (write, fsync, rename, directory fsync) mean 4646 µs, max 7983 µs
256 MiB sequential with fdatasync 365 ms

Cluster storage should be compared against this baseline. A class whose synchronous-write p99 approaches the 2 s renewal deadline cannot host regions under the current lease policy.

A region host is small at this profile. Its requests are set by headroom, not by the simulation itself. Proposed starting values, to be confirmed by in-cluster measurement in M8.4: requests: {cpu: 50m, memory: 64Mi} and limits: {memory: 256Mi} with no CPU limit (throttling would delay renewals). The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and restore and verification buffers several times over. Larger profiles (populations, fields) must be re-measured before these values are reused.

Budget (proposal from measured inputs, 2026-09-29)

Host. With about 500 GB added, host memory is not a limit for dev and stage. VM memory can be raised on request (maintainer), but that never adds a failure domain.

Per worker (31.3 GiB capacity, no kubelet reservation configured):

Reserve Proposal Basis
Kubelet/system reservation Configure about 1 GiB system-reserved plus 1 GiB kube-reserved, and a 500 Mi hard eviction threshold Allocatable currently equals capacity, so pod usage can starve the node's own services
Existing platform and other projects Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) top-nodes.txt and nodes-describe.txt
AEvolution dev, steady Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is not on the workers (separate VM) Measured host footprint of about 9.7 MiB (above)
AEvolution recovery headroom A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers ADR-0064/0065/0067
Observability (M8.5) Up to about 0.5 GiB for scrape and log agents ADR-0066 bounds each region scrape at 8 KiB

Proposed dev placement (to confirm in M8.4):

  • Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB with 38 % used, and PVs there are pinned to it.
  • Never place them on rke2-worker-3. Its root filesystem is 87 % full (13 GB free). That is near the kubelet's default nodefs eviction threshold and leaves no room for retained history.
  • Place the region host's object PVC on rke2-worker-1, which had the lowest measured latency, or rke2-worker-2, which has the most free space (60 GB). Use a node label and affinity. local-path pins the volume to that node anyway; choosing it explicitly keeps the failure domain visible, and no replica count is claimed.

On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its successor headroom and the migrate Job). The database load stays on its VM.

The first capacity limits are expected to be:

  • object publication latency on the current disks (about 30–36 ms mean);
  • database connections (one per host plus the migrator);
  • the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99 storage plus database latency.

Per-node region counts and CPU budgets are set only after M8.4's in-cluster measurement, and re-checked after the disk and RAID rework.

In-cluster storage probe (2026-09-29, current disk layout)

The maintainer applied deploy/kubernetes/inventory/disk-probe-local-path.yaml by hand, then deleted it: one PVC and Job per worker on local-path. Every worker's local-path sits on the node's root LVM volume (ext4, 97 GB).

Worker Root filesystem used 4 KiB O_DSYNC mean 40 KiB object publish mean / max 256 MiB fdatasync
rke2-worker-1 54 % (43 GB free) 1906 µs 28.7 / 60.6 ms 1003 ms (about 255 MB/s)
rke2-worker-2 35 % (60 GB free) 2511 µs 36.0 / 73.4 ms 2122 ms (about 121 MB/s)
rke2-worker-3 87 % (13 GB free) 2239 µs 32.1 / 61.0 ms 2339 ms (about 109 MB/s)
Workstation baseline (WSL2, NVMe) 3 % 1169 µs 4.6 / 8.0 ms 365 ms

Interpretation:

  • Synchronous 4 KiB writes are 1.6–2.1× the workstation baseline, which is unremarkable.
  • Object publication (write, fsync, rename, directory fsync) is 6–8× slower, at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and trace) before its database transaction. That adds roughly 60–150 ms per batch, plus the network round trip to the database VM.
  • Lease margins hold. That per-batch cost stays far inside the 2 s renewal and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval. Acknowledgement latency in the cluster should be a little above the local 549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must confirm both in the cluster.
  • Sequential throughput is 110–255 MB/s. That is ample for 38 KB to 4.3 MB payloads, and for audit and clone reads.
  • Provisional. These numbers describe the current disks. Re-run the probe after the RAID and storage change, before M8.3's storage decision is final.