# Host and cluster inventory (M8.1) Status: **complete for the current layout, 2026-09-29** (cluster inspected and storage probed on every worker). Sources, kept separate below: - maintainer statements; - the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by hand (read requests only; Secrets listed by name, type and key count). No automated collection is planned; - local measurements. The disks and RAID layout will change, so every storage number here describes the current layout only. ## Maintainer-stated facts | Fact | Source | Implication | |---|---|---| | One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. | | One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. | | Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. | | Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. | | Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. | | The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev` holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. | **The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster. `aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details. Its values were not read. Consequences: - **Stores.** The database is not a Kubernetes workload, so its memory and disk are not part of the worker budget. Region hosts reach it across the VM network. Their NetworkPolicy egress must allow exactly that address and port (`PGHOSTADDR` / `PGPORT`) and nothing else. - **Failure domain.** The database VM is a separate VM, but on the same single hypervisor unless the maintainer says otherwise. It is separate from worker failures, not from host loss. Backups must leave the host (M8.7). - **Checks before use (M8.3).** These are enforced by the code, not by inventory: - the adapter refuses any server other than 18.6 (ADR-0059); - the migration audit refuses a privileged or owner-identity runtime role (ADR-0060); - the adapter reads a mounted password file, never `PGPASSWORD`; - `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`. The separate `evolution-postgres-migrator` and `evolution-postgres-runtime` Secrets (3 keys each) match the required role split. ## Cluster (collected 2026-09-29) **Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs and the separate PostgreSQL VM. The maintainer states that memory is not overcommitted and that the host now has about 500 GB more RAM than the brief's 128 GB. Everything shares one physical failure domain. The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or quotas beyond the defaults: - Memory allocatable equals capacity on every node, so no kube-reserved or system-reserved memory is configured, and only the default eviction threshold protects the node. - The only priority classes are the two system ones. - There are no ResourceQuotas or LimitRanges anywhere. - PodDisruptionBudgets exist only for Gitea and Keycloak. | Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem | |---|---|---|---|---|---|---|---| | rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) | | rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi | | rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi | | rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi | **Storage.** - `local-path` (rancher.io/local-path) is the only StorageClass. It is the default, reclaims with **Delete**, and binds WaitForFirstConsumer. - There are no CSI drivers and no VolumeSnapshotClasses. - Every existing PV is a directory pinned to one node. They are spread across all four nodes, including the control plane. Consequences for M8.3: - A durable store on `local-path` lives and dies with one VM's disk. Its pod can run only on that node. - Deleting its PVC destroys the data, and so would an Argo prune of the PVC. - Before any durable world data, M8.3 needs: - a Retain-policy class (for example, a second local-path class with `reclaimPolicy: Retain`) or a replicated CSI; - off-node backups (M8.7). **Network.** - CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced through Calico, and the existing deny-all policies rely on that. - Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes. - Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer (Gitea SSH, 172.10.1.201). - DNS: CoreDNS 1.12.1, two replicas. **TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and `letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`. Every certificate is Ready except `stone-server/webclient-tls`, which belongs to another project. **Delivery (Argo CD).** | Application | Path | Sync policy | |---|---|---| | `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync | | `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal | | `aeternalis` | other repository | Manual | The maintainer's statement that Argo deploys on every change applies to the automated applications, not to AEvolution today. The `deploy/README.md` guidance (manual full sync, no automated prune) matches the live application. Keep it that way for anything durable. Harbor publication on push is confirmed by the CI runs. **`evolution-dev` contents.** - The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all NetworkPolicy. - ServiceAccounts `aevolution-proof`, plus three created by the maintainer shortly before collection: `aevolution-migrator`, `aevolution-region` and `aevolution-backup`. - Secrets `harbor-pull`, `aevolution-db` (13 keys), and `evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each, also new). Their values were not read. The separate migrator and runtime Secrets match the ADR-0060 role split. **Existing data services.** - `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution, whose database is the separate VM above. - `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned. INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default. The region host's object store is a filesystem PVC (ADR-0061), so no object server is required yet. **Collector identity.** The maintainer ran the collector by hand as `system:admin` (system:masters). It made only read requests. No automated collection or dedicated collection identity is planned. ## Measured workload profile (local fixture) The measurements come from `durability_qualification` (engineering/DURABILITY_QUALIFICATION.md): one lineage region under the production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6. | Measure (Release, three successive owners) | Value | |---|---| | `evo-server` peak resident memory (VmHWM) | 9,940 KiB (about 9.7 MiB) | | CPU while serving, including start and recovery | 60 ms over 16.3 s of wall time, about 0.3 % of one core | | Committed payload | 37,756 bytes | | Head verification p50 / max | 1.6 / 12 ms | | Acknowledgement latency p50 / p99 (1 s flush) | 549 / 1048 ms | | SIGKILL to successor ready | 19.6 s (lease-bound) | Local storage baseline: the same script as the disk probe, run in WSL2 on ext4 over the workstation's NVMe (VHD): | Pattern | Result | |---|---| | 4 KiB O_DSYNC writes | mean 1169 µs | | 40 KiB object publication (write, fsync, rename, directory fsync) | mean 4646 µs, max 7983 µs | | 256 MiB sequential with fdatasync | 365 ms | Cluster storage should be compared against this baseline. A class whose synchronous-write p99 approaches the 2 s renewal deadline cannot host regions under the current lease policy. A region host is small at this profile. Its requests are set by headroom, not by the simulation itself. **Proposed** starting values, to be confirmed by in-cluster measurement in M8.4: `requests: {cpu: 50m, memory: 64Mi}` and `limits: {memory: 256Mi}` with no CPU limit (throttling would delay renewals). The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and restore and verification buffers several times over. Larger profiles (populations, fields) must be re-measured before these values are reused. ## Budget (proposal from measured inputs, 2026-09-29) **Host.** With about 500 GB added, host memory is not a limit for dev and stage. VM memory can be raised on request (maintainer), but that never adds a failure domain. **Per worker** (31.3 GiB capacity, no kubelet reservation configured): | Reserve | Proposal | Basis | |---|---|---| | Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services | | Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt | | AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) | | AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 | | Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB | **Proposed dev placement** (to confirm in M8.4): - Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB with 38 % used, and PVs there are pinned to it. - **Never place them on rke2-worker-3.** Its root filesystem is 87 % full (13 GB free). That is near the kubelet's default nodefs eviction threshold and leaves no room for retained history. - Place the region host's object PVC on **rke2-worker-1**, which had the lowest measured latency, or **rke2-worker-2**, which has the most free space (60 GB). Use a node label and affinity. local-path pins the volume to that node anyway; choosing it explicitly keeps the failure domain visible, and no replica count is claimed. On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its successor headroom and the migrate Job). The database load stays on its VM. The first capacity limits are expected to be: - object publication latency on the current disks (about 30–36 ms mean); - database connections (one per host plus the migrator); - the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99 storage plus database latency. Per-node region counts and CPU budgets are set only after M8.4's in-cluster measurement, and re-checked after the disk and RAID rework. ## In-cluster storage probe (2026-09-29, current disk layout) The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml` by hand, then deleted it: one PVC and Job per worker on `local-path`. Every worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB). | Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync | |---|---|---|---|---| | rke2-worker-1 | 54 % (43 GB free) | 1906 µs | 28.7 / 60.6 ms | 1003 ms (about 255 MB/s) | | rke2-worker-2 | 35 % (60 GB free) | 2511 µs | 36.0 / 73.4 ms | 2122 ms (about 121 MB/s) | | rke2-worker-3 | 87 % (13 GB free) | 2239 µs | 32.1 / 61.0 ms | 2339 ms (about 109 MB/s) | | Workstation baseline (WSL2, NVMe) | 3 % | 1169 µs | 4.6 / 8.0 ms | 365 ms | Interpretation: - **Synchronous 4 KiB writes** are 1.6–2.1× the workstation baseline, which is unremarkable. - **Object publication** (write, fsync, rename, directory fsync) is 6–8× slower, at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and trace) before its database transaction. That adds roughly 60–150 ms per batch, plus the network round trip to the database VM. - **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval. Acknowledgement latency in the cluster should be a little above the local 549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must confirm both in the cluster. - **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB payloads, and for audit and clone reads. - **Provisional.** These numbers describe the current disks. Re-run the probe after the RAID and storage change, before M8.3's storage decision is final.