Record the maintainer-run local-path probe on all three workers (1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication, 110-255 MB/s sequential), well inside the lease deadlines, and flag worker-3's 87% full root filesystem. Apply the maintainer's corrections: about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM outside Kubernetes, and manual-only inventory collection. Storage figures stay provisional until the planned RAID change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
239 lines
14 KiB
Markdown
239 lines
14 KiB
Markdown
# Host and cluster inventory (M8.1)
|
||
|
||
Status: **complete for the current layout, 2026-09-29** (cluster inspected and
|
||
storage probed on every worker).
|
||
Sources, kept separate below:
|
||
- maintainer statements;
|
||
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by
|
||
hand (read requests only; Secrets listed by name, type and key count). No
|
||
automated collection is planned;
|
||
- local measurements.
|
||
|
||
The disks and RAID layout will change, so every storage number here describes
|
||
the current layout only.
|
||
|
||
## Maintainer-stated facts
|
||
|
||
| Fact | Source | Implication |
|
||
|---|---|---|
|
||
| One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. |
|
||
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
|
||
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
|
||
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
|
||
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
|
||
| The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev` holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. |
|
||
|
||
**The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster.
|
||
`aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details.
|
||
Its values were not read. Consequences:
|
||
- **Stores.** The database is not a Kubernetes workload, so its memory and disk
|
||
are not part of the worker budget. Region hosts reach it across the VM network.
|
||
Their NetworkPolicy egress must allow exactly that address and port
|
||
(`PGHOSTADDR` / `PGPORT`) and nothing else.
|
||
- **Failure domain.** The database VM is a separate VM, but on the same single
|
||
hypervisor unless the maintainer says otherwise. It is separate from worker
|
||
failures, not from host loss. Backups must leave the host (M8.7).
|
||
- **Checks before use (M8.3).** These are enforced by the code, not by inventory:
|
||
- the adapter refuses any server other than 18.6 (ADR-0059);
|
||
- the migration audit refuses a privileged or owner-identity runtime role
|
||
(ADR-0060);
|
||
- the adapter reads a mounted password file, never `PGPASSWORD`;
|
||
- `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`.
|
||
The separate `evolution-postgres-migrator` and `evolution-postgres-runtime`
|
||
Secrets (3 keys each) match the required role split.
|
||
|
||
## Cluster (collected 2026-09-29)
|
||
|
||
**Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs
|
||
and the separate PostgreSQL VM. The maintainer states that memory is not
|
||
overcommitted and that the host now has about 500 GB more RAM than the brief's
|
||
128 GB. Everything shares one physical failure domain.
|
||
|
||
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
|
||
quotas beyond the defaults:
|
||
- Memory allocatable equals capacity on every node, so no kube-reserved or
|
||
system-reserved memory is configured, and only the default eviction threshold
|
||
protects the node.
|
||
- The only priority classes are the two system ones.
|
||
- There are no ResourceQuotas or LimitRanges anywhere.
|
||
- PodDisruptionBudgets exist only for Gitea and Keycloak.
|
||
|
||
| Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem |
|
||
|---|---|---|---|---|---|---|---|
|
||
| rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) |
|
||
| rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi |
|
||
| rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi |
|
||
| rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi |
|
||
|
||
**Storage.**
|
||
- `local-path` (rancher.io/local-path) is the only StorageClass. It is the
|
||
default, reclaims with **Delete**, and binds WaitForFirstConsumer.
|
||
- There are no CSI drivers and no VolumeSnapshotClasses.
|
||
- Every existing PV is a directory pinned to one node. They are spread across all
|
||
four nodes, including the control plane.
|
||
|
||
Consequences for M8.3:
|
||
- A durable store on `local-path` lives and dies with one VM's disk. Its pod can
|
||
run only on that node.
|
||
- Deleting its PVC destroys the data, and so would an Argo prune of the PVC.
|
||
- Before any durable world data, M8.3 needs:
|
||
- a Retain-policy class (for example, a second local-path class with
|
||
`reclaimPolicy: Retain`) or a replicated CSI;
|
||
- off-node backups (M8.7).
|
||
|
||
**Network.**
|
||
- CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced
|
||
through Calico, and the existing deny-all policies rely on that.
|
||
- Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes.
|
||
- Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer
|
||
(Gitea SSH, 172.10.1.201).
|
||
- DNS: CoreDNS 1.12.1, two replicas.
|
||
|
||
**TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and
|
||
`letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`.
|
||
Every certificate is Ready except `stone-server/webclient-tls`, which belongs to
|
||
another project.
|
||
|
||
**Delivery (Argo CD).**
|
||
|
||
| Application | Path | Sync policy |
|
||
|---|---|---|
|
||
| `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync |
|
||
| `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal |
|
||
| `aeternalis` | other repository | Manual |
|
||
|
||
The maintainer's statement that Argo deploys on every change applies to the
|
||
automated applications, not to AEvolution today. The `deploy/README.md` guidance
|
||
(manual full sync, no automated prune) matches the live application. Keep it that
|
||
way for anything durable. Harbor publication on push is confirmed by the CI runs.
|
||
|
||
**`evolution-dev` contents.**
|
||
- The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all
|
||
NetworkPolicy.
|
||
- ServiceAccounts `aevolution-proof`, plus three created by the maintainer
|
||
shortly before collection: `aevolution-migrator`, `aevolution-region` and
|
||
`aevolution-backup`.
|
||
- Secrets `harbor-pull`, `aevolution-db` (13 keys), and
|
||
`evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each,
|
||
also new). Their values were not read. The separate migrator and runtime
|
||
Secrets match the ADR-0060 role split.
|
||
|
||
**Existing data services.**
|
||
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
|
||
harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution,
|
||
whose database is the separate VM above.
|
||
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
|
||
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
|
||
The region host's object store is a filesystem PVC (ADR-0061), so no object
|
||
server is required yet.
|
||
|
||
**Collector identity.** The maintainer ran the collector by hand as
|
||
`system:admin` (system:masters). It made only read requests. No automated
|
||
collection or dedicated collection identity is planned.
|
||
|
||
## Measured workload profile (local fixture)
|
||
|
||
The measurements come from `durability_qualification`
|
||
(engineering/DURABILITY_QUALIFICATION.md): one lineage region under the
|
||
production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.
|
||
|
||
| Measure (Release, three successive owners) | Value |
|
||
|---|---|
|
||
| `evo-server` peak resident memory (VmHWM) | 9,940 KiB (about 9.7 MiB) |
|
||
| CPU while serving, including start and recovery | 60 ms over 16.3 s of wall time, about 0.3 % of one core |
|
||
| Committed payload | 37,756 bytes |
|
||
| Head verification p50 / max | 1.6 / 12 ms |
|
||
| Acknowledgement latency p50 / p99 (1 s flush) | 549 / 1048 ms |
|
||
| SIGKILL to successor ready | 19.6 s (lease-bound) |
|
||
|
||
Local storage baseline: the same script as the disk probe, run in WSL2 on ext4
|
||
over the workstation's NVMe (VHD):
|
||
|
||
| Pattern | Result |
|
||
|---|---|
|
||
| 4 KiB O_DSYNC writes | mean 1169 µs |
|
||
| 40 KiB object publication (write, fsync, rename, directory fsync) | mean 4646 µs, max 7983 µs |
|
||
| 256 MiB sequential with fdatasync | 365 ms |
|
||
|
||
Cluster storage should be compared against this baseline. A class whose
|
||
synchronous-write p99 approaches the 2 s renewal deadline cannot host regions
|
||
under the current lease policy.
|
||
|
||
A region host is small at this profile. Its requests are set by headroom, not by
|
||
the simulation itself. **Proposed** starting values, to be confirmed by in-cluster
|
||
measurement in M8.4: `requests: {cpu: 50m, memory: 64Mi}` and
|
||
`limits: {memory: 256Mi}` with no CPU limit (throttling would delay renewals).
|
||
The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
|
||
restore and verification buffers several times over. Larger profiles
|
||
(populations, fields) must be re-measured before these values are reused.
|
||
|
||
## Budget (proposal from measured inputs, 2026-09-29)
|
||
|
||
**Host.** With about 500 GB added, host memory is not a limit for dev and stage.
|
||
VM memory can be raised on request (maintainer), but that never adds a failure
|
||
domain.
|
||
|
||
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
|
||
|
||
| Reserve | Proposal | Basis |
|
||
|---|---|---|
|
||
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
|
||
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
|
||
| AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) |
|
||
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
|
||
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
|
||
|
||
**Proposed dev placement** (to confirm in M8.4):
|
||
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
|
||
with 38 % used, and PVs there are pinned to it.
|
||
- **Never place them on rke2-worker-3.** Its root filesystem is 87 % full
|
||
(13 GB free). That is near the kubelet's default nodefs eviction threshold and
|
||
leaves no room for retained history.
|
||
- Place the region host's object PVC on **rke2-worker-1**, which had the lowest
|
||
measured latency, or **rke2-worker-2**, which has the most free space (60 GB).
|
||
Use a node label and affinity. local-path pins the volume to that node anyway;
|
||
choosing it explicitly keeps the failure domain visible, and no replica count
|
||
is claimed.
|
||
|
||
On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its
|
||
successor headroom and the migrate Job). The database load stays on its VM.
|
||
|
||
The first capacity limits are expected to be:
|
||
- object publication latency on the current disks (about 30–36 ms mean);
|
||
- database connections (one per host plus the migrator);
|
||
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
|
||
storage plus database latency.
|
||
|
||
Per-node region counts and CPU budgets are set only after M8.4's in-cluster
|
||
measurement, and re-checked after the disk and RAID rework.
|
||
|
||
## In-cluster storage probe (2026-09-29, current disk layout)
|
||
|
||
The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml`
|
||
by hand, then deleted it: one PVC and Job per worker on `local-path`. Every
|
||
worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB).
|
||
|
||
| Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|
||
|---|---|---|---|---|
|
||
| rke2-worker-1 | 54 % (43 GB free) | 1906 µs | 28.7 / 60.6 ms | 1003 ms (about 255 MB/s) |
|
||
| rke2-worker-2 | 35 % (60 GB free) | 2511 µs | 36.0 / 73.4 ms | 2122 ms (about 121 MB/s) |
|
||
| rke2-worker-3 | 87 % (13 GB free) | 2239 µs | 32.1 / 61.0 ms | 2339 ms (about 109 MB/s) |
|
||
| Workstation baseline (WSL2, NVMe) | 3 % | 1169 µs | 4.6 / 8.0 ms | 365 ms |
|
||
|
||
Interpretation:
|
||
- **Synchronous 4 KiB writes** are 1.6–2.1× the workstation baseline, which is
|
||
unremarkable.
|
||
- **Object publication** (write, fsync, rename, directory fsync) is 6–8× slower,
|
||
at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and
|
||
trace) before its database transaction. That adds roughly 60–150 ms per batch,
|
||
plus the network round trip to the database VM.
|
||
- **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal
|
||
and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval.
|
||
Acknowledgement latency in the cluster should be a little above the local
|
||
549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must
|
||
confirm both in the cluster.
|
||
- **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB
|
||
payloads, and for audit and clone reads.
|
||
- **Provisional.** These numbers describe the current disks. Re-run the probe
|
||
after the RAID and storage change, before M8.3's storage decision is final.
|