Files
dingoandClaude Opus 5.5 18708340c3 Complete M8.1 with per-worker storage probe results
Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:33:04 -04:00

239 lines
14 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Host and cluster inventory (M8.1)
Status: **complete for the current layout, 2026-09-29** (cluster inspected and
storage probed on every worker).
Sources, kept separate below:
- maintainer statements;
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by
hand (read requests only; Secrets listed by name, type and key count). No
automated collection is planned;
- local measurements.
The disks and RAID layout will change, so every storage number here describes
the current layout only.
## Maintainer-stated facts
| Fact | Source | Implication |
|---|---|---|
| One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. |
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
| The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev` holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. |
**The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster.
`aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details.
Its values were not read. Consequences:
- **Stores.** The database is not a Kubernetes workload, so its memory and disk
are not part of the worker budget. Region hosts reach it across the VM network.
Their NetworkPolicy egress must allow exactly that address and port
(`PGHOSTADDR` / `PGPORT`) and nothing else.
- **Failure domain.** The database VM is a separate VM, but on the same single
hypervisor unless the maintainer says otherwise. It is separate from worker
failures, not from host loss. Backups must leave the host (M8.7).
- **Checks before use (M8.3).** These are enforced by the code, not by inventory:
- the adapter refuses any server other than 18.6 (ADR-0059);
- the migration audit refuses a privileged or owner-identity runtime role
(ADR-0060);
- the adapter reads a mounted password file, never `PGPASSWORD`;
- `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`.
The separate `evolution-postgres-migrator` and `evolution-postgres-runtime`
Secrets (3 keys each) match the required role split.
## Cluster (collected 2026-09-29)
**Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs
and the separate PostgreSQL VM. The maintainer states that memory is not
overcommitted and that the host now has about 500 GB more RAM than the brief's
128 GB. Everything shares one physical failure domain.
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
quotas beyond the defaults:
- Memory allocatable equals capacity on every node, so no kube-reserved or
system-reserved memory is configured, and only the default eviction threshold
protects the node.
- The only priority classes are the two system ones.
- There are no ResourceQuotas or LimitRanges anywhere.
- PodDisruptionBudgets exist only for Gitea and Keycloak.
| Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem |
|---|---|---|---|---|---|---|---|
| rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) |
| rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi |
| rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi |
| rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi |
**Storage.**
- `local-path` (rancher.io/local-path) is the only StorageClass. It is the
default, reclaims with **Delete**, and binds WaitForFirstConsumer.
- There are no CSI drivers and no VolumeSnapshotClasses.
- Every existing PV is a directory pinned to one node. They are spread across all
four nodes, including the control plane.
Consequences for M8.3:
- A durable store on `local-path` lives and dies with one VM's disk. Its pod can
run only on that node.
- Deleting its PVC destroys the data, and so would an Argo prune of the PVC.
- Before any durable world data, M8.3 needs:
- a Retain-policy class (for example, a second local-path class with
`reclaimPolicy: Retain`) or a replicated CSI;
- off-node backups (M8.7).
**Network.**
- CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced
through Calico, and the existing deny-all policies rely on that.
- Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes.
- Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer
(Gitea SSH, 172.10.1.201).
- DNS: CoreDNS 1.12.1, two replicas.
**TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and
`letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`.
Every certificate is Ready except `stone-server/webclient-tls`, which belongs to
another project.
**Delivery (Argo CD).**
| Application | Path | Sync policy |
|---|---|---|
| `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync |
| `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal |
| `aeternalis` | other repository | Manual |
The maintainer's statement that Argo deploys on every change applies to the
automated applications, not to AEvolution today. The `deploy/README.md` guidance
(manual full sync, no automated prune) matches the live application. Keep it that
way for anything durable. Harbor publication on push is confirmed by the CI runs.
**`evolution-dev` contents.**
- The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all
NetworkPolicy.
- ServiceAccounts `aevolution-proof`, plus three created by the maintainer
shortly before collection: `aevolution-migrator`, `aevolution-region` and
`aevolution-backup`.
- Secrets `harbor-pull`, `aevolution-db` (13 keys), and
`evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each,
also new). Their values were not read. The separate migrator and runtime
Secrets match the ADR-0060 role split.
**Existing data services.**
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution,
whose database is the separate VM above.
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
The region host's object store is a filesystem PVC (ADR-0061), so no object
server is required yet.
**Collector identity.** The maintainer ran the collector by hand as
`system:admin` (system:masters). It made only read requests. No automated
collection or dedicated collection identity is planned.
## Measured workload profile (local fixture)
The measurements come from `durability_qualification`
(engineering/DURABILITY_QUALIFICATION.md): one lineage region under the
production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.
| Measure (Release, three successive owners) | Value |
|---|---|
| `evo-server` peak resident memory (VmHWM) | 9,940 KiB (about 9.7 MiB) |
| CPU while serving, including start and recovery | 60 ms over 16.3 s of wall time, about 0.3 % of one core |
| Committed payload | 37,756 bytes |
| Head verification p50 / max | 1.6 / 12 ms |
| Acknowledgement latency p50 / p99 (1 s flush) | 549 / 1048 ms |
| SIGKILL to successor ready | 19.6 s (lease-bound) |
Local storage baseline: the same script as the disk probe, run in WSL2 on ext4
over the workstation's NVMe (VHD):
| Pattern | Result |
|---|---|
| 4 KiB O_DSYNC writes | mean 1169 µs |
| 40 KiB object publication (write, fsync, rename, directory fsync) | mean 4646 µs, max 7983 µs |
| 256 MiB sequential with fdatasync | 365 ms |
Cluster storage should be compared against this baseline. A class whose
synchronous-write p99 approaches the 2 s renewal deadline cannot host regions
under the current lease policy.
A region host is small at this profile. Its requests are set by headroom, not by
the simulation itself. **Proposed** starting values, to be confirmed by in-cluster
measurement in M8.4: `requests: {cpu: 50m, memory: 64Mi}` and
`limits: {memory: 256Mi}` with no CPU limit (throttling would delay renewals).
The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
restore and verification buffers several times over. Larger profiles
(populations, fields) must be re-measured before these values are reused.
## Budget (proposal from measured inputs, 2026-09-29)
**Host.** With about 500 GB added, host memory is not a limit for dev and stage.
VM memory can be raised on request (maintainer), but that never adds a failure
domain.
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
| Reserve | Proposal | Basis |
|---|---|---|
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
| AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) |
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
**Proposed dev placement** (to confirm in M8.4):
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
with 38 % used, and PVs there are pinned to it.
- **Never place them on rke2-worker-3.** Its root filesystem is 87 % full
(13 GB free). That is near the kubelet's default nodefs eviction threshold and
leaves no room for retained history.
- Place the region host's object PVC on **rke2-worker-1**, which had the lowest
measured latency, or **rke2-worker-2**, which has the most free space (60 GB).
Use a node label and affinity. local-path pins the volume to that node anyway;
choosing it explicitly keeps the failure domain visible, and no replica count
is claimed.
On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its
successor headroom and the migrate Job). The database load stays on its VM.
The first capacity limits are expected to be:
- object publication latency on the current disks (about 30–36 ms mean);
- database connections (one per host plus the migrator);
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
storage plus database latency.
Per-node region counts and CPU budgets are set only after M8.4's in-cluster
measurement, and re-checked after the disk and RAID rework.
## In-cluster storage probe (2026-09-29, current disk layout)
The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml`
by hand, then deleted it: one PVC and Job per worker on `local-path`. Every
worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB).
| Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|---|---|---|---|---|
| rke2-worker-1 | 54 % (43 GB free) | 1906 µs | 28.7 / 60.6 ms | 1003 ms (about 255 MB/s) |
| rke2-worker-2 | 35 % (60 GB free) | 2511 µs | 36.0 / 73.4 ms | 2122 ms (about 121 MB/s) |
| rke2-worker-3 | 87 % (13 GB free) | 2239 µs | 32.1 / 61.0 ms | 2339 ms (about 109 MB/s) |
| Workstation baseline (WSL2, NVMe) | 3 % | 1169 µs | 4.6 / 8.0 ms | 365 ms |
Interpretation:
- **Synchronous 4 KiB writes** are 1.6–2.1× the workstation baseline, which is
unremarkable.
- **Object publication** (write, fsync, rename, directory fsync) is 6–8× slower,
at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and
trace) before its database transaction. That adds roughly 60–150 ms per batch,
plus the network round trip to the database VM.
- **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal
and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval.
Acknowledgement latency in the cluster should be a little above the local
549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must
confirm both in the cluster.
- **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB
payloads, and for audit and clone reads.
- **Provisional.** These numbers describe the current disks. Re-run the probe
after the RAID and storage change, before M8.3's storage decision is final.