Complete M8.1 with per-worker storage probe results

Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 12:33:04 -04:00
co-authored by Claude Opus 5.5
parent 2475563959
commit 18708340c3
5 changed files with 121 additions and 65 deletions
+5
View File
@@ -601,3 +601,8 @@ M7.9b implemented and locally qualified (2026-09-29), under ADR-0068:
- rule and format changes are refused explicitly;
- measured recovery (19.6 s against a 60 s target) and batching latency.
M7 is complete locally. Next: M8.1, the host inventory and budgets.
M8.1 complete (2026-09-29): docs/infrastructure/INVENTORY.md records the cluster,
the separate PostgreSQL VM, the measured workload profile and a per-worker
`local-path` storage probe. Storage figures are provisional until the planned RAID
and storage change. Next: M8.2, environment isolation and secrets.
+13 -4
View File
@@ -980,8 +980,17 @@ M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md):
- RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor.
- `local-path` is the only storage (node-pinned, reclaim Delete).
- `aevolution-dev` syncs manually.
- No PostgreSQL 18.6 exists yet.
- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker.
- AEvolution's PostgreSQL is a separate VM outside Kubernetes (`aevolution-db`),
so it is not in the worker budget.
- A proposed budget keeps AEvolution dev well under 1 GiB on one labeled worker
(not worker-3, whose disk is 87 % full).
Remaining for M8.1: the per-worker storage probe, which is approved and ready to
apply.
M8.1 complete (2026-09-29) for the current disk layout. The per-worker
`local-path` probe measured:
- 1.9–2.5 ms for synchronous 4 KiB writes;
- 29–36 ms mean (73 ms max) per object publication;
- 110–255 MB/s sequential.
These are well inside the lease deadlines. Re-probe after the planned RAID and
storage change. Host memory was raised by about 500 GB.
Next: M8.2, environment isolation and secrets.
@@ -1,8 +1,10 @@
# Infrastructure and Kubernetes topology
Status: production-shaped proposal, not deployed workloads. Initial hardware is
one R730xd with approximately 128GB RAM. Kubernetes is available per brief; actual
CPU/storage/CNI/CSI/backup/network details remain uninspected. Replicas on one
one R730xd, with about 128GB RAM in the brief and about 500GB added since. The
cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a
separate PostgreSQL VM were inventoried in M8.1; see
[INVENTORY.md](../infrastructure/INVENTORY.md). Replicas on one
physical host do not create an independent failure domain.
```mermaid
@@ -62,7 +64,7 @@ unbounded replica creation. Pod disruption budgets help planned maintenance but
cannot protect against chassis loss. Local volumes constrain scheduling; adding
compute nodes does not automatically make stateful storage portable.
128GB is a capacity envelope, not all available simulation memory. Inventory OS/
Host memory is a capacity envelope, not all available simulation memory. Inventory OS/
Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery
headroom before region budgets. Start measured namespace quotas and leave explicit
recovery/compaction headroom. No unverified entity/player-per-node promises.
+19
View File
@@ -2620,3 +2620,22 @@ Evidence:
every Argo path and has **not yet been applied**, because this workstation has
no kubeconfig. The maintainer approved running it. Its results go in
INVENTORY.md.
## M8.1c storage probe and corrections - 2026-09-29
- The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml`
by hand on the cluster, and all three Jobs completed. The resources were then
deleted.
- worker-1: 1906 µs dsync, 28.7/60.6 ms object publish, 1003 ms for 256 MiB.
- worker-2: 2511 µs, 36.0/73.4 ms, 2122 ms.
- worker-3: 2239 µs, 32.1/61.0 ms, 2339 ms. Its root filesystem is 87 % full.
Every `local-path` volume is on the node's root LVM ext4 volume. The figures are
provisional until the planned RAID change.
- Corrections from the maintainer:
- Host RAM was increased by about 500 GB, so the earlier "about 20 GiB left"
estimate is withdrawn.
- AEvolution's PostgreSQL is a separate VM outside Kubernetes, reached through
`aevolution-db`.
- Collection is manual, with no automation planned.
- No credential values were read. The 18.6 version, role and TLS requirements are
enforced by the adapter and migrations when M8.3 connects.
+79 -58
View File
@@ -1,10 +1,12 @@
# Host and cluster inventory (M8.1)
Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.**
Status: **complete for the current layout, 2026-09-29** (cluster inspected and
storage probed on every worker).
Sources, kept separate below:
- maintainer statements;
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran
(read requests only; Secrets listed by name, type and key count);
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by
hand (read requests only; Secrets listed by name, type and key count). No
automated collection is planned;
- local measurements.
The disks and RAID layout will change, so every storage number here describes
@@ -14,37 +16,38 @@ the current layout only.
| Fact | Source | Implication |
|---|---|---|
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
| One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. |
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
| The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev` holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. |
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
`PGHOSTADDR`, `PGPASSWORD`, `PGPORT`, `PGSSLMODE`, `PGSSLROOTCERT`, `PGUSER` and
`POSTGRES_*`. That points to a single database login provisioned outside this
repository. Before M8.3 can use it, confirm:
- **Server version.** The adapter and migrations require exactly 18.6 (ADR-0059).
Any other version is refused, not tolerated.
- **Role split.** ADR-0060 requires a migrator/owner login plus a separate
NOLOGIN runtime group with its own member login. The migration audit refuses a
privileged or owner-identity runtime role. One login for both would fail that
audit by design.
- **Password delivery.** `PGPASSWORD` in the environment is not how the adapter
is used; it reads a mounted password file (`--pg-passfile`, POSTGRESQL.md).
Mount the Secret as files.
- **TLS.** `PGSSLMODE` and `PGSSLROOTCERT` suggest TLS with a private CA. Use
`verify-full` with the CA mounted.
- **Backup and failure domain.** Where does the database live? It is part of the
M8.3/M8.7 evidence.
**The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster.
`aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details.
Its values were not read. Consequences:
- **Stores.** The database is not a Kubernetes workload, so its memory and disk
are not part of the worker budget. Region hosts reach it across the VM network.
Their NetworkPolicy egress must allow exactly that address and port
(`PGHOSTADDR` / `PGPORT`) and nothing else.
- **Failure domain.** The database VM is a separate VM, but on the same single
hypervisor unless the maintainer says otherwise. It is separate from worker
failures, not from host loss. Backups must leave the host (M8.7).
- **Checks before use (M8.3).** These are enforced by the code, not by inventory:
- the adapter refuses any server other than 18.6 (ADR-0059);
- the migration audit refuses a privileged or owner-identity runtime role
(ADR-0060);
- the adapter reads a mounted password file, never `PGPASSWORD`;
- `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`.
The separate `evolution-postgres-migrator` and `evolution-postgres-runtime`
Secrets (3 keys each) match the required role split.
## Cluster (collected 2026-09-29)
**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the
maintainer states that memory is not overcommitted. Together the VMs hold about
108 GiB of the host's roughly 128 GB. Everything shares one physical failure
domain.
**Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs
and the separate PostgreSQL VM. The maintainer states that memory is not
overcommitted and that the host now has about 500 GB more RAM than the brief's
128 GB. Everything shares one physical failure domain.
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
quotas beyond the defaults:
@@ -117,18 +120,16 @@ way for anything durable. Harbor publication on push is confirmed by the CI runs
**Existing data services.**
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3
therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must
first check what `aevolution-db` points at.
harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution,
whose database is the separate VM above.
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
The region host's object store is a filesystem PVC (ADR-0061), so no object
server is required yet.
**Collector identity.** The output was collected as `system:admin`
(system:masters). The collector made only read requests. M8.2 should create a
genuinely read-only inventory identity, so this is not repeated with
cluster-admin.
**Collector identity.** The maintainer ran the collector by hand as
`system:admin` (system:masters). It made only read requests. No automated
collection or dedicated collection identity is planned.
## Measured workload profile (local fixture)
@@ -168,9 +169,9 @@ restore and verification buffers several times over. Larger profiles
## Budget (proposal from measured inputs, 2026-09-29)
**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the
hypervisor and host. Adding memory to a VM is possible (maintainer), but it does
not add a failure domain.
**Host.** With about 500 GB added, host memory is not a limit for dev and stage.
VM memory can be raised on request (maintainer), but that never adds a failure
domain.
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
@@ -178,40 +179,60 @@ not add a failure domain.
|---|---|---|
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm |
| AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) |
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
**Proposed dev placement** (to confirm in M8.3/8.4):
**Proposed dev placement** (to confirm in M8.4):
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
with 38 % used, and PVs there are pinned to it.
- Place PostgreSQL and the region host's object PVC on **one** worker (currently
the least-used, worker-2), using a node label and affinity.
- local-path pins both volumes to that node anyway. Choosing the node explicitly
keeps the failure domain visible, and no replica count is claimed.
- **Never place them on rke2-worker-3.** Its root filesystem is 87 % full
(13 GB free). That is near the kubelet's default nodefs eviction threshold and
leaves no room for retained history.
- Place the region host's object PVC on **rke2-worker-1**, which had the lowest
measured latency, or **rke2-worker-2**, which has the most free space (60 GB).
Use a node label and affinity. local-path pins the volume to that node anyway;
choosing it explicitly keeps the failure domain visible, and no replica count
is claimed.
Even with these reservations, the AEvolution dev footprint stays under 3 GiB on
that worker.
On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its
successor headroom and the migrate Job). The database load stays on its VM.
The first capacity limits are expected to be:
- storage latency and fsync behaviour (probe pending);
- object publication latency on the current disks (about 30–36 ms mean);
- database connections (one per host plus the migrator);
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
storage plus database latency.
Per-node region counts and CPU budgets are set only after the in-cluster probe
and M8.4's in-cluster measurement.
Per-node region counts and CPU budgets are set only after M8.4's in-cluster
measurement, and re-checked after the disk and RAID rework.
## In-cluster storage probe
## In-cluster storage probe (2026-09-29, current disk layout)
`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job
per worker on `local-path`, pinned by hostname. It is outside every Argo path and
applied by hand, with the maintainer's approval (2026-09-29). The commands are in
the file header. Record each worker's line here and compare it with the local
baseline above.
The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml`
by hand, then deleted it: one PVC and Job per worker on `local-path`. Every
worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB).
| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|---|---|---|---|
| rke2-worker-1 | pending | pending | pending |
| rke2-worker-2 | pending | pending | pending |
| rke2-worker-3 | pending | pending | pending |
| Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|---|---|---|---|---|
| rke2-worker-1 | 54 % (43 GB free) | 1906 µs | 28.7 / 60.6 ms | 1003 ms (about 255 MB/s) |
| rke2-worker-2 | 35 % (60 GB free) | 2511 µs | 36.0 / 73.4 ms | 2122 ms (about 121 MB/s) |
| rke2-worker-3 | 87 % (13 GB free) | 2239 µs | 32.1 / 61.0 ms | 2339 ms (about 109 MB/s) |
| Workstation baseline (WSL2, NVMe) | 3 % | 1169 µs | 4.6 / 8.0 ms | 365 ms |
Interpretation:
- **Synchronous 4 KiB writes** are 1.6–2.1× the workstation baseline, which is
unremarkable.
- **Object publication** (write, fsync, rename, directory fsync) is 6–8× slower,
at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and
trace) before its database transaction. That adds roughly 60–150 ms per batch,
plus the network round trip to the database VM.
- **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal
and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval.
Acknowledgement latency in the cluster should be a little above the local
549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must
confirm both in the cluster.
- **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB
payloads, and for audit and clone reads.
- **Provisional.** These numbers describe the current disks. Re-run the probe
after the RAID and storage change, before M8.3's storage decision is final.