Record the maintainer-run local-path probe on all three workers
(1.9-2.5 ms synchronous 4 KiB writes, 29-36 ms mean object publication,
110-255 MB/s sequential), well inside the lease deadlines, and flag
worker-3's 87% full root filesystem. Apply the maintainer's corrections:
about 500 GB more host RAM, AEvolution PostgreSQL on a separate VM
outside Kubernetes, and manual-only inventory collection. Storage
figures stay provisional until the planned RAID change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Status: **complete for the current layout, 2026-09-29** (cluster inspected and
storage probed on every worker).
Sources, kept separate below:
- maintainer statements;
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran
(read requests only; Secrets listed by name, type and key count);
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by
hand (read requests only; Secrets listed by name, type and key count). No
automated collection is planned;
- local measurements.
The disks and RAID layout will change, so every storage number here describes
@@ -14,37 +16,38 @@ the current layout only.
| Fact | Source | Implication |
|---|---|---|
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
| One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. |
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
| The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev`holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. |
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
`PGHOSTADDR`, `PGPASSWORD`, `PGPORT`, `PGSSLMODE`, `PGSSLROOTCERT`, `PGUSER` and
`POSTGRES_*`. That points to a single database login provisioned outside this
repository. Before M8.3 can use it, confirm:
- **Server version.** The adapter and migrations require exactly 18.6 (ADR-0059).
Any other version is refused, not tolerated.
- **Role split.** ADR-0060 requires a migrator/owner login plus a separate
NOLOGIN runtime group with its own member login. The migration audit refuses a
privileged or owner-identity runtime role. One login for both would fail that
audit by design.
- **Password delivery.**`PGPASSWORD` in the environment is not how the adapter
is used; it reads a mounted password file (`--pg-passfile`, POSTGRESQL.md).
Mount the Secret as files.
- **TLS.**`PGSSLMODE` and `PGSSLROOTCERT` suggest TLS with a private CA. Use
`verify-full` with the CA mounted.
- **Backup and failure domain.** Where does the database live? It is part of the
M8.3/M8.7 evidence.
**The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster.
`aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details.
Its values were not read. Consequences:
- **Stores.** The database is not a Kubernetes workload, so its memory and disk
are not part of the worker budget. Region hosts reach it across the VM network.
Their NetworkPolicy egress must allow exactly that address and port
(`PGHOSTADDR` / `PGPORT`) and nothing else.
- **Failure domain.** The database VM is a separate VM, but on the same single
hypervisor unless the maintainer says otherwise. It is separate from worker
failures, not from host loss. Backups must leave the host (M8.7).
- **Checks before use (M8.3).** These are enforced by the code, not by inventory:
- the adapter refuses any server other than 18.6 (ADR-0059);
- the migration audit refuses a privileged or owner-identity runtime role
(ADR-0060);
- the adapter reads a mounted password file, never `PGPASSWORD`;
- `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`.
The separate `evolution-postgres-migrator` and `evolution-postgres-runtime`
Secrets (3 keys each) match the required role split.
## Cluster (collected 2026-09-29)
**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the
maintainer states that memory is not overcommitted. Together the VMs hold about
108 GiB of the host's roughly 128 GB. Everything shares one physical failure
domain.
**Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs
and the separate PostgreSQL VM. The maintainer states that memory is not
overcommitted and that the host now has about 500 GB more RAM than the brief's
128 GB. Everything shares one physical failure domain.
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
quotas beyond the defaults:
@@ -117,18 +120,16 @@ way for anything durable. Harbor publication on push is confirmed by the CI runs
**Existing data services.**
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3
therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must
first check what `aevolution-db` points at.
harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution,
whose database is the separate VM above.
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
The region host's object store is a filesystem PVC (ADR-0061), so no object
server is required yet.
**Collector identity.** The output was collected as `system:admin`
(system:masters). The collector made only read requests. M8.2 should create a
genuinely read-only inventory identity, so this is not repeated with
cluster-admin.
**Collector identity.** The maintainer ran the collector by hand as
`system:admin`(system:masters). It made only read requests. No automated
collection or dedicated collection identity is planned.
## Measured workload profile (local fixture)
@@ -168,9 +169,9 @@ restore and verification buffers several times over. Larger profiles
## Budget (proposal from measured inputs, 2026-09-29)
**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the
hypervisor and host. Adding memory to a VM is possible (maintainer), but it does
not add a failure domain.
**Host.** With about 500 GB added, host memory is not a limit for dev and stage.
VM memory can be raised on request (maintainer), but that never adds a failure
domain.
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
@@ -178,40 +179,60 @@ not add a failure domain.
|---|---|---|
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm |
| AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) |
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
**Proposed dev placement** (to confirm in M8.3/8.4):
**Proposed dev placement** (to confirm in M8.4):
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
with 38 % used, and PVs there are pinned to it.
- Place PostgreSQL and the region host's object PVC on **one** worker (currently
the least-used, worker-2), using a node label and affinity.
- local-path pins both volumes to that node anyway. Choosing the node explicitly
keeps the failure domain visible, and no replica count is claimed.
- **Never place them on rke2-worker-3.** Its root filesystem is 87 % full
(13 GB free). That is near the kubelet's default nodefs eviction threshold and
leaves no room for retained history.
- Place the region host's object PVC on **rke2-worker-1**, which had the lowest
measured latency, or **rke2-worker-2**, which has the most free space (60 GB).
Use a node label and affinity. local-path pins the volume to that node anyway;
choosing it explicitly keeps the failure domain visible, and no replica count
is claimed.
Even with these reservations, the AEvolution dev footprint stays under 3 GiB on
that worker.
On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its
successor headroom and the migrate Job). The database load stays on its VM.
The first capacity limits are expected to be:
- storage latency and fsync behaviour (probe pending);
- object publication latency on the current disks (about 30–36 ms mean);
- database connections (one per host plus the migrator);
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
storage plus database latency.
Per-node region counts and CPU budgets are set only after the in-cluster probe
and M8.4's in-cluster measurement.
Per-node region counts and CPU budgets are set only after M8.4's in-cluster
measurement, and re-checked after the disk and RAID rework.
## In-cluster storage probe
## In-cluster storage probe (2026-09-29, current disk layout)
`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job
per worker on `local-path`, pinned by hostname. It is outside every Argo path and
applied by hand, with the maintainer's approval (2026-09-29). The commands are in
the file header. Record each worker's line here and compare it with the local
baseline above.
The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml`
by hand, then deleted it: one PVC and Job per worker on `local-path`. Every
worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB).
| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|---|---|---|---|
| rke2-worker-1 | pending | pending | pending |
| rke2-worker-2 | pending | pending | pending |
| rke2-worker-3 | pending | pending | pending |
| Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and
trace) before its database transaction. That adds roughly 60–150 ms per batch,
plus the network round trip to the database VM.
- **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal
and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval.
Acknowledgement latency in the cluster should be a little above the local
549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must
confirm both in the cluster.
- **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB
payloads, and for audit and clone reads.
- **Provisional.** These numbers describe the current disks. Re-run the probe
after the RAID and storage change, before M8.3's storage decision is final.
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.