Record M8.1 cluster inventory and per-worker storage probe

Summarize the maintainer-collected read-only inventory: RKE2 v1.32.5,
an untainted control plane and three 32 GB workers on one hypervisor,
no kubelet reservations or quotas, local-path as the only (node-pinned,
reclaim Delete) storage, Canal/ingress-nginx/MetalLB/cert-manager, a
manually synced aevolution-dev application and no PostgreSQL 18.6.
Propose a measured budget and dev placement, and add a ready-to-apply
local-path probe with one PVC and Job pinned to each worker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 12:20:45 -04:00
co-authored by Claude Opus 5.5
parent 680e7e5725
commit 2475563959
5 changed files with 470 additions and 45 deletions
+4 -4
View File
@@ -74,10 +74,10 @@ of the live application's configuration:
| Project | aevolution-dev, or an equivalently restricted project |
| Sync | Manual full sync; no automated prune/self-heal (recommended) |
The maintainer reports (2026-09-29) that Argo CD currently deploys each Git change
automatically. Treat every file under a watched path as live on push, and confirm
the actual sync and prune policy with `tools/inventory/collect-cluster.sh`
(docs/infrastructure/INVENTORY.md).
Inventory (2026-09-29, docs/infrastructure/INVENTORY.md): the live `aevolution-dev`
application has no automated sync, which matches this guidance. Other projects'
applications do auto-sync with prune. Keep AEvolution durable resources out of any
automated-prune application.
Bootstrap examples in `deploy/argocd/*.yaml.example` are outside the workload
path. Replace the Argo namespace and registered cluster API URL, or use its UI.
@@ -0,0 +1,291 @@
# M8.1 storage latency probe for the cluster's only StorageClass, local-path
# (rancher.io/local-path: node-local directories, reclaim Delete,
# WaitForFirstConsumer). A local-path volume is created on the node that first
# runs its pod, so each worker gets its own PVC and Job pinned by hostname.
# Generated from disk-probe.yaml.example; outside every Argo path, applied by hand
# with the maintainer's approval (2026-09-29):
# kubectl apply -f deploy/kubernetes/inventory/disk-probe-local-path.yaml
# kubectl wait -n evolution-dev --for=condition=complete --timeout=600s \
# job -l app.kubernetes.io/name=evo-disk-probe
# kubectl logs -n evolution-dev -l app.kubernetes.io/name=evo-disk-probe --prefix
# kubectl delete -f deploy/kubernetes/inventory/disk-probe-local-path.yaml
# Deleting the PVCs deletes the probe data (reclaim Delete); nothing else is touched.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: evo-disk-probe
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: evo-disk-probe
policyTypes: [Ingress, Egress] # No traffic in or out.
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: evo-disk-probe-w1
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 2Gi
---
apiVersion: batch/v1
kind: Job
metadata:
name: evo-disk-probe-w1
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
backoffLimit: 0
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
restartPolicy: Never
automountServiceAccountToken: false
imagePullSecrets:
- name: harbor-pull
nodeSelector:
kubernetes.io/hostname: rke2-worker-1
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
command: ["/bin/bash", "-c"]
args:
- |
set -eu
cd /data
ns() { date +%s%N; }
echo "node=rke2-worker-1 filesystem: $(df -hT /data | tail -1)"
t0=$(ns)
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
t1=$(ns)
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
head -c 40960 /dev/urandom > payload.src
mkdir -p objects
worst=0; t0=$(ns)
for i in $(seq 1 200); do
s=$(ns)
dd if=payload.src of=objects/.tmp status=none conv=fsync
mv objects/.tmp objects/o$i
sync objects
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
done
t1=$(ns)
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
t0=$(ns)
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
t1=$(ns)
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
rm -rf dsync.bin payload.src objects seq.bin
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 500m, memory: 128Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumeMounts:
- {name: data, mountPath: /data}
volumes:
- name: data
persistentVolumeClaim:
claimName: evo-disk-probe-w1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: evo-disk-probe-w2
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 2Gi
---
apiVersion: batch/v1
kind: Job
metadata:
name: evo-disk-probe-w2
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
backoffLimit: 0
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
restartPolicy: Never
automountServiceAccountToken: false
imagePullSecrets:
- name: harbor-pull
nodeSelector:
kubernetes.io/hostname: rke2-worker-2
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
command: ["/bin/bash", "-c"]
args:
- |
set -eu
cd /data
ns() { date +%s%N; }
echo "node=rke2-worker-2 filesystem: $(df -hT /data | tail -1)"
t0=$(ns)
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
t1=$(ns)
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
head -c 40960 /dev/urandom > payload.src
mkdir -p objects
worst=0; t0=$(ns)
for i in $(seq 1 200); do
s=$(ns)
dd if=payload.src of=objects/.tmp status=none conv=fsync
mv objects/.tmp objects/o$i
sync objects
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
done
t1=$(ns)
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
t0=$(ns)
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
t1=$(ns)
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
rm -rf dsync.bin payload.src objects seq.bin
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 500m, memory: 128Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumeMounts:
- {name: data, mountPath: /data}
volumes:
- name: data
persistentVolumeClaim:
claimName: evo-disk-probe-w2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: evo-disk-probe-w3
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
accessModes: [ReadWriteOnce]
storageClassName: local-path
resources:
requests:
storage: 2Gi
---
apiVersion: batch/v1
kind: Job
metadata:
name: evo-disk-probe-w3
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
backoffLimit: 0
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
restartPolicy: Never
automountServiceAccountToken: false
imagePullSecrets:
- name: harbor-pull
nodeSelector:
kubernetes.io/hostname: rke2-worker-3
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
command: ["/bin/bash", "-c"]
args:
- |
set -eu
cd /data
ns() { date +%s%N; }
echo "node=rke2-worker-3 filesystem: $(df -hT /data | tail -1)"
t0=$(ns)
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
t1=$(ns)
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
head -c 40960 /dev/urandom > payload.src
mkdir -p objects
worst=0; t0=$(ns)
for i in $(seq 1 200); do
s=$(ns)
dd if=payload.src of=objects/.tmp status=none conv=fsync
mv objects/.tmp objects/o$i
sync objects
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
done
t1=$(ns)
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
t0=$(ns)
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
t1=$(ns)
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
rm -rf dsync.bin payload.src objects seq.bin
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 500m, memory: 128Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumeMounts:
- {name: data, mountPath: /data}
volumes:
- name: data
persistentVolumeClaim:
claimName: evo-disk-probe-w3
+10
View File
@@ -975,3 +975,13 @@ Still needed:
- run the read-only `tools/inventory/collect-cluster.sh` with cluster access;
- hypervisor, VM and disk details from the maintainer;
- the storage probe on each candidate StorageClass, after approval.
M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md):
- RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor.
- `local-path` is the only storage (node-pinned, reclaim Delete).
- `aevolution-dev` syncs manually.
- No PostgreSQL 18.6 exists yet.
- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker.
Remaining for M8.1: the per-worker storage probe, which is approved and ready to
apply.
+20
View File
@@ -2600,3 +2600,23 @@ Evidence:
three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of
serving. The full PostgreSQL Release suite passed with it. The analysis build
compiled clean, and the format check passed.
## M8.1b cluster inventory - 2026-09-29
- The maintainer ran `tools/inventory/collect-cluster.sh`. It made read requests
only, and listed Secrets as name, type and key count. The output is in
`C:\tmp\cluster` on the workstation and is not committed. The results are
summarized in docs/infrastructure/INVENTORY.md:
- RKE2 v1.32.5 with one untainted control plane (6 vCPU / 11.7 GiB) and three
workers (8 vCPU / 31.3 GiB).
- No kubelet reservations, quotas or LimitRanges.
- Storage is `local-path` only (reclaim Delete, no CSI, no snapshots).
- Canal/Calico, ingress-nginx, MetalLB and cert-manager with Let's Encrypt.
- `aevolution-dev` has no automated sync.
- No PostgreSQL 18.6 in the cluster (Gitea runs 17.6).
- The collector ran as cluster-admin. A read-only identity is an M8.2 item.
- `deploy/kubernetes/inventory/disk-probe-local-path.yaml` parses as one
NetworkPolicy and three PVC/Job pairs, pinned to worker-1..3. It is outside
every Argo path and has **not yet been applied**, because this workstation has
no kubeconfig. The maintainer approved running it. Its results go in
INVENTORY.md.
+145 -41
View File
@@ -1,18 +1,24 @@
# Host and cluster inventory (M8.1)
Status: **partial, 2026-09-29.** This file separates what has been measured, what
the maintainer has stated, and what has not been inspected yet. Nothing in the
"Not inspected" table may be assumed. Budgets below are proposals until the
cluster facts and the storage probe replace their inputs.
Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.**
Sources, kept separate below:
- maintainer statements;
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran
(read requests only; Secrets listed by name, type and key count);
- local measurements.
The disks and RAID layout will change, so every storage number here describes
the current layout only.
## Maintainer-stated facts
| Fact | Source | Implication |
|---|---|---|
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. |
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
@@ -33,24 +39,96 @@ repository. Before M8.3 can use it, confirm:
- **Backup and failure domain.** Where does the database live? It is part of the
M8.3/M8.7 evidence.
## Not inspected (needs read-only cluster access)
## Cluster (collected 2026-09-29)
This workstation has no kubeconfig. Run
`tools/inventory/collect-cluster.sh <dir>` with a read-only (get/list) identity.
It never fetches Secret or ConfigMap data. Then record the results here.
**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the
maintainer states that memory is not overcommitted. Together the VMs hold about
108 GiB of the host's roughly 128 GB. Everything shares one physical failure
domain.
| Item | Collected by | Needed for |
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
quotas beyond the defaults:
- Memory allocatable equals capacity on every node, so no kube-reserved or
system-reserved memory is configured, and only the default eviction threshold
protects the node.
- The only priority classes are the two system ones.
- There are no ResourceQuotas or LimitRanges anywhere.
- PodDisruptionBudgets exist only for Gitea and Keycloak.
| Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem |
|---|---|---|---|---|---|---|---|
| rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) |
| rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi |
| rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi |
| rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi |
**Storage.**
- `local-path` (rancher.io/local-path) is the only StorageClass. It is the
default, reclaims with **Delete**, and binds WaitForFirstConsumer.
- There are no CSI drivers and no VolumeSnapshotClasses.
- Every existing PV is a directory pinned to one node. They are spread across all
four nodes, including the control plane.
Consequences for M8.3:
- A durable store on `local-path` lives and dies with one VM's disk. Its pod can
run only on that node.
- Deleting its PVC destroys the data, and so would an Argo prune of the PVC.
- Before any durable world data, M8.3 needs:
- a Retain-policy class (for example, a second local-path class with
`reclaimPolicy: Retain`) or a replicated CSI;
- off-node backups (M8.7).
**Network.**
- CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced
through Calico, and the existing deny-all policies rely on that.
- Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes.
- Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer
(Gitea SSH, 172.10.1.201).
- DNS: CoreDNS 1.12.1, two replicas.
**TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and
`letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`.
Every certificate is Ready except `stone-server/webclient-tls`, which belongs to
another project.
**Delivery (Argo CD).**
| Application | Path | Sync policy |
|---|---|---|
| Node count and roles, CPU, allocatable memory, kernel, runtime, taints | nodes.txt, node-resources.txt, nodes-describe.txt | Budgets, affinity plan (8.9) |
| Hypervisor, VM layout, host disks and RAID, overcommit | Maintainer (outside Kubernetes) | Failure domains, the 128 GB envelope |
| CNI and NetworkPolicy enforcement | daemonsets.txt, kube-system-pods.txt | The deny-by-default flows the region example relies on |
| CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs | storageclasses.*, pv.txt, csidrivers.txt | 8.3. A local PV binds a region to one node (risk named in the roadmap) |
| Volume snapshots | volumesnapshotclasses.txt | Backup design (8.7) |
| Ingress, load balancers, DNS, certificate issuers | ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt | Metrics and probe exposure, later gateway |
| Quotas, limit ranges, priority classes, PDBs | quotas.txt, priorityclasses.txt, pdbs.txt | 8.2 isolation |
| Argo CD applications, auto-sync/prune, projects | argocd-apps.txt, argocd-projects.yaml | 8.2 World0 prune protection |
| Current usage | top-nodes.txt, top-pods.txt (needs metrics-server) | Available capacity |
| Storage latency per candidate class | `deploy/kubernetes/inventory/disk-probe.yaml.example`, applied by hand | Commit latency and lease margins |
| `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync |
| `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal |
| `aeternalis` | other repository | Manual |
The maintainer's statement that Argo deploys on every change applies to the
automated applications, not to AEvolution today. The `deploy/README.md` guidance
(manual full sync, no automated prune) matches the live application. Keep it that
way for anything durable. Harbor publication on push is confirmed by the CI runs.
**`evolution-dev` contents.**
- The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all
NetworkPolicy.
- ServiceAccounts `aevolution-proof`, plus three created by the maintainer
shortly before collection: `aevolution-migrator`, `aevolution-region` and
`aevolution-backup`.
- Secrets `harbor-pull`, `aevolution-db` (13 keys), and
`evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each,
also new). Their values were not read. The separate migrator and runtime
Secrets match the ADR-0060 role split.
**Existing data services.**
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3
therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must
first check what `aevolution-db` points at.
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
The region host's object store is a filesystem PVC (ADR-0061), so no object
server is required yet.
**Collector identity.** The output was collected as `system:admin`
(system:masters). The collector made only read requests. M8.2 should create a
genuinely read-only inventory identity, so this is not repeated with
cluster-admin.
## Measured workload profile (local fixture)
@@ -88,26 +166,52 @@ The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
restore and verification buffers several times over. Larger profiles
(populations, fields) must be re-measured before these values are reused.
## Proposed budget model (to be replaced by measured inputs)
Reserve per 32 GB worker before placing any region:
- **Operating system and Kubernetes.** Kubelet, system and eviction reserves, and
the CNI/CSI agents: take the difference between capacity and allocatable from
`node-resources.txt`, plus the measured daemon usage.
- **Stores.** PostgreSQL memory (shared_buffers, work_mem × connections,
maintenance) and the object store, on the node or nodes that host them, sized by
M8.3 rather than guessed.
- **Observability.** Metrics and log agents, and at most 8 KiB of exposition per
region scrape (ADR-0066).
- **Recovery headroom.** A successor host can start while its fenced predecessor
still holds memory. Budget two region footprints per region. On the database
and object store, keep headroom for audit and clone work (read-only, bounded by
payload × chain length).
## Budget (proposal from measured inputs, 2026-09-29)
Region hosts are not the memory driver at the current profile size (see the
measured peak above). The first capacity limits are expected to be:
- storage latency and fsync behaviour;
**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the
hypervisor and host. Adding memory to a VM is possible (maintainer), but it does
not add a failure domain.
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
| Reserve | Proposal | Basis |
|---|---|---|
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm |
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
**Proposed dev placement** (to confirm in M8.3/8.4):
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
with 38 % used, and PVs there are pinned to it.
- Place PostgreSQL and the region host's object PVC on **one** worker (currently
the least-used, worker-2), using a node label and affinity.
- local-path pins both volumes to that node anyway. Choosing the node explicitly
keeps the failure domain visible, and no replica count is claimed.
Even with these reservations, the AEvolution dev footprint stays under 3 GiB on
that worker.
The first capacity limits are expected to be:
- storage latency and fsync behaviour (probe pending);
- database connections (one per host plus the migrator);
- the lease margins: the 2 s renewal deadline and 5 s commit deadline
(ADR-0060) must stay well above p99 storage plus database latency.
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
storage plus database latency.
Do not set per-node region counts until the probe results exist.
Per-node region counts and CPU budgets are set only after the in-cluster probe
and M8.4's in-cluster measurement.
## In-cluster storage probe
`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job
per worker on `local-path`, pinned by hostname. It is outside every Argo path and
applied by hand, with the maintainer's approval (2026-09-29). The commands are in
the file header. Record each worker's line here and compare it with the local
baseline above.
| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|---|---|---|---|
| rke2-worker-1 | pending | pending | pending |
| rke2-worker-2 | pending | pending | pending |
| rke2-worker-3 | pending | pending | pending |