diff --git a/deploy/README.md b/deploy/README.md index 115bb96..6aed57a 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -74,10 +74,10 @@ of the live application's configuration: | Project | aevolution-dev, or an equivalently restricted project | | Sync | Manual full sync; no automated prune/self-heal (recommended) | -The maintainer reports (2026-09-29) that Argo CD currently deploys each Git change -automatically. Treat every file under a watched path as live on push, and confirm -the actual sync and prune policy with `tools/inventory/collect-cluster.sh` -(docs/infrastructure/INVENTORY.md). +Inventory (2026-09-29, docs/infrastructure/INVENTORY.md): the live `aevolution-dev` +application has no automated sync, which matches this guidance. Other projects' +applications do auto-sync with prune. Keep AEvolution durable resources out of any +automated-prune application. Bootstrap examples in `deploy/argocd/*.yaml.example` are outside the workload path. Replace the Argo namespace and registered cluster API URL, or use its UI. diff --git a/deploy/kubernetes/inventory/disk-probe-local-path.yaml b/deploy/kubernetes/inventory/disk-probe-local-path.yaml new file mode 100644 index 0000000..5699f49 --- /dev/null +++ b/deploy/kubernetes/inventory/disk-probe-local-path.yaml @@ -0,0 +1,291 @@ +# M8.1 storage latency probe for the cluster's only StorageClass, local-path +# (rancher.io/local-path: node-local directories, reclaim Delete, +# WaitForFirstConsumer). A local-path volume is created on the node that first +# runs its pod, so each worker gets its own PVC and Job pinned by hostname. +# Generated from disk-probe.yaml.example; outside every Argo path, applied by hand +# with the maintainer's approval (2026-09-29): +# kubectl apply -f deploy/kubernetes/inventory/disk-probe-local-path.yaml +# kubectl wait -n evolution-dev --for=condition=complete --timeout=600s \ +# job -l app.kubernetes.io/name=evo-disk-probe +# kubectl logs -n evolution-dev -l app.kubernetes.io/name=evo-disk-probe --prefix +# kubectl delete -f deploy/kubernetes/inventory/disk-probe-local-path.yaml +# Deleting the PVCs deletes the probe data (reclaim Delete); nothing else is touched. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: evo-disk-probe + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: evo-disk-probe + policyTypes: [Ingress, Egress] # No traffic in or out. +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w1 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w1 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-1 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-1 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w1 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w2 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w2 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-2 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-2 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w2 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w3 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w3 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-3 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-3 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w3 diff --git a/docs/STATUS.md b/docs/STATUS.md index a2fb1a4..da8e641 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -975,3 +975,13 @@ Still needed: - run the read-only `tools/inventory/collect-cluster.sh` with cluster access; - hypervisor, VM and disk details from the maintainer; - the storage probe on each candidate StorageClass, after approval. + +M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md): +- RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor. +- `local-path` is the only storage (node-pinned, reclaim Delete). +- `aevolution-dev` syncs manually. +- No PostgreSQL 18.6 exists yet. +- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker. + +Remaining for M8.1: the per-worker storage probe, which is approved and ready to +apply. diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 0dc0521..2cd45b6 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2600,3 +2600,23 @@ Evidence: three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of serving. The full PostgreSQL Release suite passed with it. The analysis build compiled clean, and the format check passed. + +## M8.1b cluster inventory - 2026-09-29 + +- The maintainer ran `tools/inventory/collect-cluster.sh`. It made read requests + only, and listed Secrets as name, type and key count. The output is in + `C:\tmp\cluster` on the workstation and is not committed. The results are + summarized in docs/infrastructure/INVENTORY.md: + - RKE2 v1.32.5 with one untainted control plane (6 vCPU / 11.7 GiB) and three + workers (8 vCPU / 31.3 GiB). + - No kubelet reservations, quotas or LimitRanges. + - Storage is `local-path` only (reclaim Delete, no CSI, no snapshots). + - Canal/Calico, ingress-nginx, MetalLB and cert-manager with Let's Encrypt. + - `aevolution-dev` has no automated sync. + - No PostgreSQL 18.6 in the cluster (Gitea runs 17.6). +- The collector ran as cluster-admin. A read-only identity is an M8.2 item. +- `deploy/kubernetes/inventory/disk-probe-local-path.yaml` parses as one + NetworkPolicy and three PVC/Job pairs, pinned to worker-1..3. It is outside + every Argo path and has **not yet been applied**, because this workstation has + no kubeconfig. The maintainer approved running it. Its results go in + INVENTORY.md. diff --git a/docs/infrastructure/INVENTORY.md b/docs/infrastructure/INVENTORY.md index 1a27fc4..eb2efcd 100644 --- a/docs/infrastructure/INVENTORY.md +++ b/docs/infrastructure/INVENTORY.md @@ -1,18 +1,24 @@ # Host and cluster inventory (M8.1) -Status: **partial, 2026-09-29.** This file separates what has been measured, what -the maintainer has stated, and what has not been inspected yet. Nothing in the -"Not inspected" table may be assumed. Budgets below are proposals until the -cluster facts and the storage probe replace their inputs. +Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.** +Sources, kept separate below: +- maintainer statements; +- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran + (read requests only; Secrets listed by name, type and key count); +- local measurements. + +The disks and RAID layout will change, so every storage number here describes +the current layout only. ## Maintainer-stated facts | Fact | Source | Implication | |---|---|---| | One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. | -| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. | +| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. | +| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. | | Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. | -| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. | +| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. | | Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. | **The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`, @@ -33,24 +39,96 @@ repository. Before M8.3 can use it, confirm: - **Backup and failure domain.** Where does the database live? It is part of the M8.3/M8.7 evidence. -## Not inspected (needs read-only cluster access) +## Cluster (collected 2026-09-29) -This workstation has no kubeconfig. Run -`tools/inventory/collect-cluster.sh ` with a read-only (get/list) identity. -It never fetches Secret or ConfigMap data. Then record the results here. +**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the +maintainer states that memory is not overcommitted. Together the VMs hold about +108 GiB of the host's roughly 128 GB. Everything shares one physical failure +domain. -| Item | Collected by | Needed for | +The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or +quotas beyond the defaults: +- Memory allocatable equals capacity on every node, so no kube-reserved or + system-reserved memory is configured, and only the default eviction threshold + protects the node. +- The only priority classes are the two system ones. +- There are no ResourceQuotas or LimitRanges anywhere. +- PodDisruptionBudgets exist only for Gitea and Keycloak. + +| Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem | +|---|---|---|---|---|---|---|---| +| rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) | +| rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi | +| rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi | +| rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi | + +**Storage.** +- `local-path` (rancher.io/local-path) is the only StorageClass. It is the + default, reclaims with **Delete**, and binds WaitForFirstConsumer. +- There are no CSI drivers and no VolumeSnapshotClasses. +- Every existing PV is a directory pinned to one node. They are spread across all + four nodes, including the control plane. + +Consequences for M8.3: +- A durable store on `local-path` lives and dies with one VM's disk. Its pod can + run only on that node. +- Deleting its PVC destroys the data, and so would an Argo prune of the PVC. +- Before any durable world data, M8.3 needs: + - a Retain-policy class (for example, a second local-path class with + `reclaimPolicy: Retain`) or a replicated CSI; + - off-node backups (M8.7). + +**Network.** +- CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced + through Calico, and the existing deny-all policies rely on that. +- Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes. +- Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer + (Gitea SSH, 172.10.1.201). +- DNS: CoreDNS 1.12.1, two replicas. + +**TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and +`letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`. +Every certificate is Ready except `stone-server/webclient-tls`, which belongs to +another project. + +**Delivery (Argo CD).** + +| Application | Path | Sync policy | |---|---|---| -| Node count and roles, CPU, allocatable memory, kernel, runtime, taints | nodes.txt, node-resources.txt, nodes-describe.txt | Budgets, affinity plan (8.9) | -| Hypervisor, VM layout, host disks and RAID, overcommit | Maintainer (outside Kubernetes) | Failure domains, the 128 GB envelope | -| CNI and NetworkPolicy enforcement | daemonsets.txt, kube-system-pods.txt | The deny-by-default flows the region example relies on | -| CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs | storageclasses.*, pv.txt, csidrivers.txt | 8.3. A local PV binds a region to one node (risk named in the roadmap) | -| Volume snapshots | volumesnapshotclasses.txt | Backup design (8.7) | -| Ingress, load balancers, DNS, certificate issuers | ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt | Metrics and probe exposure, later gateway | -| Quotas, limit ranges, priority classes, PDBs | quotas.txt, priorityclasses.txt, pdbs.txt | 8.2 isolation | -| Argo CD applications, auto-sync/prune, projects | argocd-apps.txt, argocd-projects.yaml | 8.2 World0 prune protection | -| Current usage | top-nodes.txt, top-pods.txt (needs metrics-server) | Available capacity | -| Storage latency per candidate class | `deploy/kubernetes/inventory/disk-probe.yaml.example`, applied by hand | Commit latency and lease margins | +| `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync | +| `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal | +| `aeternalis` | other repository | Manual | + +The maintainer's statement that Argo deploys on every change applies to the +automated applications, not to AEvolution today. The `deploy/README.md` guidance +(manual full sync, no automated prune) matches the live application. Keep it that +way for anything durable. Harbor publication on push is confirmed by the CI runs. + +**`evolution-dev` contents.** +- The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all + NetworkPolicy. +- ServiceAccounts `aevolution-proof`, plus three created by the maintainer + shortly before collection: `aevolution-migrator`, `aevolution-region` and + `aevolution-backup`. +- Secrets `harbor-pull`, `aevolution-db` (13 keys), and + `evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each, + also new). Their values were not read. The separate migrator and runtime + Secrets match the ADR-0060 role split. + +**Existing data services.** +- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs + harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3 + therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must + first check what `aevolution-db` points at. +- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned. + INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default. + The region host's object store is a filesystem PVC (ADR-0061), so no object + server is required yet. + +**Collector identity.** The output was collected as `system:admin` +(system:masters). The collector made only read requests. M8.2 should create a +genuinely read-only inventory identity, so this is not repeated with +cluster-admin. ## Measured workload profile (local fixture) @@ -88,26 +166,52 @@ The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and restore and verification buffers several times over. Larger profiles (populations, fields) must be re-measured before these values are reused. -## Proposed budget model (to be replaced by measured inputs) -Reserve per 32 GB worker before placing any region: -- **Operating system and Kubernetes.** Kubelet, system and eviction reserves, and - the CNI/CSI agents: take the difference between capacity and allocatable from - `node-resources.txt`, plus the measured daemon usage. -- **Stores.** PostgreSQL memory (shared_buffers, work_mem × connections, - maintenance) and the object store, on the node or nodes that host them, sized by - M8.3 rather than guessed. -- **Observability.** Metrics and log agents, and at most 8 KiB of exposition per - region scrape (ADR-0066). -- **Recovery headroom.** A successor host can start while its fenced predecessor - still holds memory. Budget two region footprints per region. On the database - and object store, keep headroom for audit and clone work (read-only, bounded by - payload × chain length). +## Budget (proposal from measured inputs, 2026-09-29) -Region hosts are not the memory driver at the current profile size (see the -measured peak above). The first capacity limits are expected to be: -- storage latency and fsync behaviour; +**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the +hypervisor and host. Adding memory to a VM is possible (maintainer), but it does +not add a failure domain. + +**Per worker** (31.3 GiB capacity, no kubelet reservation configured): + +| Reserve | Proposal | Basis | +|---|---|---| +| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services | +| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt | +| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm | +| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 | +| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB | + +**Proposed dev placement** (to confirm in M8.3/8.4): +- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB + with 38 % used, and PVs there are pinned to it. +- Place PostgreSQL and the region host's object PVC on **one** worker (currently + the least-used, worker-2), using a node label and affinity. +- local-path pins both volumes to that node anyway. Choosing the node explicitly + keeps the failure domain visible, and no replica count is claimed. + +Even with these reservations, the AEvolution dev footprint stays under 3 GiB on +that worker. + +The first capacity limits are expected to be: +- storage latency and fsync behaviour (probe pending); - database connections (one per host plus the migrator); -- the lease margins: the 2 s renewal deadline and 5 s commit deadline - (ADR-0060) must stay well above p99 storage plus database latency. +- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99 + storage plus database latency. -Do not set per-node region counts until the probe results exist. +Per-node region counts and CPU budgets are set only after the in-cluster probe +and M8.4's in-cluster measurement. + +## In-cluster storage probe + +`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job +per worker on `local-path`, pinned by hostname. It is outside every Argo path and +applied by hand, with the maintainer's approval (2026-09-29). The commands are in +the file header. Record each worker's line here and compare it with the local +baseline above. + +| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync | +|---|---|---|---| +| rke2-worker-1 | pending | pending | pending | +| rke2-worker-2 | pending | pending | pending | +| rke2-worker-3 | pending | pending | pending |