diff --git a/deploy/README.md b/deploy/README.md index 115bb96..6aed57a 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -74,10 +74,10 @@ of the live application's configuration: | Project | aevolution-dev, or an equivalently restricted project | | Sync | Manual full sync; no automated prune/self-heal (recommended) | -The maintainer reports (2026-09-29) that Argo CD currently deploys each Git change -automatically. Treat every file under a watched path as live on push, and confirm -the actual sync and prune policy with `tools/inventory/collect-cluster.sh` -(docs/infrastructure/INVENTORY.md). +Inventory (2026-09-29, docs/infrastructure/INVENTORY.md): the live `aevolution-dev` +application has no automated sync, which matches this guidance. Other projects' +applications do auto-sync with prune. Keep AEvolution durable resources out of any +automated-prune application. Bootstrap examples in `deploy/argocd/*.yaml.example` are outside the workload path. Replace the Argo namespace and registered cluster API URL, or use its UI. diff --git a/deploy/kubernetes/inventory/disk-probe-local-path.yaml b/deploy/kubernetes/inventory/disk-probe-local-path.yaml new file mode 100644 index 0000000..5699f49 --- /dev/null +++ b/deploy/kubernetes/inventory/disk-probe-local-path.yaml @@ -0,0 +1,291 @@ +# M8.1 storage latency probe for the cluster's only StorageClass, local-path +# (rancher.io/local-path: node-local directories, reclaim Delete, +# WaitForFirstConsumer). A local-path volume is created on the node that first +# runs its pod, so each worker gets its own PVC and Job pinned by hostname. +# Generated from disk-probe.yaml.example; outside every Argo path, applied by hand +# with the maintainer's approval (2026-09-29): +# kubectl apply -f deploy/kubernetes/inventory/disk-probe-local-path.yaml +# kubectl wait -n evolution-dev --for=condition=complete --timeout=600s \ +# job -l app.kubernetes.io/name=evo-disk-probe +# kubectl logs -n evolution-dev -l app.kubernetes.io/name=evo-disk-probe --prefix +# kubectl delete -f deploy/kubernetes/inventory/disk-probe-local-path.yaml +# Deleting the PVCs deletes the probe data (reclaim Delete); nothing else is touched. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: evo-disk-probe + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: evo-disk-probe + policyTypes: [Ingress, Egress] # No traffic in or out. +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w1 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w1 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-1 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-1 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w1 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w2 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w2 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-2 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-2 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w2 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: evo-disk-probe-w3 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + accessModes: [ReadWriteOnce] + storageClassName: local-path + resources: + requests: + storage: 2Gi +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: evo-disk-probe-w3 + namespace: evolution-dev + labels: + app.kubernetes.io/name: evo-disk-probe +spec: + backoffLimit: 0 + activeDeadlineSeconds: 600 + template: + metadata: + labels: + app.kubernetes.io/name: evo-disk-probe + spec: + restartPolicy: Never + automountServiceAccountToken: false + imagePullSecrets: + - name: harbor-pull + nodeSelector: + kubernetes.io/hostname: rke2-worker-3 + securityContext: + runAsNonRoot: true + runAsUser: 10001 + runAsGroup: 10001 + fsGroup: 10001 + seccompProfile: + type: RuntimeDefault + containers: + - name: probe + # Same repository and digest as deploy/kubernetes/dev (verified run 1399). + image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d + command: ["/bin/bash", "-c"] + args: + - | + set -eu + cd /data + ns() { date +%s%N; } + echo "node=rke2-worker-3 filesystem: $(df -hT /data | tail -1)" + t0=$(ns) + dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none + t1=$(ns) + echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))" + head -c 40960 /dev/urandom > payload.src + mkdir -p objects + worst=0; t0=$(ns) + for i in $(seq 1 200); do + s=$(ns) + dd if=payload.src of=objects/.tmp status=none conv=fsync + mv objects/.tmp objects/o$i + sync objects + e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e + done + t1=$(ns) + echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst" + t0=$(ns) + dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none + t1=$(ns) + echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))" + rm -rf dsync.bin payload.src objects seq.bin + resources: + requests: {cpu: 100m, memory: 64Mi} + limits: {cpu: 500m, memory: 128Mi} + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: [ALL] + volumeMounts: + - {name: data, mountPath: /data} + volumes: + - name: data + persistentVolumeClaim: + claimName: evo-disk-probe-w3 diff --git a/docs/STATUS.md b/docs/STATUS.md index a2fb1a4..da8e641 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -975,3 +975,13 @@ Still needed: - run the read-only `tools/inventory/collect-cluster.sh` with cluster access; - hypervisor, VM and disk details from the maintainer; - the storage probe on each candidate StorageClass, after approval. + +M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md): +- RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor. +- `local-path` is the only storage (node-pinned, reclaim Delete). +- `aevolution-dev` syncs manually. +- No PostgreSQL 18.6 exists yet. +- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker. + +Remaining for M8.1: the per-worker storage probe, which is approved and ready to +apply. diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 0dc0521..2cd45b6 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2600,3 +2600,23 @@ Evidence: three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of serving. The full PostgreSQL Release suite passed with it. The analysis build compiled clean, and the format check passed. + +## M8.1b cluster inventory - 2026-09-29 + +- The maintainer ran `tools/inventory/collect-cluster.sh`. It made read requests + only, and listed Secrets as name, type and key count. The output is in + `C:\tmp\cluster` on the workstation and is not committed. The results are + summarized in docs/infrastructure/INVENTORY.md: + - RKE2 v1.32.5 with one untainted control plane (6 vCPU / 11.7 GiB) and three + workers (8 vCPU / 31.3 GiB). + - No kubelet reservations, quotas or LimitRanges. + - Storage is `local-path` only (reclaim Delete, no CSI, no snapshots). + - Canal/Calico, ingress-nginx, MetalLB and cert-manager with Let's Encrypt. + - `aevolution-dev` has no automated sync. + - No PostgreSQL 18.6 in the cluster (Gitea runs 17.6). +- The collector ran as cluster-admin. A read-only identity is an M8.2 item. +- `deploy/kubernetes/inventory/disk-probe-local-path.yaml` parses as one + NetworkPolicy and three PVC/Job pairs, pinned to worker-1..3. It is outside + every Argo path and has **not yet been applied**, because this workstation has + no kubeconfig. The maintainer approved running it. Its results go in + INVENTORY.md. diff --git a/docs/infrastructure/INVENTORY.md b/docs/infrastructure/INVENTORY.md index 1a27fc4..eb2efcd 100644 --- a/docs/infrastructure/INVENTORY.md +++ b/docs/infrastructure/INVENTORY.md @@ -1,18 +1,24 @@ # Host and cluster inventory (M8.1) -Status: **partial, 2026-09-29.** This file separates what has been measured, what -the maintainer has stated, and what has not been inspected yet. Nothing in the -"Not inspected" table may be assumed. Budgets below are proposals until the -cluster facts and the storage probe replace their inputs. +Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.** +Sources, kept separate below: +- maintainer statements; +- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran + (read requests only; Secrets listed by name, type and key count); +- local measurements. + +The disks and RAID layout will change, so every storage number here describes +the current layout only. ## Maintainer-stated facts | Fact | Source | Implication | |---|---|---| | One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. | -| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. | +| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. | +| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. | | Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. | -| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. | +| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. | | Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. | **The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`, @@ -33,24 +39,96 @@ repository. Before M8.3 can use it, confirm: - **Backup and failure domain.** Where does the database live? It is part of the M8.3/M8.7 evidence. -## Not inspected (needs read-only cluster access) +## Cluster (collected 2026-09-29) -This workstation has no kubeconfig. Run -`tools/inventory/collect-cluster.sh