Record M8.1 cluster inventory and per-worker storage probe
Summarize the maintainer-collected read-only inventory: RKE2 v1.32.5, an untainted control plane and three 32 GB workers on one hypervisor, no kubelet reservations or quotas, local-path as the only (node-pinned, reclaim Delete) storage, Canal/ingress-nginx/MetalLB/cert-manager, a manually synced aevolution-dev application and no PostgreSQL 18.6. Propose a measured budget and dev placement, and add a ready-to-apply local-path probe with one PVC and Job pinned to each worker. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+4
-4
@@ -74,10 +74,10 @@ of the live application's configuration:
|
||||
| Project | aevolution-dev, or an equivalently restricted project |
|
||||
| Sync | Manual full sync; no automated prune/self-heal (recommended) |
|
||||
|
||||
The maintainer reports (2026-09-29) that Argo CD currently deploys each Git change
|
||||
automatically. Treat every file under a watched path as live on push, and confirm
|
||||
the actual sync and prune policy with `tools/inventory/collect-cluster.sh`
|
||||
(docs/infrastructure/INVENTORY.md).
|
||||
Inventory (2026-09-29, docs/infrastructure/INVENTORY.md): the live `aevolution-dev`
|
||||
application has no automated sync, which matches this guidance. Other projects'
|
||||
applications do auto-sync with prune. Keep AEvolution durable resources out of any
|
||||
automated-prune application.
|
||||
|
||||
Bootstrap examples in `deploy/argocd/*.yaml.example` are outside the workload
|
||||
path. Replace the Argo namespace and registered cluster API URL, or use its UI.
|
||||
|
||||
@@ -0,0 +1,291 @@
|
||||
# M8.1 storage latency probe for the cluster's only StorageClass, local-path
|
||||
# (rancher.io/local-path: node-local directories, reclaim Delete,
|
||||
# WaitForFirstConsumer). A local-path volume is created on the node that first
|
||||
# runs its pod, so each worker gets its own PVC and Job pinned by hostname.
|
||||
# Generated from disk-probe.yaml.example; outside every Argo path, applied by hand
|
||||
# with the maintainer's approval (2026-09-29):
|
||||
# kubectl apply -f deploy/kubernetes/inventory/disk-probe-local-path.yaml
|
||||
# kubectl wait -n evolution-dev --for=condition=complete --timeout=600s \
|
||||
# job -l app.kubernetes.io/name=evo-disk-probe
|
||||
# kubectl logs -n evolution-dev -l app.kubernetes.io/name=evo-disk-probe --prefix
|
||||
# kubectl delete -f deploy/kubernetes/inventory/disk-probe-local-path.yaml
|
||||
# Deleting the PVCs deletes the probe data (reclaim Delete); nothing else is touched.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: evo-disk-probe
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
policyTypes: [Ingress, Egress] # No traffic in or out.
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: evo-disk-probe-w1
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evo-disk-probe-w1
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
imagePullSecrets:
|
||||
- name: harbor-pull
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: rke2-worker-1
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: probe
|
||||
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
|
||||
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
cd /data
|
||||
ns() { date +%s%N; }
|
||||
echo "node=rke2-worker-1 filesystem: $(df -hT /data | tail -1)"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
|
||||
t1=$(ns)
|
||||
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
|
||||
head -c 40960 /dev/urandom > payload.src
|
||||
mkdir -p objects
|
||||
worst=0; t0=$(ns)
|
||||
for i in $(seq 1 200); do
|
||||
s=$(ns)
|
||||
dd if=payload.src of=objects/.tmp status=none conv=fsync
|
||||
mv objects/.tmp objects/o$i
|
||||
sync objects
|
||||
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
|
||||
done
|
||||
t1=$(ns)
|
||||
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
|
||||
t1=$(ns)
|
||||
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
|
||||
rm -rf dsync.bin payload.src objects seq.bin
|
||||
resources:
|
||||
requests: {cpu: 100m, memory: 64Mi}
|
||||
limits: {cpu: 500m, memory: 128Mi}
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: evo-disk-probe-w1
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: evo-disk-probe-w2
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evo-disk-probe-w2
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
imagePullSecrets:
|
||||
- name: harbor-pull
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: rke2-worker-2
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: probe
|
||||
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
|
||||
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
cd /data
|
||||
ns() { date +%s%N; }
|
||||
echo "node=rke2-worker-2 filesystem: $(df -hT /data | tail -1)"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
|
||||
t1=$(ns)
|
||||
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
|
||||
head -c 40960 /dev/urandom > payload.src
|
||||
mkdir -p objects
|
||||
worst=0; t0=$(ns)
|
||||
for i in $(seq 1 200); do
|
||||
s=$(ns)
|
||||
dd if=payload.src of=objects/.tmp status=none conv=fsync
|
||||
mv objects/.tmp objects/o$i
|
||||
sync objects
|
||||
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
|
||||
done
|
||||
t1=$(ns)
|
||||
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
|
||||
t1=$(ns)
|
||||
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
|
||||
rm -rf dsync.bin payload.src objects seq.bin
|
||||
resources:
|
||||
requests: {cpu: 100m, memory: 64Mi}
|
||||
limits: {cpu: 500m, memory: 128Mi}
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: evo-disk-probe-w2
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: evo-disk-probe-w3
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: local-path
|
||||
resources:
|
||||
requests:
|
||||
storage: 2Gi
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evo-disk-probe-w3
|
||||
namespace: evolution-dev
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
activeDeadlineSeconds: 600
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: evo-disk-probe
|
||||
spec:
|
||||
restartPolicy: Never
|
||||
automountServiceAccountToken: false
|
||||
imagePullSecrets:
|
||||
- name: harbor-pull
|
||||
nodeSelector:
|
||||
kubernetes.io/hostname: rke2-worker-3
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: probe
|
||||
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
|
||||
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
cd /data
|
||||
ns() { date +%s%N; }
|
||||
echo "node=rke2-worker-3 filesystem: $(df -hT /data | tail -1)"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
|
||||
t1=$(ns)
|
||||
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
|
||||
head -c 40960 /dev/urandom > payload.src
|
||||
mkdir -p objects
|
||||
worst=0; t0=$(ns)
|
||||
for i in $(seq 1 200); do
|
||||
s=$(ns)
|
||||
dd if=payload.src of=objects/.tmp status=none conv=fsync
|
||||
mv objects/.tmp objects/o$i
|
||||
sync objects
|
||||
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
|
||||
done
|
||||
t1=$(ns)
|
||||
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
|
||||
t0=$(ns)
|
||||
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
|
||||
t1=$(ns)
|
||||
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
|
||||
rm -rf dsync.bin payload.src objects seq.bin
|
||||
resources:
|
||||
requests: {cpu: 100m, memory: 64Mi}
|
||||
limits: {cpu: 500m, memory: 128Mi}
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: [ALL]
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: evo-disk-probe-w3
|
||||
@@ -975,3 +975,13 @@ Still needed:
|
||||
- run the read-only `tools/inventory/collect-cluster.sh` with cluster access;
|
||||
- hypervisor, VM and disk details from the maintainer;
|
||||
- the storage probe on each candidate StorageClass, after approval.
|
||||
|
||||
M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md):
|
||||
- RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor.
|
||||
- `local-path` is the only storage (node-pinned, reclaim Delete).
|
||||
- `aevolution-dev` syncs manually.
|
||||
- No PostgreSQL 18.6 exists yet.
|
||||
- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker.
|
||||
|
||||
Remaining for M8.1: the per-worker storage probe, which is approved and ready to
|
||||
apply.
|
||||
|
||||
@@ -2600,3 +2600,23 @@ Evidence:
|
||||
three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of
|
||||
serving. The full PostgreSQL Release suite passed with it. The analysis build
|
||||
compiled clean, and the format check passed.
|
||||
|
||||
## M8.1b cluster inventory - 2026-09-29
|
||||
|
||||
- The maintainer ran `tools/inventory/collect-cluster.sh`. It made read requests
|
||||
only, and listed Secrets as name, type and key count. The output is in
|
||||
`C:\tmp\cluster` on the workstation and is not committed. The results are
|
||||
summarized in docs/infrastructure/INVENTORY.md:
|
||||
- RKE2 v1.32.5 with one untainted control plane (6 vCPU / 11.7 GiB) and three
|
||||
workers (8 vCPU / 31.3 GiB).
|
||||
- No kubelet reservations, quotas or LimitRanges.
|
||||
- Storage is `local-path` only (reclaim Delete, no CSI, no snapshots).
|
||||
- Canal/Calico, ingress-nginx, MetalLB and cert-manager with Let's Encrypt.
|
||||
- `aevolution-dev` has no automated sync.
|
||||
- No PostgreSQL 18.6 in the cluster (Gitea runs 17.6).
|
||||
- The collector ran as cluster-admin. A read-only identity is an M8.2 item.
|
||||
- `deploy/kubernetes/inventory/disk-probe-local-path.yaml` parses as one
|
||||
NetworkPolicy and three PVC/Job pairs, pinned to worker-1..3. It is outside
|
||||
every Argo path and has **not yet been applied**, because this workstation has
|
||||
no kubeconfig. The maintainer approved running it. Its results go in
|
||||
INVENTORY.md.
|
||||
|
||||
@@ -1,18 +1,24 @@
|
||||
# Host and cluster inventory (M8.1)
|
||||
|
||||
Status: **partial, 2026-09-29.** This file separates what has been measured, what
|
||||
the maintainer has stated, and what has not been inspected yet. Nothing in the
|
||||
"Not inspected" table may be assumed. Budgets below are proposals until the
|
||||
cluster facts and the storage probe replace their inputs.
|
||||
Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.**
|
||||
Sources, kept separate below:
|
||||
- maintainer statements;
|
||||
- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran
|
||||
(read requests only; Secrets listed by name, type and key count);
|
||||
- local measurements.
|
||||
|
||||
The disks and RAID layout will change, so every storage number here describes
|
||||
the current layout only.
|
||||
|
||||
## Maintainer-stated facts
|
||||
|
||||
| Fact | Source | Implication |
|
||||
|---|---|---|
|
||||
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
|
||||
| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. |
|
||||
| One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. |
|
||||
| Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. |
|
||||
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
|
||||
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. |
|
||||
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. |
|
||||
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
|
||||
|
||||
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
|
||||
@@ -33,24 +39,96 @@ repository. Before M8.3 can use it, confirm:
|
||||
- **Backup and failure domain.** Where does the database live? It is part of the
|
||||
M8.3/M8.7 evidence.
|
||||
|
||||
## Not inspected (needs read-only cluster access)
|
||||
## Cluster (collected 2026-09-29)
|
||||
|
||||
This workstation has no kubeconfig. Run
|
||||
`tools/inventory/collect-cluster.sh <dir>` with a read-only (get/list) identity.
|
||||
It never fetches Secret or ConfigMap data. Then record the results here.
|
||||
**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the
|
||||
maintainer states that memory is not overcommitted. Together the VMs hold about
|
||||
108 GiB of the host's roughly 128 GB. Everything shares one physical failure
|
||||
domain.
|
||||
|
||||
| Item | Collected by | Needed for |
|
||||
The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or
|
||||
quotas beyond the defaults:
|
||||
- Memory allocatable equals capacity on every node, so no kube-reserved or
|
||||
system-reserved memory is configured, and only the default eviction threshold
|
||||
protects the node.
|
||||
- The only priority classes are the two system ones.
|
||||
- There are no ResourceQuotas or LimitRanges anywhere.
|
||||
- PodDisruptionBudgets exist only for Gitea and Keycloak.
|
||||
|
||||
| Node | vCPU | Memory | Ephemeral | Taints | Requests CPU / mem | Limits CPU / mem | Used CPU / mem |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| rke2-master-1 (control plane) | 6 | 11.7 GiB | 48 GiB | none, so workloads schedule here | 2675m / 3890Mi | 3950m / 4032Mi | 548m / 4549Mi (38 %) |
|
||||
| rke2-worker-1 | 8 | 31.3 GiB | 97 GiB | none | 2200m / 1882Mi | 6150m / 4288Mi | 2416m / 2466Mi |
|
||||
| rke2-worker-2 | 8 | 31.3 GiB | 97 GiB | none | 2300m / 2010Mi | 6300m / 4480Mi | 382m / 2517Mi |
|
||||
| rke2-worker-3 | 8 | 31.3 GiB | 97 GiB | none | 3200m / 4058Mi | 8550m (106 %) / 8896Mi | 345m / 5955Mi |
|
||||
|
||||
**Storage.**
|
||||
- `local-path` (rancher.io/local-path) is the only StorageClass. It is the
|
||||
default, reclaims with **Delete**, and binds WaitForFirstConsumer.
|
||||
- There are no CSI drivers and no VolumeSnapshotClasses.
|
||||
- Every existing PV is a directory pinned to one node. They are spread across all
|
||||
four nodes, including the control plane.
|
||||
|
||||
Consequences for M8.3:
|
||||
- A durable store on `local-path` lives and dies with one VM's disk. Its pod can
|
||||
run only on that node.
|
||||
- Deleting its PVC destroys the data, and so would an Argo prune of the PVC.
|
||||
- Before any durable world data, M8.3 needs:
|
||||
- a Retain-policy class (for example, a second local-path class with
|
||||
`reclaimPolicy: Retain`) or a replicated CSI;
|
||||
- off-node backups (M8.7).
|
||||
|
||||
**Network.**
|
||||
- CNI: `rke2-canal` (Calico v3.30 with flannel). NetworkPolicy is enforced
|
||||
through Calico, and the existing deny-all policies rely on that.
|
||||
- Ingress: `nginx` (ingress-nginx 1.12.1) as a DaemonSet on all four nodes.
|
||||
- Load balancing: MetalLB 0.15.2 in FRR mode, with a single LoadBalancer
|
||||
(Gitea SSH, 172.10.1.201).
|
||||
- DNS: CoreDNS 1.12.1, two replicas.
|
||||
|
||||
**TLS.** cert-manager, with ClusterIssuers `letsencrypt-godaddy` and
|
||||
`letsencrypt-godaddy-wildcard`. The wildcard certificate is in `kube-system`.
|
||||
Every certificate is Ready except `stone-server/webclient-tls`, which belongs to
|
||||
another project.
|
||||
|
||||
**Delivery (Argo CD).**
|
||||
|
||||
| Application | Path | Sync policy |
|
||||
|---|---|---|
|
||||
| Node count and roles, CPU, allocatable memory, kernel, runtime, taints | nodes.txt, node-resources.txt, nodes-describe.txt | Budgets, affinity plan (8.9) |
|
||||
| Hypervisor, VM layout, host disks and RAID, overcommit | Maintainer (outside Kubernetes) | Failure domains, the 128 GB envelope |
|
||||
| CNI and NetworkPolicy enforcement | daemonsets.txt, kube-system-pods.txt | The deny-by-default flows the region example relies on |
|
||||
| CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs | storageclasses.*, pv.txt, csidrivers.txt | 8.3. A local PV binds a region to one node (risk named in the roadmap) |
|
||||
| Volume snapshots | volumesnapshotclasses.txt | Backup design (8.7) |
|
||||
| Ingress, load balancers, DNS, certificate issuers | ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt | Metrics and probe exposure, later gateway |
|
||||
| Quotas, limit ranges, priority classes, PDBs | quotas.txt, priorityclasses.txt, pdbs.txt | 8.2 isolation |
|
||||
| Argo CD applications, auto-sync/prune, projects | argocd-apps.txt, argocd-projects.yaml | 8.2 World0 prune protection |
|
||||
| Current usage | top-nodes.txt, top-pods.txt (needs metrics-server) | Available capacity |
|
||||
| Storage latency per candidate class | `deploy/kubernetes/inventory/disk-probe.yaml.example`, applied by hand | Commit latency and lease margins |
|
||||
| `aevolution-dev` (project `aevolution-dev`) | `deploy/kubernetes/dev` on `main` | **No automated sync.** Currently Synced/Healthy; deploys only on a manual sync |
|
||||
| `sor-server`, `sor-webclient` | other repositories | Automated, with prune and self-heal |
|
||||
| `aeternalis` | other repository | Manual |
|
||||
|
||||
The maintainer's statement that Argo deploys on every change applies to the
|
||||
automated applications, not to AEvolution today. The `deploy/README.md` guidance
|
||||
(manual full sync, no automated prune) matches the live application. Keep it that
|
||||
way for anything durable. Harbor publication on push is confirmed by the CI runs.
|
||||
|
||||
**`evolution-dev` contents.**
|
||||
- The completed `aevolution-dev-proof` Job (run 1399 digest) and its deny-all
|
||||
NetworkPolicy.
|
||||
- ServiceAccounts `aevolution-proof`, plus three created by the maintainer
|
||||
shortly before collection: `aevolution-migrator`, `aevolution-region` and
|
||||
`aevolution-backup`.
|
||||
- Secrets `harbor-pull`, `aevolution-db` (13 keys), and
|
||||
`evolution-postgres-migrator` and `evolution-postgres-runtime` (3 keys each,
|
||||
also new). Their values were not read. The separate migrator and runtime
|
||||
Secrets match the ADR-0060 role split.
|
||||
|
||||
**Existing data services.**
|
||||
- `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs
|
||||
harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3
|
||||
therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must
|
||||
first check what `aevolution-db` points at.
|
||||
- `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned.
|
||||
INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default.
|
||||
The region host's object store is a filesystem PVC (ADR-0061), so no object
|
||||
server is required yet.
|
||||
|
||||
**Collector identity.** The output was collected as `system:admin`
|
||||
(system:masters). The collector made only read requests. M8.2 should create a
|
||||
genuinely read-only inventory identity, so this is not repeated with
|
||||
cluster-admin.
|
||||
|
||||
## Measured workload profile (local fixture)
|
||||
|
||||
@@ -88,26 +166,52 @@ The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
|
||||
restore and verification buffers several times over. Larger profiles
|
||||
(populations, fields) must be re-measured before these values are reused.
|
||||
|
||||
## Proposed budget model (to be replaced by measured inputs)
|
||||
Reserve per 32 GB worker before placing any region:
|
||||
- **Operating system and Kubernetes.** Kubelet, system and eviction reserves, and
|
||||
the CNI/CSI agents: take the difference between capacity and allocatable from
|
||||
`node-resources.txt`, plus the measured daemon usage.
|
||||
- **Stores.** PostgreSQL memory (shared_buffers, work_mem × connections,
|
||||
maintenance) and the object store, on the node or nodes that host them, sized by
|
||||
M8.3 rather than guessed.
|
||||
- **Observability.** Metrics and log agents, and at most 8 KiB of exposition per
|
||||
region scrape (ADR-0066).
|
||||
- **Recovery headroom.** A successor host can start while its fenced predecessor
|
||||
still holds memory. Budget two region footprints per region. On the database
|
||||
and object store, keep headroom for audit and clone work (read-only, bounded by
|
||||
payload × chain length).
|
||||
## Budget (proposal from measured inputs, 2026-09-29)
|
||||
|
||||
Region hosts are not the memory driver at the current profile size (see the
|
||||
measured peak above). The first capacity limits are expected to be:
|
||||
- storage latency and fsync behaviour;
|
||||
**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the
|
||||
hypervisor and host. Adding memory to a VM is possible (maintainer), but it does
|
||||
not add a failure domain.
|
||||
|
||||
**Per worker** (31.3 GiB capacity, no kubelet reservation configured):
|
||||
|
||||
| Reserve | Proposal | Basis |
|
||||
|---|---|---|
|
||||
| Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services |
|
||||
| Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt |
|
||||
| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm |
|
||||
| AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 |
|
||||
| Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB |
|
||||
|
||||
**Proposed dev placement** (to confirm in M8.3/8.4):
|
||||
- Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB
|
||||
with 38 % used, and PVs there are pinned to it.
|
||||
- Place PostgreSQL and the region host's object PVC on **one** worker (currently
|
||||
the least-used, worker-2), using a node label and affinity.
|
||||
- local-path pins both volumes to that node anyway. Choosing the node explicitly
|
||||
keeps the failure domain visible, and no replica count is claimed.
|
||||
|
||||
Even with these reservations, the AEvolution dev footprint stays under 3 GiB on
|
||||
that worker.
|
||||
|
||||
The first capacity limits are expected to be:
|
||||
- storage latency and fsync behaviour (probe pending);
|
||||
- database connections (one per host plus the migrator);
|
||||
- the lease margins: the 2 s renewal deadline and 5 s commit deadline
|
||||
(ADR-0060) must stay well above p99 storage plus database latency.
|
||||
- the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99
|
||||
storage plus database latency.
|
||||
|
||||
Do not set per-node region counts until the probe results exist.
|
||||
Per-node region counts and CPU budgets are set only after the in-cluster probe
|
||||
and M8.4's in-cluster measurement.
|
||||
|
||||
## In-cluster storage probe
|
||||
|
||||
`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job
|
||||
per worker on `local-path`, pinned by hostname. It is outside every Argo path and
|
||||
applied by hand, with the maintainer's approval (2026-09-29). The commands are in
|
||||
the file header. Record each worker's line here and compare it with the local
|
||||
baseline above.
|
||||
|
||||
| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync |
|
||||
|---|---|---|---|
|
||||
| rke2-worker-1 | pending | pending | pending |
|
||||
| rke2-worker-2 | pending | pending | pending |
|
||||
| rke2-worker-3 | pending | pending | pending |
|
||||
|
||||
Reference in New Issue
Block a user