diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index df41bd1..9fecadf 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -601,3 +601,8 @@ M7.9b implemented and locally qualified (2026-09-29), under ADR-0068: - rule and format changes are refused explicitly; - measured recovery (19.6 s against a 60 s target) and batching latency. M7 is complete locally. Next: M8.1, the host inventory and budgets. + +M8.1 complete (2026-09-29): docs/infrastructure/INVENTORY.md records the cluster, +the separate PostgreSQL VM, the measured workload profile and a per-worker +`local-path` storage probe. Storage figures are provisional until the planned RAID +and storage change. Next: M8.2, environment isolation and secrets. diff --git a/docs/STATUS.md b/docs/STATUS.md index da8e641..ad7cf98 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -980,8 +980,17 @@ M8.1 cluster inventory collected (2026-09-29, docs/infrastructure/INVENTORY.md): - RKE2 v1.32.5: one control plane and three 32 GB workers on one hypervisor. - `local-path` is the only storage (node-pinned, reclaim Delete). - `aevolution-dev` syncs manually. -- No PostgreSQL 18.6 exists yet. -- A proposed budget keeps AEvolution dev under 3 GiB on one labeled worker. +- AEvolution's PostgreSQL is a separate VM outside Kubernetes (`aevolution-db`), + so it is not in the worker budget. +- A proposed budget keeps AEvolution dev well under 1 GiB on one labeled worker + (not worker-3, whose disk is 87 % full). -Remaining for M8.1: the per-worker storage probe, which is approved and ready to -apply. +M8.1 complete (2026-09-29) for the current disk layout. The per-worker +`local-path` probe measured: +- 1.9–2.5 ms for synchronous 4 KiB writes; +- 29–36 ms mean (73 ms max) per object publication; +- 110–255 MB/s sequential. +These are well inside the lease deadlines. Re-probe after the planned RAID and +storage change. Host memory was raised by about 500 GB. + +Next: M8.2, environment isolation and secrets. \ No newline at end of file diff --git a/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md b/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md index 65bc03e..949c4d9 100644 --- a/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md +++ b/docs/architecture/INFRASTRUCTURE_ARCHITECTURE.md @@ -1,8 +1,10 @@ # Infrastructure and Kubernetes topology Status: production-shaped proposal, not deployed workloads. Initial hardware is -one R730xd with approximately 128GB RAM. Kubernetes is available per brief; actual -CPU/storage/CNI/CSI/backup/network details remain uninspected. Replicas on one +one R730xd, with about 128GB RAM in the brief and about 500GB added since. The +cluster (RKE2, one control plane plus three 32GB workers, local-path storage) and a +separate PostgreSQL VM were inventoried in M8.1; see +[INVENTORY.md](../infrastructure/INVENTORY.md). Replicas on one physical host do not create an independent failure domain. ```mermaid @@ -62,7 +64,7 @@ unbounded replica creation. Pod disruption budgets help planned maintenance but cannot protect against chassis loss. Local volumes constrain scheduling; adding compute nodes does not automatically make stateful storage portable. -128GB is a capacity envelope, not all available simulation memory. Inventory OS/ +Host memory is a capacity envelope, not all available simulation memory. Inventory OS/ Kubernetes overhead, DB cache, object storage, observability, dev/stage and recovery headroom before region budgets. Start measured namespace quotas and leave explicit recovery/compaction headroom. No unverified entity/player-per-node promises. diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 2cd45b6..2f2f850 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2620,3 +2620,22 @@ Evidence: every Argo path and has **not yet been applied**, because this workstation has no kubeconfig. The maintainer approved running it. Its results go in INVENTORY.md. + +## M8.1c storage probe and corrections - 2026-09-29 + +- The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml` + by hand on the cluster, and all three Jobs completed. The resources were then + deleted. + - worker-1: 1906 µs dsync, 28.7/60.6 ms object publish, 1003 ms for 256 MiB. + - worker-2: 2511 µs, 36.0/73.4 ms, 2122 ms. + - worker-3: 2239 µs, 32.1/61.0 ms, 2339 ms. Its root filesystem is 87 % full. + Every `local-path` volume is on the node's root LVM ext4 volume. The figures are + provisional until the planned RAID change. +- Corrections from the maintainer: + - Host RAM was increased by about 500 GB, so the earlier "about 20 GiB left" + estimate is withdrawn. + - AEvolution's PostgreSQL is a separate VM outside Kubernetes, reached through + `aevolution-db`. + - Collection is manual, with no automation planned. +- No credential values were read. The 18.6 version, role and TLS requirements are + enforced by the adapter and migrations when M8.3 connects. diff --git a/docs/infrastructure/INVENTORY.md b/docs/infrastructure/INVENTORY.md index eb2efcd..85b28a6 100644 --- a/docs/infrastructure/INVENTORY.md +++ b/docs/infrastructure/INVENTORY.md @@ -1,10 +1,12 @@ # Host and cluster inventory (M8.1) -Status: **cluster inspected 2026-09-29; in-cluster storage probe pending.** +Status: **complete for the current layout, 2026-09-29** (cluster inspected and +storage probed on every worker). Sources, kept separate below: - maintainer statements; -- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran - (read requests only; Secrets listed by name, type and key count); +- the output of `tools/inventory/collect-cluster.sh`, which the maintainer ran by + hand (read requests only; Secrets listed by name, type and key count). No + automated collection is planned; - local measurements. The disks and RAID layout will change, so every storage number here describes @@ -14,37 +16,38 @@ the current layout only. | Fact | Source | Implication | |---|---|---| -| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. | +| One Dell R730xd. The founding brief said about 128 GB of RAM; the maintainer has since added about 500 GB | Founding brief; maintainer, 2026-09-29 | Host memory is no longer the binding limit, and more VM memory can be added. It is still one physical failure domain: memory adds capacity, not fault tolerance. The exact total is not recorded here. | | One hypervisor running three worker VMs (32 GB each) and the control plane; VM memory not overcommitted; more memory can be added | Maintainer, 2026-09-29; confirmed by node capacities | All four nodes share one physical failure domain. Adding memory does not add fault tolerance. | | Disks and RAID will be reorganized later | Maintainer, 2026-09-29 | Every storage measurement is provisional, and storage decisions (M8.3) must be re-verified afterwards. | | Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. | | Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | True for the automated applications (`sor-*`). The live `aevolution-dev` application has **no automated sync** (see Cluster). Anything added to its path is deployed by the next manual sync, which is why every M7/M8 example stays `.yaml.example` or outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git. | -| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. | +| The AEvolution PostgreSQL server is a separate VM outside Kubernetes; Secret `aevolution-db` in `evolution-dev` holds everything needed to connect | Maintainer, 2026-09-29 | See below. The Secret's values were not read. | -**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`, -`PGHOSTADDR`, `PGPASSWORD`, `PGPORT`, `PGSSLMODE`, `PGSSLROOTCERT`, `PGUSER` and -`POSTGRES_*`. That points to a single database login provisioned outside this -repository. Before M8.3 can use it, confirm: -- **Server version.** The adapter and migrations require exactly 18.6 (ADR-0059). - Any other version is refused, not tolerated. -- **Role split.** ADR-0060 requires a migrator/owner login plus a separate - NOLOGIN runtime group with its own member login. The migration audit refuses a - privileged or owner-identity runtime role. One login for both would fail that - audit by design. -- **Password delivery.** `PGPASSWORD` in the environment is not how the adapter - is used; it reads a mounted password file (`--pg-passfile`, POSTGRESQL.md). - Mount the Secret as files. -- **TLS.** `PGSSLMODE` and `PGSSLROOTCERT` suggest TLS with a private CA. Use - `verify-full` with the CA mounted. -- **Backup and failure domain.** Where does the database live? It is part of the - M8.3/M8.7 evidence. +**The AEvolution database.** PostgreSQL runs on its own VM, outside the cluster. +`aevolution-db` (13 keys: `PG*` and `POSTGRES_*`) holds its connection details. +Its values were not read. Consequences: +- **Stores.** The database is not a Kubernetes workload, so its memory and disk + are not part of the worker budget. Region hosts reach it across the VM network. + Their NetworkPolicy egress must allow exactly that address and port + (`PGHOSTADDR` / `PGPORT`) and nothing else. +- **Failure domain.** The database VM is a separate VM, but on the same single + hypervisor unless the maintainer says otherwise. It is separate from worker + failures, not from host loss. Backups must leave the host (M8.7). +- **Checks before use (M8.3).** These are enforced by the code, not by inventory: + - the adapter refuses any server other than 18.6 (ADR-0059); + - the migration audit refuses a privileged or owner-identity runtime role + (ADR-0060); + - the adapter reads a mounted password file, never `PGPASSWORD`; + - `PGSSLROOTCERT` indicates TLS with a CA, so connect with `verify-full`. + The separate `evolution-postgres-migrator` and `evolution-postgres-runtime` + Secrets (3 keys each) match the required role split. ## Cluster (collected 2026-09-29) -**Hypervisor and nodes.** One hypervisor on the R730xd runs four VMs, and the -maintainer states that memory is not overcommitted. Together the VMs hold about -108 GiB of the host's roughly 128 GB. Everything shares one physical failure -domain. +**Hypervisor and nodes.** One hypervisor on the R730xd runs the four cluster VMs +and the separate PostgreSQL VM. The maintainer states that memory is not +overcommitted and that the host now has about 500 GB more RAM than the brief's +128 GB. Everything shares one physical failure domain. The cluster runs RKE2 v1.32.5. It has no reserved resources, priority classes or quotas beyond the defaults: @@ -117,18 +120,16 @@ way for anything durable. Harbor publication on push is confirmed by the CI runs **Existing data services.** - `gitea-postgresql` runs 17.6 (bitnamilegacy) and `harbor-database` runs - harbor-db 2.13.1. Neither is 18.6, and both belong to other systems. M8.3 - therefore needs a dedicated, pinned PostgreSQL 18.6 for AEvolution, and must - first check what `aevolution-db` points at. + harbor-db 2.13.1. Both belong to other systems and are not used by AEvolution, + whose database is the separate VM above. - `cdn/minio` runs `quay.io/minio/minio:latest`, which is unpinned. INFRASTRUCTURE_ARCHITECTURE already rules out adopting MinIO as a new default. The region host's object store is a filesystem PVC (ADR-0061), so no object server is required yet. -**Collector identity.** The output was collected as `system:admin` -(system:masters). The collector made only read requests. M8.2 should create a -genuinely read-only inventory identity, so this is not repeated with -cluster-admin. +**Collector identity.** The maintainer ran the collector by hand as +`system:admin` (system:masters). It made only read requests. No automated +collection or dedicated collection identity is planned. ## Measured workload profile (local fixture) @@ -168,9 +169,9 @@ restore and verification buffers several times over. Larger profiles ## Budget (proposal from measured inputs, 2026-09-29) -**Host.** The four VMs take about 108 GiB, leaving about 20 GiB for the -hypervisor and host. Adding memory to a VM is possible (maintainer), but it does -not add a failure domain. +**Host.** With about 500 GB added, host memory is not a limit for dev and stage. +VM memory can be raised on request (maintainer), but that never adds a failure +domain. **Per worker** (31.3 GiB capacity, no kubelet reservation configured): @@ -178,40 +179,60 @@ not add a failure domain. |---|---|---| | Kubelet/system reservation | Configure about 1 GiB `system-reserved` plus 1 GiB `kube-reserved`, and a 500 Mi hard eviction threshold | Allocatable currently equals capacity, so pod usage can starve the node's own services | | Existing platform and other projects | Leave today's usage in place (2.5–6 GiB) plus its limits (4.3–8.9 GiB) | top-nodes.txt and nodes-describe.txt | -| AEvolution dev, steady | PostgreSQL 18.6: 1 GiB request, 2 GiB limit. Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process | Measured host footprint of about 9.7 MiB (above). The PostgreSQL size is a dev starting point for M8.3 to confirm | +| AEvolution dev, steady | Region host: 64 Mi request, 256 Mi limit. Filesystem object PVC: no process. PostgreSQL is **not** on the workers (separate VM) | Measured host footprint of about 9.7 MiB (above) | | AEvolution recovery headroom | A second region host (a successor while the fenced one exits), a migrate Job of about 128 Mi, and audit and clone buffers | ADR-0064/0065/0067 | | Observability (M8.5) | Up to about 0.5 GiB for scrape and log agents | ADR-0066 bounds each region scrape at 8 KiB | -**Proposed dev placement** (to confirm in M8.3/8.4): +**Proposed dev placement** (to confirm in M8.4): - Keep AEvolution's durable pods off the untainted control plane. It has 11.7 GiB with 38 % used, and PVs there are pinned to it. -- Place PostgreSQL and the region host's object PVC on **one** worker (currently - the least-used, worker-2), using a node label and affinity. -- local-path pins both volumes to that node anyway. Choosing the node explicitly - keeps the failure domain visible, and no replica count is claimed. +- **Never place them on rke2-worker-3.** Its root filesystem is 87 % full + (13 GB free). That is near the kubelet's default nodefs eviction threshold and + leaves no room for retained history. +- Place the region host's object PVC on **rke2-worker-1**, which had the lowest + measured latency, or **rke2-worker-2**, which has the most free space (60 GB). + Use a node label and affinity. local-path pins the volume to that node anyway; + choosing it explicitly keeps the failure domain visible, and no replica count + is claimed. -Even with these reservations, the AEvolution dev footprint stays under 3 GiB on -that worker. +On the chosen worker, AEvolution dev uses well under 1 GiB (the region host, its +successor headroom and the migrate Job). The database load stays on its VM. The first capacity limits are expected to be: -- storage latency and fsync behaviour (probe pending); +- object publication latency on the current disks (about 30–36 ms mean); - database connections (one per host plus the migrator); - the ADR-0060 margins: the 2 s renewal and 5 s commit deadlines against p99 storage plus database latency. -Per-node region counts and CPU budgets are set only after the in-cluster probe -and M8.4's in-cluster measurement. +Per-node region counts and CPU budgets are set only after M8.4's in-cluster +measurement, and re-checked after the disk and RAID rework. -## In-cluster storage probe +## In-cluster storage probe (2026-09-29, current disk layout) -`deploy/kubernetes/inventory/disk-probe-local-path.yaml` defines one PVC and Job -per worker on `local-path`, pinned by hostname. It is outside every Argo path and -applied by hand, with the maintainer's approval (2026-09-29). The commands are in -the file header. Record each worker's line here and compare it with the local -baseline above. +The maintainer applied `deploy/kubernetes/inventory/disk-probe-local-path.yaml` +by hand, then deleted it: one PVC and Job per worker on `local-path`. Every +worker's `local-path` sits on the node's root LVM volume (ext4, 97 GB). -| Worker | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync | -|---|---|---|---| -| rke2-worker-1 | pending | pending | pending | -| rke2-worker-2 | pending | pending | pending | -| rke2-worker-3 | pending | pending | pending | +| Worker | Root filesystem used | 4 KiB O_DSYNC mean | 40 KiB object publish mean / max | 256 MiB fdatasync | +|---|---|---|---|---| +| rke2-worker-1 | 54 % (43 GB free) | 1906 µs | 28.7 / 60.6 ms | 1003 ms (about 255 MB/s) | +| rke2-worker-2 | 35 % (60 GB free) | 2511 µs | 36.0 / 73.4 ms | 2122 ms (about 121 MB/s) | +| rke2-worker-3 | 87 % (13 GB free) | 2239 µs | 32.1 / 61.0 ms | 2339 ms (about 109 MB/s) | +| Workstation baseline (WSL2, NVMe) | 3 % | 1169 µs | 4.6 / 8.0 ms | 365 ms | + +Interpretation: +- **Synchronous 4 KiB writes** are 1.6–2.1× the workstation baseline, which is + unremarkable. +- **Object publication** (write, fsync, rename, directory fsync) is 6–8× slower, + at 29–36 ms mean and up to 73 ms. A batch publishes two objects (payload and + trace) before its database transaction. That adds roughly 60–150 ms per batch, + plus the network round trip to the database VM. +- **Lease margins hold.** That per-batch cost stays far inside the 2 s renewal + and 5 s commit deadlines (ADR-0060), and inside the 1 s flush interval. + Acknowledgement latency in the cluster should be a little above the local + 549 ms p50, and SIGKILL recovery remains lease-bound (about 20 s). M8.4 must + confirm both in the cluster. +- **Sequential throughput** is 110–255 MB/s. That is ample for 38 KB to 4.3 MB + payloads, and for audit and clone reads. +- **Provisional.** These numbers describe the current disks. Re-run the probe + after the RAID and storage change, before M8.3's storage decision is final.