Files
aevolution/docs/infrastructure/INVENTORY.md
T
dingoandClaude Opus 5.5 680e7e5725 Start M8.1 inventory: stated facts, workload profile and collectors
Record the maintainer-stated host facts (R730xd ~128 GB, 32 GB workers,
automatic Harbor publish and Argo CD deploy, existing aevolution-db
Secret and its conflicts with the pinned version and role split), the
measured evo-server profile (peak RSS ~9.7 MiB, ~0.3% of a core) and a
local storage baseline. Add a read-only cluster collector that never
fetches Secret data and a by-hand storage latency probe outside the
Argo path. Cluster facts remain uninspected pending read-only access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 11:57:08 -04:00

7.3 KiB
Raw Blame History

Host and cluster inventory (M8.1)

Status: partial, 2026-09-29. This file separates what has been measured, what the maintainer has stated, and what has not been inspected yet. Nothing in the "Not inspected" table may be assumed. Budgets below are proposals until the cluster facts and the storage probe replace their inputs.

Maintainer-stated facts

Fact Source Implication
One Dell R730xd, about 128 GB RAM Founding brief The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory.
Each Kubernetes worker node has 32 GB RAM Maintainer, 2026-09-29 Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain.
Every Git push builds and publishes to Harbor Maintainer, 2026-09-29; Gitea runs 1399/1402 Publication is automatic. It is not promotion: nothing reaches a cluster except through Git.
Argo CD pulls Git and images and deploys to Kubernetes on each change Maintainer, 2026-09-29 Deployment is automatic. deploy/README.md still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see argocd-apps.txt). Anything added to a watched path is live on push. That is why every M7/M8 example stays .yaml.example outside deploy/kubernetes/dev. The dev Job image is pinned by digest in Git, so a push alone does not change the running image.
Secret aevolution-db exists in evolution-dev Maintainer review note, 2026-09-29 See below. Its values were not read.

The existing database secret. It has these keys: PGDATABASE, PGHOST, PGHOSTADDR, PGPASSWORD, PGPORT, PGSSLMODE, PGSSLROOTCERT, PGUSER and POSTGRES_*. That points to a single database login provisioned outside this repository. Before M8.3 can use it, confirm:

  • Server version. The adapter and migrations require exactly 18.6 (ADR-0059). Any other version is refused, not tolerated.
  • Role split. ADR-0060 requires a migrator/owner login plus a separate NOLOGIN runtime group with its own member login. The migration audit refuses a privileged or owner-identity runtime role. One login for both would fail that audit by design.
  • Password delivery. PGPASSWORD in the environment is not how the adapter is used; it reads a mounted password file (--pg-passfile, POSTGRESQL.md). Mount the Secret as files.
  • TLS. PGSSLMODE and PGSSLROOTCERT suggest TLS with a private CA. Use verify-full with the CA mounted.
  • Backup and failure domain. Where does the database live? It is part of the M8.3/M8.7 evidence.

Not inspected (needs read-only cluster access)

This workstation has no kubeconfig. Run tools/inventory/collect-cluster.sh <dir> with a read-only (get/list) identity. It never fetches Secret or ConfigMap data. Then record the results here.

Item Collected by Needed for
Node count and roles, CPU, allocatable memory, kernel, runtime, taints nodes.txt, node-resources.txt, nodes-describe.txt Budgets, affinity plan (8.9)
Hypervisor, VM layout, host disks and RAID, overcommit Maintainer (outside Kubernetes) Failure domains, the 128 GB envelope
CNI and NetworkPolicy enforcement daemonsets.txt, kube-system-pods.txt The deny-by-default flows the region example relies on
CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs storageclasses.*, pv.txt, csidrivers.txt 8.3. A local PV binds a region to one node (risk named in the roadmap)
Volume snapshots volumesnapshotclasses.txt Backup design (8.7)
Ingress, load balancers, DNS, certificate issuers ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt Metrics and probe exposure, later gateway
Quotas, limit ranges, priority classes, PDBs quotas.txt, priorityclasses.txt, pdbs.txt 8.2 isolation
Argo CD applications, auto-sync/prune, projects argocd-apps.txt, argocd-projects.yaml 8.2 World0 prune protection
Current usage top-nodes.txt, top-pods.txt (needs metrics-server) Available capacity
Storage latency per candidate class deploy/kubernetes/inventory/disk-probe.yaml.example, applied by hand Commit latency and lease margins

Measured workload profile (local fixture)

The measurements come from durability_qualification (engineering/DURABILITY_QUALIFICATION.md): one lineage region under the production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.

Measure (Release, three successive owners) Value
evo-server peak resident memory (VmHWM) 9,940 KiB (about 9.7 MiB)
CPU while serving, including start and recovery 60 ms over 16.3 s of wall time, about 0.3 % of one core
Committed payload 37,756 bytes
Head verification p50 / max 1.6 / 12 ms
Acknowledgement latency p50 / p99 (1 s flush) 549 / 1048 ms
SIGKILL to successor ready 19.6 s (lease-bound)

Local storage baseline: the same script as the disk probe, run in WSL2 on ext4 over the workstation's NVMe (VHD):

Pattern Result
4 KiB O_DSYNC writes mean 1169 µs
40 KiB object publication (write, fsync, rename, directory fsync) mean 4646 µs, max 7983 µs
256 MiB sequential with fdatasync 365 ms

Cluster storage should be compared against this baseline. A class whose synchronous-write p99 approaches the 2 s renewal deadline cannot host regions under the current lease policy.

A region host is small at this profile. Its requests are set by headroom, not by the simulation itself. Proposed starting values, to be confirmed by in-cluster measurement in M8.4: requests: {cpu: 50m, memory: 64Mi} and limits: {memory: 256Mi} with no CPU limit (throttling would delay renewals). The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and restore and verification buffers several times over. Larger profiles (populations, fields) must be re-measured before these values are reused.

Proposed budget model (to be replaced by measured inputs)

Reserve per 32 GB worker before placing any region:

  • Operating system and Kubernetes. Kubelet, system and eviction reserves, and the CNI/CSI agents: take the difference between capacity and allocatable from node-resources.txt, plus the measured daemon usage.
  • Stores. PostgreSQL memory (shared_buffers, work_mem × connections, maintenance) and the object store, on the node or nodes that host them, sized by M8.3 rather than guessed.
  • Observability. Metrics and log agents, and at most 8 KiB of exposition per region scrape (ADR-0066).
  • Recovery headroom. A successor host can start while its fenced predecessor still holds memory. Budget two region footprints per region. On the database and object store, keep headroom for audit and clone work (read-only, bounded by payload × chain length).

Region hosts are not the memory driver at the current profile size (see the measured peak above). The first capacity limits are expected to be:

  • storage latency and fsync behaviour;
  • database connections (one per host plus the migrator);
  • the lease margins: the 2 s renewal deadline and 5 s commit deadline (ADR-0060) must stay well above p99 storage plus database latency.

Do not set per-node region counts until the probe results exist.