Start M8.1 inventory: stated facts, workload profile and collectors
Record the maintainer-stated host facts (R730xd ~128 GB, 32 GB workers, automatic Harbor publish and Argo CD deploy, existing aevolution-db Secret and its conflicts with the pinned version and role split), the measured evo-server profile (peak RSS ~9.7 MiB, ~0.3% of a core) and a local storage baseline. Add a read-only cluster collector that never fetches Secret data and a by-hand storage latency probe outside the Argo path. Cluster facts remain uninspected pending read-only access. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -959,3 +959,19 @@ Review fixes (2026-09-29): schema v9 serializes build retirement with acquisitio
|
||||
leases and provisioning under one lock order. It also serializes clone-world
|
||||
classification per world. The operations listener's accept work per loop pass is
|
||||
now bounded. Regression tests and five mutations cover each fix.
|
||||
|
||||
## M8.1 inventory (in progress, 2026-09-29)
|
||||
|
||||
docs/infrastructure/INVENTORY.md records:
|
||||
- the maintainer's facts: an R730xd with about 128 GB, 32 GB worker nodes,
|
||||
automatic publish to Harbor and automatic Argo CD deployment, and the existing
|
||||
`aevolution-db` Secret. The Secret's single login conflicts with the required
|
||||
18.6 pin and with the separate migrator/runtime roles;
|
||||
- the measured workload profile: `evo-server` peaks at about 9.7 MiB and 0.3 % of
|
||||
one core;
|
||||
- a local storage baseline, and a proposed budget model.
|
||||
|
||||
Still needed:
|
||||
- run the read-only `tools/inventory/collect-cluster.sh` with cluster access;
|
||||
- hypervisor, VM and disk details from the maintainer;
|
||||
- the storage probe on each candidate StorageClass, after approval.
|
||||
|
||||
@@ -2580,3 +2580,23 @@ Evidence:
|
||||
- an accept budget of 1000.
|
||||
- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s.
|
||||
- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files.
|
||||
|
||||
## M8.1a inventory preparation - 2026-09-29
|
||||
|
||||
- No cluster access was available: there is no kubeconfig on the workstation.
|
||||
Nothing in the cluster was read or changed, and the `aevolution-db` Secret was
|
||||
not accessed. Only the key names the maintainer supplied are recorded.
|
||||
- `tools/inventory/collect-cluster.sh` passed `bash -n`. It has not been run
|
||||
against a cluster. It only reads, and it lists Secrets through the server-side
|
||||
table, so no data is fetched.
|
||||
- `deploy/kubernetes/inventory/disk-probe.yaml.example` parses as a PVC,
|
||||
NetworkPolicy and Job. Its script was run locally in WSL2 on ext4 as a
|
||||
baseline:
|
||||
- 4 KiB O_DSYNC writes: mean 1169 µs;
|
||||
- 40 KiB object publication: mean 4646 µs, max 7983 µs;
|
||||
- 256 MiB fdatasync: 365 ms.
|
||||
It has not been applied in any cluster.
|
||||
- The qualification now records the server's resource profile from /proc. Over
|
||||
three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of
|
||||
serving. The full PostgreSQL Release suite passed with it. The analysis build
|
||||
compiled clean, and the format check passed.
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
# Host and cluster inventory (M8.1)
|
||||
|
||||
Status: **partial, 2026-09-29.** This file separates what has been measured, what
|
||||
the maintainer has stated, and what has not been inspected yet. Nothing in the
|
||||
"Not inspected" table may be assumed. Budgets below are proposals until the
|
||||
cluster facts and the storage probe replace their inputs.
|
||||
|
||||
## Maintainer-stated facts
|
||||
|
||||
| Fact | Source | Implication |
|
||||
|---|---|---|
|
||||
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
|
||||
| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. |
|
||||
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
|
||||
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. |
|
||||
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
|
||||
|
||||
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
|
||||
`PGHOSTADDR`, `PGPASSWORD`, `PGPORT`, `PGSSLMODE`, `PGSSLROOTCERT`, `PGUSER` and
|
||||
`POSTGRES_*`. That points to a single database login provisioned outside this
|
||||
repository. Before M8.3 can use it, confirm:
|
||||
- **Server version.** The adapter and migrations require exactly 18.6 (ADR-0059).
|
||||
Any other version is refused, not tolerated.
|
||||
- **Role split.** ADR-0060 requires a migrator/owner login plus a separate
|
||||
NOLOGIN runtime group with its own member login. The migration audit refuses a
|
||||
privileged or owner-identity runtime role. One login for both would fail that
|
||||
audit by design.
|
||||
- **Password delivery.** `PGPASSWORD` in the environment is not how the adapter
|
||||
is used; it reads a mounted password file (`--pg-passfile`, POSTGRESQL.md).
|
||||
Mount the Secret as files.
|
||||
- **TLS.** `PGSSLMODE` and `PGSSLROOTCERT` suggest TLS with a private CA. Use
|
||||
`verify-full` with the CA mounted.
|
||||
- **Backup and failure domain.** Where does the database live? It is part of the
|
||||
M8.3/M8.7 evidence.
|
||||
|
||||
## Not inspected (needs read-only cluster access)
|
||||
|
||||
This workstation has no kubeconfig. Run
|
||||
`tools/inventory/collect-cluster.sh <dir>` with a read-only (get/list) identity.
|
||||
It never fetches Secret or ConfigMap data. Then record the results here.
|
||||
|
||||
| Item | Collected by | Needed for |
|
||||
|---|---|---|
|
||||
| Node count and roles, CPU, allocatable memory, kernel, runtime, taints | nodes.txt, node-resources.txt, nodes-describe.txt | Budgets, affinity plan (8.9) |
|
||||
| Hypervisor, VM layout, host disks and RAID, overcommit | Maintainer (outside Kubernetes) | Failure domains, the 128 GB envelope |
|
||||
| CNI and NetworkPolicy enforcement | daemonsets.txt, kube-system-pods.txt | The deny-by-default flows the region example relies on |
|
||||
| CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs | storageclasses.*, pv.txt, csidrivers.txt | 8.3. A local PV binds a region to one node (risk named in the roadmap) |
|
||||
| Volume snapshots | volumesnapshotclasses.txt | Backup design (8.7) |
|
||||
| Ingress, load balancers, DNS, certificate issuers | ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt | Metrics and probe exposure, later gateway |
|
||||
| Quotas, limit ranges, priority classes, PDBs | quotas.txt, priorityclasses.txt, pdbs.txt | 8.2 isolation |
|
||||
| Argo CD applications, auto-sync/prune, projects | argocd-apps.txt, argocd-projects.yaml | 8.2 World0 prune protection |
|
||||
| Current usage | top-nodes.txt, top-pods.txt (needs metrics-server) | Available capacity |
|
||||
| Storage latency per candidate class | `deploy/kubernetes/inventory/disk-probe.yaml.example`, applied by hand | Commit latency and lease margins |
|
||||
|
||||
## Measured workload profile (local fixture)
|
||||
|
||||
The measurements come from `durability_qualification`
|
||||
(engineering/DURABILITY_QUALIFICATION.md): one lineage region under the
|
||||
production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.
|
||||
|
||||
| Measure (Release, three successive owners) | Value |
|
||||
|---|---|
|
||||
| `evo-server` peak resident memory (VmHWM) | 9,940 KiB (about 9.7 MiB) |
|
||||
| CPU while serving, including start and recovery | 60 ms over 16.3 s of wall time, about 0.3 % of one core |
|
||||
| Committed payload | 37,756 bytes |
|
||||
| Head verification p50 / max | 1.6 / 12 ms |
|
||||
| Acknowledgement latency p50 / p99 (1 s flush) | 549 / 1048 ms |
|
||||
| SIGKILL to successor ready | 19.6 s (lease-bound) |
|
||||
|
||||
Local storage baseline: the same script as the disk probe, run in WSL2 on ext4
|
||||
over the workstation's NVMe (VHD):
|
||||
|
||||
| Pattern | Result |
|
||||
|---|---|
|
||||
| 4 KiB O_DSYNC writes | mean 1169 µs |
|
||||
| 40 KiB object publication (write, fsync, rename, directory fsync) | mean 4646 µs, max 7983 µs |
|
||||
| 256 MiB sequential with fdatasync | 365 ms |
|
||||
|
||||
Cluster storage should be compared against this baseline. A class whose
|
||||
synchronous-write p99 approaches the 2 s renewal deadline cannot host regions
|
||||
under the current lease policy.
|
||||
|
||||
A region host is small at this profile. Its requests are set by headroom, not by
|
||||
the simulation itself. **Proposed** starting values, to be confirmed by in-cluster
|
||||
measurement in M8.4: `requests: {cpu: 50m, memory: 64Mi}` and
|
||||
`limits: {memory: 256Mi}` with no CPU limit (throttling would delay renewals).
|
||||
The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
|
||||
restore and verification buffers several times over. Larger profiles
|
||||
(populations, fields) must be re-measured before these values are reused.
|
||||
|
||||
## Proposed budget model (to be replaced by measured inputs)
|
||||
Reserve per 32 GB worker before placing any region:
|
||||
- **Operating system and Kubernetes.** Kubelet, system and eviction reserves, and
|
||||
the CNI/CSI agents: take the difference between capacity and allocatable from
|
||||
`node-resources.txt`, plus the measured daemon usage.
|
||||
- **Stores.** PostgreSQL memory (shared_buffers, work_mem × connections,
|
||||
maintenance) and the object store, on the node or nodes that host them, sized by
|
||||
M8.3 rather than guessed.
|
||||
- **Observability.** Metrics and log agents, and at most 8 KiB of exposition per
|
||||
region scrape (ADR-0066).
|
||||
- **Recovery headroom.** A successor host can start while its fenced predecessor
|
||||
still holds memory. Budget two region footprints per region. On the database
|
||||
and object store, keep headroom for audit and clone work (read-only, bounded by
|
||||
payload × chain length).
|
||||
|
||||
Region hosts are not the memory driver at the current profile size (see the
|
||||
measured peak above). The first capacity limits are expected to be:
|
||||
- storage latency and fsync behaviour;
|
||||
- database connections (one per host plus the migrator);
|
||||
- the lease margins: the 2 s renewal deadline and 5 s commit deadline
|
||||
(ADR-0060) must stay well above p99 storage plus database latency.
|
||||
|
||||
Do not set per-node region counts until the probe results exist.
|
||||
Reference in New Issue
Block a user