Add M8.3 dev/stage stores: Retain volumes, free-space reserve, store check

Keep the filesystem object store on a node-local Retain-class PVC per
environment (ADR-0070), with the database on the external PostgreSQL VM.
Because local-path does not enforce sizes, the object store now refuses
new objects below a free-space reserve with a distinct full error, so a
full volume publishes and acknowledges nothing. evo-world-clone gains
check-store to qualify volume retention, reserve, database version/TLS
and the runtime privilege boundary from inside the cluster; it now ships
in the server image. Add the storage class, object claims, hand-applied
store Jobs, policy rules forbidding password env vars, and the runbook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 13:51:15 -04:00
co-authored by Claude Opus 5.5
parent 9d9e00c307
commit 2d950060db
23 changed files with 833 additions and 24 deletions
+1
View File
@@ -7,6 +7,7 @@ names it. Remove an item only when it is done, and record where it was done.
| Item | Raised | Why deferred | Needed before |
|---|---|---|---|
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
| **Backup destination for dev/stage stores.** Decide where backups go: the AEvolution PostgreSQL VM (WAL archive or base backups) and each environment's `aevolution-objects` volume. It should be off the host for any claim beyond disposable environments. | M8.3, 2026-09-29 | Needs the maintainer's choice of target | M8.7 backup and restore; any World0 data |
| **Storage after the RAID/disk rework.** Re-run `deploy/kubernetes/inventory/disk-probe-local-path.yaml` (or a new class), and update INVENTORY.md. | M8.1, 2026-09-29 | Maintainer is reorganizing disks | M8.3's final storage decision |
## Accepted risks (maintainer decisions)
+8
View File
@@ -616,3 +616,11 @@ M8.2 designed and checked offline (2026-09-29), under ADR-0069:
- an offline policy check in CI, and a live isolation check.
The live checks run once the maintainer applies the platform layer. Next: M8.3,
durable stores in dev and stage.
M8.3 implemented and locally qualified (2026-09-29), under ADR-0070:
- each environment's database lives on the external PostgreSQL VM;
- object volumes are node-local and use the Retain class;
- a free-space reserve with a distinct full-volume refusal;
- `evo-world-clone check-store` for in-cluster qualification.
The in-cluster evidence and the backup destination wait on the maintainer
(docs/BACKLOG.md).
+21
View File
@@ -1015,3 +1015,24 @@ M8.2 applied and verified live (2026-09-29): 175 PASS and 2 SKIP. The only 3 FAI
were a script defect in the project check, now fixed. The maintainer accepted
leaving Argo's `default` project unrestricted. Break-glass membership is in the
new docs/BACKLOG.md. Next: M8.3, durable stores in dev and stage.
## M8.3 dev/stage stores (2026-09-29)
ADR-0070 sets up the dev and stage stores:
- **Database:** each environment gets its own database and logins on the external
PostgreSQL VM.
- **Object store:** the filesystem store stays, on a node-local PVC in the new
Retain class `aevolution-local-retain`, pinned to one labelled worker. There is
no object server.
- **Capacity:** node-local volumes ignore their requested size, so the object
store now enforces a free-space reserve (`--objects-reserve-mib`, default 1024).
A full volume refuses new objects with a distinct error, and nothing is
acknowledged.
- **Qualification:** `evo-world-clone check-store`, now in the server image,
checks from inside the cluster: volume presence and retention, the reserve,
database version and TLS, and the runtime privilege boundary.
Not yet done:
- the in-cluster run (it needs Secrets with the documented key names and
published image digests);
- choosing a backup destination (docs/BACKLOG.md).
+102
View File
@@ -0,0 +1,102 @@
# ADR-0070: Durable stores for dev and stage
## Status
Accepted for M8.3, 2026-09-29. It builds on ADR-0059 (PostgreSQL), ADR-0061
(object store), ADR-0069 (environments) and the M8.1 inventory. Storage decisions
are provisional until the planned RAID and disk rework (docs/BACKLOG.md).
## Context
Inventory facts that drive this decision:
- PostgreSQL runs on a separate VM outside Kubernetes (maintainer). The adapter
already requires exactly 18.6 and `verify-full` TLS off-host, and the migration
audits the runtime role.
- The cluster's only StorageClass, `local-path`, is node-local, reclaims with
Delete and does not enforce requested sizes.
- Object publication on the workers measured 29–36 ms mean (73 ms max). Worker-3's
root filesystem is 87 % full.
- MinIO in the cluster is unpinned (`latest`) and belongs to another project.
The roadmap asks for:
- a supported PostgreSQL and a maintained object store, with pinned versions;
- PV retention and a backup destination;
- evidence for pod restart, missing-volume refusal, object semantics, credential
rotation and full disk.
The named risk is that a local PV binds failover to a node that has been lost.
## Decision
**Database.** Use the maintainer's PostgreSQL VM. Each environment gets its own
database, and its own migrator login and NOLOGIN runtime group with a member login
(ADR-0060). The version pin (18.6), TLS verification and privilege audit are
enforced by code at connect and migrate time; they are not assumed. Migrations run
as a hand-applied Job using the pinned `postgres-tools` image.
**Object store.** Keep the ADR-0061 filesystem store. There is **no object
server**: MinIO is ruled out as a new default (INFRASTRUCTURE_ARCHITECTURE), and an
S3 adapter would be a new dependency. Instead:
- The store lives on one PVC per environment, `aevolution-objects`, in the new
class `aevolution-local-retain` (the same provisioner, with **Retain** and
WaitForFirstConsumer; never the default class).
- The PVC is platform-owned (hand-applied, never Argo).
- Every consumer selects the node label `aevolution.io/stores=true`, which is put
on exactly one worker. worker-1 or worker-2 is recommended; never worker-3 or
the control plane.
**Capacity guard.** `local-path` ignores the requested size, so
`FilesystemObjectStore` now takes a free-space reserve. `evo-server` exposes it as
`--objects-reserve-mib`, default 1024.
- A new object that would leave less free space is refused with a distinct
`ObjectError::full`, before anything is written.
- Identical objects that already exist stay reusable.
- The host treats a refused upload as a retryable store failure, so nothing is
published or acknowledged, and a drain that cannot publish ends without
committing anything (exit 3).
- A genuinely failing write (ENOSPC or EFBIG) is `io`, and leaves no temporary
file or object behind.
**Qualification tool.** `evo-world-clone check-store` runs in the cluster as a Job
with the runtime identity. It:
- refuses a missing or unmounted object root;
- creates a fixed 64 KiB probe object once, then verifies it by digest on every
later run (proof of retention across pods);
- reports free space against the reserve;
- reports the server version, user and TLS;
- requires that the runtime login reaches the fenced functions but is denied
table access.
The server image now includes the tool.
**Secret contract.**
- `aevolution-db` supplies only `PGHOST`, `PGHOSTADDR`, `PGPORT` and `PGDATABASE`.
- `evolution-postgres-runtime` supplies `user`, `pgpass` and `ca.crt`.
- `evolution-postgres-migrator` supplies `pg_service.conf`, `pgpass` and `ca.crt`.
Passwords are only ever mounted as files, and copied into private memory
(mode 0600) before use, because libpq ignores group-readable password files. The
policy check forbids `PGPASSWORD`/`PGUSER` references and whole-Secret `envFrom`.
**Retention and backup.**
- Deleting the claim leaves the PV Released with its data. Recovery is an explicit
rebind by an administrator, verified by `check-store` ("probe verified").
- The backup destination for the database VM and the object PVC is **not decided
yet**. It is a maintainer decision, recorded in docs/BACKLOG.md and required
before M8.7. Dev and stage run without a verified backup, which is acceptable
only because they are disposable. World0 remains blocked.
## Consequences
- A lost worker takes its object volume with it. The dev and stage regions on that
worker are then unavailable (the server refuses a missing volume and quarantines
missing objects); they are never silently re-initialized. This is the named risk,
accepted for dev and stage only. World0 needs replicated or off-node storage
(M8.7/M8.8).
- The reserve and the kubelet's nodefs eviction threshold interact. Set the reserve
above the node's remaining headroom target, and alert on free space (M8.5).
- Re-probe and re-decide after the disk rework.
## Alternatives rejected
- **MinIO or another object server.** It adds an unpinned or new dependency, a
second durability layer to back up, and network latency, for no gain on one
node.
- **Leaving `local-path`/Delete for durable data.** Deleting the claim or an Argo
prune would erase the store.
- **Trusting PVC sizes.** `local-path` does not enforce them.
+1
View File
@@ -80,3 +80,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0067: Region clones and retained-history audit](0067-region-clones.md) - accepted for M7.9a re-identified clone genesis naming its source cut, isolated clone databases and full-chain retention audit.
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
- [ADR-0069: Environment isolation and World0 protection](0069-environment-isolation.md) - accepted for M8.2 per-environment guardrails, native World0 admission protection, scoped Argo projects and verification.
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
+39
View File
@@ -2689,3 +2689,42 @@ admission policy and the stage and World0 AppProjects, then ran
`[]`. The corrected script has not been re-run on the cluster.
- **WARN:** the `default` AppProject allows every namespace. The maintainer
accepted this (docs/BACKLOG.md).
## M8.3 dev/stage stores - 2026-09-29
- WSL Linux `checks` set: CTest 59/59 in each configuration. The format check
passed.
- PostgreSQL suites: Release and ASan/UBSan passed with the new
`region_store_tests`. Analysis passed after one clang-tidy fix (an int
multiplication widened in the MiB report).
- Store tests:
- An unreachable reserve refuses new objects with `full`, leaving no files. An
identical existing object is still reused.
- A lineage batch through a full store fails as `upload`/`full`, and the head is
unchanged.
- RLIMIT_FSIZE makes a real write fail with `io`, leaving no object or
temporary.
- `check-store`:
- its first run creates the probe and its second verifies it;
- a missing root exits 4 without creating it;
- a full volume exits 4;
- an owner login exits 3 (tables READABLE);
- the runtime login reports `version=180006 ... ssl=false` (local socket), and
`tables=denied functions=reachable`.
- An `evo-server` whose reserve cannot be met simulated but acknowledged
nothing. Its drain exited 3, and the head was unchanged.
- Mutations (restored and rebuilt), each caught:
- removing the reserve check;
- `check-store` always succeeding.
- `tools/ci/deploy-policy-check.py`: 41 objects, 0 failures. Four new-rule
mutations were caught:
- a Delete class;
- a platform claim on `local-path`;
- a `PGPASSWORD` key reference;
- a whole-Secret `envFrom`.
- Kustomize renders the dev and stage overlays with the `aevolution-objects`
claim; World0 has none.
- The server image now contains `evo-world-clone`, and its smoke script checks the
tool's usage exit. It has not been built locally, because Docker is not running.
- Not executed: anything in the cluster, the store Jobs, retention, rebind and
rotation evidence, and the image build.
+47 -5
View File
@@ -21,12 +21,12 @@ manual, and none is performed by Argo CD.
Each namespace gets its own Secrets, with environment-specific values:
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
database.
database. Keys: `pg_service.conf`, `pgpass`, `ca.crt`.
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
NOLOGIN runtime group (ADR-0060).
- `aevolution-db`: connection settings (host, address, port, database, TLS mode
and root certificate). Mount it as files. The adapter reads a password file,
never `PGPASSWORD`.
NOLOGIN runtime group (ADR-0060). Keys: `user`, `pgpass`, `ca.crt`.
- `aevolution-db`: connection settings. Only `PGHOST`, `PGHOSTADDR` (numeric),
`PGPORT` and `PGDATABASE` are ever read, as individual variables. `PGPASSWORD`
and `PGUSER` are never mounted; the policy check forbids it.
A World0 credential must never exist in the dev or stage namespaces, and no dev or
stage credential may reach the World0 database. Rotate by replacing the Secret and
@@ -52,6 +52,48 @@ bash tools/ops/verify-isolation.sh
Create the Secrets listed above in each namespace before any workload syncs.
## Stores (M8.3, ADR-0070)
Run this once per disposable environment (`dev`, then `stage`).
**1. Label the store node, and apply the storage class and claim.**
```sh
# Label exactly one worker (worker-1 or worker-2; never worker-3 or the control plane).
kubectl label node rke2-worker-1 aevolution.io/stores=true
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
kubectl apply -k deploy/kubernetes/platform/dev # adds the aevolution-objects claim
```
**2. Prepare the store Jobs.** Copy `deploy/kubernetes/stores/stores.yaml.example`,
replace its `REPLACE_*` markers (namespace, database address and port, runtime
role, image digests), and create the Secrets above.
**3. Run and qualify.**
```sh
kubectl apply -f stores-dev.yaml # policy + both Jobs
kubectl logs -n evolution-dev job/aevolution-db-migrate # schema and audit
kubectl logs -n evolution-dev job/aevolution-store-check # expect exit 0 and "probe created"
```
Evidence to record in docs/engineering/VALIDATION.md:
- **Pod restart:** delete the check Job and apply it again. It must print
`probe verified`, with the same digest.
- **Missing volume:** run the check with the `objects` volume removed. It must exit
4 with `objects refused missing-or-not-a-directory`.
- **Full volume:** add `--reserve-mib 1048576` and use a new probe region, such as
`--region 2`. It must exit 4 with `probe refused full`.
- **Retention:** `kubectl delete pvc aevolution-objects -n evolution-dev`. The PV
must become `Released`, not deleted. To rebind:
- clear the PV's `spec.claimRef`;
- recreate the claim with `volumeName` set to that PV;
- rerun the check, which must print `probe verified`.
- **Credential rotation:**
- Change the runtime login's password on the database VM, and run the check
before updating the Secret. It must exit 3 with `database refused connect`.
- Update the Secret's `pgpass` and rerun. It must exit 0.
- **Privilege boundary:** every run must print
`database tables=denied functions=reachable`.
## Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,