Add M8.3 dev/stage stores: Retain volumes, free-space reserve, store check

Keep the filesystem object store on a node-local Retain-class PVC per
environment (ADR-0070), with the database on the external PostgreSQL VM.
Because local-path does not enforce sizes, the object store now refuses
new objects below a free-space reserve with a distinct full error, so a
full volume publishes and acknowledges nothing. evo-world-clone gains
check-store to qualify volume retention, reserve, database version/TLS
and the runtime privilege boundary from inside the cluster; it now ships
in the server image. Add the storage class, object claims, hand-applied
store Jobs, policy rules forbidding password env vars, and the runbook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 13:51:15 -04:00
co-authored by Claude Opus 5.5
parent 9d9e00c307
commit 2d950060db
23 changed files with 833 additions and 24 deletions
+47 -5
View File
@@ -21,12 +21,12 @@ manual, and none is performed by Argo CD.
Each namespace gets its own Secrets, with environment-specific values:
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
database.
database. Keys: `pg_service.conf`, `pgpass`, `ca.crt`.
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
NOLOGIN runtime group (ADR-0060).
- `aevolution-db`: connection settings (host, address, port, database, TLS mode
and root certificate). Mount it as files. The adapter reads a password file,
never `PGPASSWORD`.
NOLOGIN runtime group (ADR-0060). Keys: `user`, `pgpass`, `ca.crt`.
- `aevolution-db`: connection settings. Only `PGHOST`, `PGHOSTADDR` (numeric),
`PGPORT` and `PGDATABASE` are ever read, as individual variables. `PGPASSWORD`
and `PGUSER` are never mounted; the policy check forbids it.
A World0 credential must never exist in the dev or stage namespaces, and no dev or
stage credential may reach the World0 database. Rotate by replacing the Secret and
@@ -52,6 +52,48 @@ bash tools/ops/verify-isolation.sh
Create the Secrets listed above in each namespace before any workload syncs.
## Stores (M8.3, ADR-0070)
Run this once per disposable environment (`dev`, then `stage`).
**1. Label the store node, and apply the storage class and claim.**
```sh
# Label exactly one worker (worker-1 or worker-2; never worker-3 or the control plane).
kubectl label node rke2-worker-1 aevolution.io/stores=true
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
kubectl apply -k deploy/kubernetes/platform/dev # adds the aevolution-objects claim
```
**2. Prepare the store Jobs.** Copy `deploy/kubernetes/stores/stores.yaml.example`,
replace its `REPLACE_*` markers (namespace, database address and port, runtime
role, image digests), and create the Secrets above.
**3. Run and qualify.**
```sh
kubectl apply -f stores-dev.yaml # policy + both Jobs
kubectl logs -n evolution-dev job/aevolution-db-migrate # schema and audit
kubectl logs -n evolution-dev job/aevolution-store-check # expect exit 0 and "probe created"
```
Evidence to record in docs/engineering/VALIDATION.md:
- **Pod restart:** delete the check Job and apply it again. It must print
`probe verified`, with the same digest.
- **Missing volume:** run the check with the `objects` volume removed. It must exit
4 with `objects refused missing-or-not-a-directory`.
- **Full volume:** add `--reserve-mib 1048576` and use a new probe region, such as
`--region 2`. It must exit 4 with `probe refused full`.
- **Retention:** `kubectl delete pvc aevolution-objects -n evolution-dev`. The PV
must become `Released`, not deleted. To rebind:
- clear the PV's `spec.claimRef`;
- recreate the claim with `volumeName` set to that PV;
- rerun the check, which must print `probe verified`.
- **Credential rotation:**
- Change the runtime login's password on the database VM, and run the check
before updating the Secret. It must exit 3 with `database refused connect`.
- Update the Secret's `pgpass` and rerun. It must exit 0.
- **Privilege boundary:** every run must print
`database tables=denied functions=reachable`.
## Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,