Add M8.3 dev/stage stores: Retain volumes, free-space reserve, store check
Keep the filesystem object store on a node-local Retain-class PVC per environment (ADR-0070), with the database on the external PostgreSQL VM. Because local-path does not enforce sizes, the object store now refuses new objects below a free-space reserve with a distinct full error, so a full volume publishes and acknowledges nothing. evo-world-clone gains check-store to qualify volume retention, reserve, database version/TLS and the runtime privilege boundary from inside the cluster; it now ships in the server image. Add the storage class, object claims, hand-applied store Jobs, policy rules forbidding password env vars, and the runbook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -7,6 +7,7 @@ names it. Remove an item only when it is done, and record where it was done.
|
||||
| Item | Raised | Why deferred | Needed before |
|
||||
|---|---|---|---|
|
||||
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
|
||||
| **Backup destination for dev/stage stores.** Decide where backups go: the AEvolution PostgreSQL VM (WAL archive or base backups) and each environment's `aevolution-objects` volume. It should be off the host for any claim beyond disposable environments. | M8.3, 2026-09-29 | Needs the maintainer's choice of target | M8.7 backup and restore; any World0 data |
|
||||
| **Storage after the RAID/disk rework.** Re-run `deploy/kubernetes/inventory/disk-probe-local-path.yaml` (or a new class), and update INVENTORY.md. | M8.1, 2026-09-29 | Maintainer is reorganizing disks | M8.3's final storage decision |
|
||||
|
||||
## Accepted risks (maintainer decisions)
|
||||
|
||||
@@ -616,3 +616,11 @@ M8.2 designed and checked offline (2026-09-29), under ADR-0069:
|
||||
- an offline policy check in CI, and a live isolation check.
|
||||
The live checks run once the maintainer applies the platform layer. Next: M8.3,
|
||||
durable stores in dev and stage.
|
||||
|
||||
M8.3 implemented and locally qualified (2026-09-29), under ADR-0070:
|
||||
- each environment's database lives on the external PostgreSQL VM;
|
||||
- object volumes are node-local and use the Retain class;
|
||||
- a free-space reserve with a distinct full-volume refusal;
|
||||
- `evo-world-clone check-store` for in-cluster qualification.
|
||||
The in-cluster evidence and the backup destination wait on the maintainer
|
||||
(docs/BACKLOG.md).
|
||||
|
||||
@@ -1015,3 +1015,24 @@ M8.2 applied and verified live (2026-09-29): 175 PASS and 2 SKIP. The only 3 FAI
|
||||
were a script defect in the project check, now fixed. The maintainer accepted
|
||||
leaving Argo's `default` project unrestricted. Break-glass membership is in the
|
||||
new docs/BACKLOG.md. Next: M8.3, durable stores in dev and stage.
|
||||
|
||||
## M8.3 dev/stage stores (2026-09-29)
|
||||
|
||||
ADR-0070 sets up the dev and stage stores:
|
||||
- **Database:** each environment gets its own database and logins on the external
|
||||
PostgreSQL VM.
|
||||
- **Object store:** the filesystem store stays, on a node-local PVC in the new
|
||||
Retain class `aevolution-local-retain`, pinned to one labelled worker. There is
|
||||
no object server.
|
||||
- **Capacity:** node-local volumes ignore their requested size, so the object
|
||||
store now enforces a free-space reserve (`--objects-reserve-mib`, default 1024).
|
||||
A full volume refuses new objects with a distinct error, and nothing is
|
||||
acknowledged.
|
||||
- **Qualification:** `evo-world-clone check-store`, now in the server image,
|
||||
checks from inside the cluster: volume presence and retention, the reserve,
|
||||
database version and TLS, and the runtime privilege boundary.
|
||||
|
||||
Not yet done:
|
||||
- the in-cluster run (it needs Secrets with the documented key names and
|
||||
published image digests);
|
||||
- choosing a backup destination (docs/BACKLOG.md).
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# ADR-0070: Durable stores for dev and stage
|
||||
|
||||
## Status
|
||||
Accepted for M8.3, 2026-09-29. It builds on ADR-0059 (PostgreSQL), ADR-0061
|
||||
(object store), ADR-0069 (environments) and the M8.1 inventory. Storage decisions
|
||||
are provisional until the planned RAID and disk rework (docs/BACKLOG.md).
|
||||
|
||||
## Context
|
||||
Inventory facts that drive this decision:
|
||||
- PostgreSQL runs on a separate VM outside Kubernetes (maintainer). The adapter
|
||||
already requires exactly 18.6 and `verify-full` TLS off-host, and the migration
|
||||
audits the runtime role.
|
||||
- The cluster's only StorageClass, `local-path`, is node-local, reclaims with
|
||||
Delete and does not enforce requested sizes.
|
||||
- Object publication on the workers measured 29–36 ms mean (73 ms max). Worker-3's
|
||||
root filesystem is 87 % full.
|
||||
- MinIO in the cluster is unpinned (`latest`) and belongs to another project.
|
||||
|
||||
The roadmap asks for:
|
||||
- a supported PostgreSQL and a maintained object store, with pinned versions;
|
||||
- PV retention and a backup destination;
|
||||
- evidence for pod restart, missing-volume refusal, object semantics, credential
|
||||
rotation and full disk.
|
||||
|
||||
The named risk is that a local PV binds failover to a node that has been lost.
|
||||
|
||||
## Decision
|
||||
**Database.** Use the maintainer's PostgreSQL VM. Each environment gets its own
|
||||
database, and its own migrator login and NOLOGIN runtime group with a member login
|
||||
(ADR-0060). The version pin (18.6), TLS verification and privilege audit are
|
||||
enforced by code at connect and migrate time; they are not assumed. Migrations run
|
||||
as a hand-applied Job using the pinned `postgres-tools` image.
|
||||
|
||||
**Object store.** Keep the ADR-0061 filesystem store. There is **no object
|
||||
server**: MinIO is ruled out as a new default (INFRASTRUCTURE_ARCHITECTURE), and an
|
||||
S3 adapter would be a new dependency. Instead:
|
||||
- The store lives on one PVC per environment, `aevolution-objects`, in the new
|
||||
class `aevolution-local-retain` (the same provisioner, with **Retain** and
|
||||
WaitForFirstConsumer; never the default class).
|
||||
- The PVC is platform-owned (hand-applied, never Argo).
|
||||
- Every consumer selects the node label `aevolution.io/stores=true`, which is put
|
||||
on exactly one worker. worker-1 or worker-2 is recommended; never worker-3 or
|
||||
the control plane.
|
||||
|
||||
**Capacity guard.** `local-path` ignores the requested size, so
|
||||
`FilesystemObjectStore` now takes a free-space reserve. `evo-server` exposes it as
|
||||
`--objects-reserve-mib`, default 1024.
|
||||
- A new object that would leave less free space is refused with a distinct
|
||||
`ObjectError::full`, before anything is written.
|
||||
- Identical objects that already exist stay reusable.
|
||||
- The host treats a refused upload as a retryable store failure, so nothing is
|
||||
published or acknowledged, and a drain that cannot publish ends without
|
||||
committing anything (exit 3).
|
||||
- A genuinely failing write (ENOSPC or EFBIG) is `io`, and leaves no temporary
|
||||
file or object behind.
|
||||
|
||||
**Qualification tool.** `evo-world-clone check-store` runs in the cluster as a Job
|
||||
with the runtime identity. It:
|
||||
- refuses a missing or unmounted object root;
|
||||
- creates a fixed 64 KiB probe object once, then verifies it by digest on every
|
||||
later run (proof of retention across pods);
|
||||
- reports free space against the reserve;
|
||||
- reports the server version, user and TLS;
|
||||
- requires that the runtime login reaches the fenced functions but is denied
|
||||
table access.
|
||||
|
||||
The server image now includes the tool.
|
||||
|
||||
**Secret contract.**
|
||||
- `aevolution-db` supplies only `PGHOST`, `PGHOSTADDR`, `PGPORT` and `PGDATABASE`.
|
||||
- `evolution-postgres-runtime` supplies `user`, `pgpass` and `ca.crt`.
|
||||
- `evolution-postgres-migrator` supplies `pg_service.conf`, `pgpass` and `ca.crt`.
|
||||
|
||||
Passwords are only ever mounted as files, and copied into private memory
|
||||
(mode 0600) before use, because libpq ignores group-readable password files. The
|
||||
policy check forbids `PGPASSWORD`/`PGUSER` references and whole-Secret `envFrom`.
|
||||
|
||||
**Retention and backup.**
|
||||
- Deleting the claim leaves the PV Released with its data. Recovery is an explicit
|
||||
rebind by an administrator, verified by `check-store` ("probe verified").
|
||||
- The backup destination for the database VM and the object PVC is **not decided
|
||||
yet**. It is a maintainer decision, recorded in docs/BACKLOG.md and required
|
||||
before M8.7. Dev and stage run without a verified backup, which is acceptable
|
||||
only because they are disposable. World0 remains blocked.
|
||||
|
||||
## Consequences
|
||||
- A lost worker takes its object volume with it. The dev and stage regions on that
|
||||
worker are then unavailable (the server refuses a missing volume and quarantines
|
||||
missing objects); they are never silently re-initialized. This is the named risk,
|
||||
accepted for dev and stage only. World0 needs replicated or off-node storage
|
||||
(M8.7/M8.8).
|
||||
- The reserve and the kubelet's nodefs eviction threshold interact. Set the reserve
|
||||
above the node's remaining headroom target, and alert on free space (M8.5).
|
||||
- Re-probe and re-decide after the disk rework.
|
||||
|
||||
## Alternatives rejected
|
||||
- **MinIO or another object server.** It adds an unpinned or new dependency, a
|
||||
second durability layer to back up, and network latency, for no gain on one
|
||||
node.
|
||||
- **Leaving `local-path`/Delete for durable data.** Deleting the claim or an Argo
|
||||
prune would erase the store.
|
||||
- **Trusting PVC sizes.** `local-path` does not enforce them.
|
||||
@@ -80,3 +80,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0067: Region clones and retained-history audit](0067-region-clones.md) - accepted for M7.9a re-identified clone genesis naming its source cut, isolated clone databases and full-chain retention audit.
|
||||
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
|
||||
- [ADR-0069: Environment isolation and World0 protection](0069-environment-isolation.md) - accepted for M8.2 per-environment guardrails, native World0 admission protection, scoped Argo projects and verification.
|
||||
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
|
||||
|
||||
@@ -2689,3 +2689,42 @@ admission policy and the stage and World0 AppProjects, then ran
|
||||
`[]`. The corrected script has not been re-run on the cluster.
|
||||
- **WARN:** the `default` AppProject allows every namespace. The maintainer
|
||||
accepted this (docs/BACKLOG.md).
|
||||
|
||||
## M8.3 dev/stage stores - 2026-09-29
|
||||
|
||||
- WSL Linux `checks` set: CTest 59/59 in each configuration. The format check
|
||||
passed.
|
||||
- PostgreSQL suites: Release and ASan/UBSan passed with the new
|
||||
`region_store_tests`. Analysis passed after one clang-tidy fix (an int
|
||||
multiplication widened in the MiB report).
|
||||
- Store tests:
|
||||
- An unreachable reserve refuses new objects with `full`, leaving no files. An
|
||||
identical existing object is still reused.
|
||||
- A lineage batch through a full store fails as `upload`/`full`, and the head is
|
||||
unchanged.
|
||||
- RLIMIT_FSIZE makes a real write fail with `io`, leaving no object or
|
||||
temporary.
|
||||
- `check-store`:
|
||||
- its first run creates the probe and its second verifies it;
|
||||
- a missing root exits 4 without creating it;
|
||||
- a full volume exits 4;
|
||||
- an owner login exits 3 (tables READABLE);
|
||||
- the runtime login reports `version=180006 ... ssl=false` (local socket), and
|
||||
`tables=denied functions=reachable`.
|
||||
- An `evo-server` whose reserve cannot be met simulated but acknowledged
|
||||
nothing. Its drain exited 3, and the head was unchanged.
|
||||
- Mutations (restored and rebuilt), each caught:
|
||||
- removing the reserve check;
|
||||
- `check-store` always succeeding.
|
||||
- `tools/ci/deploy-policy-check.py`: 41 objects, 0 failures. Four new-rule
|
||||
mutations were caught:
|
||||
- a Delete class;
|
||||
- a platform claim on `local-path`;
|
||||
- a `PGPASSWORD` key reference;
|
||||
- a whole-Secret `envFrom`.
|
||||
- Kustomize renders the dev and stage overlays with the `aevolution-objects`
|
||||
claim; World0 has none.
|
||||
- The server image now contains `evo-world-clone`, and its smoke script checks the
|
||||
tool's usage exit. It has not been built locally, because Docker is not running.
|
||||
- Not executed: anything in the cluster, the store Jobs, retention, rebind and
|
||||
rotation evidence, and the image build.
|
||||
|
||||
@@ -21,12 +21,12 @@ manual, and none is performed by Argo CD.
|
||||
Each namespace gets its own Secrets, with environment-specific values:
|
||||
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
|
||||
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
|
||||
database.
|
||||
database. Keys: `pg_service.conf`, `pgpass`, `ca.crt`.
|
||||
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
|
||||
NOLOGIN runtime group (ADR-0060).
|
||||
- `aevolution-db`: connection settings (host, address, port, database, TLS mode
|
||||
and root certificate). Mount it as files. The adapter reads a password file,
|
||||
never `PGPASSWORD`.
|
||||
NOLOGIN runtime group (ADR-0060). Keys: `user`, `pgpass`, `ca.crt`.
|
||||
- `aevolution-db`: connection settings. Only `PGHOST`, `PGHOSTADDR` (numeric),
|
||||
`PGPORT` and `PGDATABASE` are ever read, as individual variables. `PGPASSWORD`
|
||||
and `PGUSER` are never mounted; the policy check forbids it.
|
||||
|
||||
A World0 credential must never exist in the dev or stage namespaces, and no dev or
|
||||
stage credential may reach the World0 database. Rotate by replacing the Secret and
|
||||
@@ -52,6 +52,48 @@ bash tools/ops/verify-isolation.sh
|
||||
|
||||
Create the Secrets listed above in each namespace before any workload syncs.
|
||||
|
||||
## Stores (M8.3, ADR-0070)
|
||||
|
||||
Run this once per disposable environment (`dev`, then `stage`).
|
||||
|
||||
**1. Label the store node, and apply the storage class and claim.**
|
||||
```sh
|
||||
# Label exactly one worker (worker-1 or worker-2; never worker-3 or the control plane).
|
||||
kubectl label node rke2-worker-1 aevolution.io/stores=true
|
||||
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
|
||||
kubectl apply -k deploy/kubernetes/platform/dev # adds the aevolution-objects claim
|
||||
```
|
||||
|
||||
**2. Prepare the store Jobs.** Copy `deploy/kubernetes/stores/stores.yaml.example`,
|
||||
replace its `REPLACE_*` markers (namespace, database address and port, runtime
|
||||
role, image digests), and create the Secrets above.
|
||||
|
||||
**3. Run and qualify.**
|
||||
```sh
|
||||
kubectl apply -f stores-dev.yaml # policy + both Jobs
|
||||
kubectl logs -n evolution-dev job/aevolution-db-migrate # schema and audit
|
||||
kubectl logs -n evolution-dev job/aevolution-store-check # expect exit 0 and "probe created"
|
||||
```
|
||||
|
||||
Evidence to record in docs/engineering/VALIDATION.md:
|
||||
- **Pod restart:** delete the check Job and apply it again. It must print
|
||||
`probe verified`, with the same digest.
|
||||
- **Missing volume:** run the check with the `objects` volume removed. It must exit
|
||||
4 with `objects refused missing-or-not-a-directory`.
|
||||
- **Full volume:** add `--reserve-mib 1048576` and use a new probe region, such as
|
||||
`--region 2`. It must exit 4 with `probe refused full`.
|
||||
- **Retention:** `kubectl delete pvc aevolution-objects -n evolution-dev`. The PV
|
||||
must become `Released`, not deleted. To rebind:
|
||||
- clear the PV's `spec.claimRef`;
|
||||
- recreate the claim with `volumeName` set to that PV;
|
||||
- rerun the check, which must print `probe verified`.
|
||||
- **Credential rotation:**
|
||||
- Change the runtime login's password on the database VM, and run the check
|
||||
before updating the Secret. It must exit 3 with `database refused connect`.
|
||||
- Update the Secret's `pgpass` and rerun. It must exit 0.
|
||||
- **Privilege boundary:** every run must print
|
||||
`database tables=denied functions=reachable`.
|
||||
|
||||
## Break-glass
|
||||
|
||||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||||
|
||||
Reference in New Issue
Block a user