Keep the filesystem object store on a node-local Retain-class PVC per environment (ADR-0070), with the database on the external PostgreSQL VM. Because local-path does not enforce sizes, the object store now refuses new objects below a free-space reserve with a distinct full error, so a full volume publishes and acknowledges nothing. evo-world-clone gains check-store to qualify volume retention, reserve, database version/TLS and the runtime privilege boundary from inside the cluster; it now ships in the server image. Add the storage class, object claims, hand-applied store Jobs, policy rules forbidding password env vars, and the runbook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5.9 KiB
AEvolution environments (M8.2)
The decision is recorded in ADR-0069. This runbook applies the platform layer, creates the per-environment references, and verifies isolation. Every step is manual, and none is performed by Argo CD.
Environments
| dev | stage | world0 | |
|---|---|---|---|
| Namespace | evolution-dev (exists) |
evolution-stage |
evolution-world0 |
| Purpose | Development | Clone, migration and soak rehearsal | Persistent world (launch gated by M8.8) |
| Protected by admission | No | No | Yes |
| Cleanup tool | Allowed | Allowed | Refused |
| AppProject / Application | aevolution-dev (live, manual) |
deploy/argocd/stage/*.example |
deploy/argocd/world0/*.example (deny window) |
| Database | Its own database and logins on the AEvolution PostgreSQL VM | Its own | Its own |
| World IDs | Development range | Clone targets (ADR-0067) | World0's own; never reused elsewhere |
Secret references (created by an operator, never committed)
Each namespace gets its own Secrets, with environment-specific values:
harbor-pull: a pull-only Harbor robot (docker-registry type).evolution-postgres-migrator: the migrator/owner login for that environment's database. Keys:pg_service.conf,pgpass,ca.crt.evolution-postgres-runtime: the runtime login, a member of that environment's NOLOGIN runtime group (ADR-0060). Keys:user,pgpass,ca.crt.aevolution-db: connection settings. OnlyPGHOST,PGHOSTADDR(numeric),PGPORTandPGDATABASEare ever read, as individual variables.PGPASSWORDandPGUSERare never mounted; the policy check forbids it.
A World0 credential must never exist in the dev or stage namespaces, and no dev or stage credential may reach the World0 database. Rotate by replacing the Secret and restarting the consumer.
Apply (cluster administrator)
# 1. Platform guardrails, per environment (dev adds labels, quota and policies to
# the existing namespace; its completed proof Job already complies).
kubectl apply -k deploy/kubernetes/platform/dev
kubectl apply -k deploy/kubernetes/platform/stage
kubectl apply -k deploy/kubernetes/platform/world0 # an empty, protected namespace
# 2. World0 admission protection (validate first, then apply).
kubectl apply --dry-run=server -f deploy/kubernetes/platform/admission/
kubectl apply -f deploy/kubernetes/platform/admission/
# 3. Projects (resolve the Application paths only when M8.4 adds workloads).
kubectl apply -f deploy/argocd/stage/project.yaml.example
kubectl apply -f deploy/argocd/world0/project.yaml.example
# 4. Verify: reads, can-i and server dry runs only.
bash tools/ops/verify-isolation.sh
Create the Secrets listed above in each namespace before any workload syncs.
Stores (M8.3, ADR-0070)
Run this once per disposable environment (dev, then stage).
1. Label the store node, and apply the storage class and claim.
# Label exactly one worker (worker-1 or worker-2; never worker-3 or the control plane).
kubectl label node rke2-worker-1 aevolution.io/stores=true
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
kubectl apply -k deploy/kubernetes/platform/dev # adds the aevolution-objects claim
2. Prepare the store Jobs. Copy deploy/kubernetes/stores/stores.yaml.example,
replace its REPLACE_* markers (namespace, database address and port, runtime
role, image digests), and create the Secrets above.
3. Run and qualify.
kubectl apply -f stores-dev.yaml # policy + both Jobs
kubectl logs -n evolution-dev job/aevolution-db-migrate # schema and audit
kubectl logs -n evolution-dev job/aevolution-store-check # expect exit 0 and "probe created"
Evidence to record in docs/engineering/VALIDATION.md:
- Pod restart: delete the check Job and apply it again. It must print
probe verified, with the same digest. - Missing volume: run the check with the
objectsvolume removed. It must exit 4 withobjects refused missing-or-not-a-directory. - Full volume: add
--reserve-mib 1048576and use a new probe region, such as--region 2. It must exit 4 withprobe refused full. - Retention:
kubectl delete pvc aevolution-objects -n evolution-dev. The PV must becomeReleased, not deleted. To rebind:- clear the PV's
spec.claimRef; - recreate the claim with
volumeNameset to that PV; - rerun the check, which must print
probe verified.
- clear the PV's
- Credential rotation:
- Change the runtime login's password on the database VM, and run the check
before updating the Secret. It must exit 3 with
database refused connect. - Update the Secret's
pgpassand rerun. It must exit 0.
- Change the runtime login's password on the database VM, and run the check
before updating the Secret. It must exit 3 with
- Privilege boundary: every run must print
database tables=denied functions=reachable.
Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is, nobody holds the group, so World0 deletions are impossible short of removing the policy.
To delete World0 data or identity on purpose:
- Record the operation and its approval.
- Use a credential whose groups include
aevolution:world0-break-glass, for example a short-lived client certificate with that organization. - Take and verify a backup first (M8.7).
Removing the admission policy itself is the break-glass of last resort. It needs cluster-admin and must be recorded the same way. Never test by deleting World0: the verify script uses server-side dry runs.
Residual risks
defaultAppProject. It still allows every destination, so its applications (other projects, two with automated prune) can create objects in AEvolution namespaces. Admission denies only World0 deletions. The maintainer accepted this risk on 2026-09-29 and is leavingdefaultas it is (docs/BACKLOG.md).- World0 volumes.
local-pathprovisions volumes with Delete. Patch each World0 PV toRetainwhen it is bound, and the policy then keeps it there. Volumes stay node-local until M8.3.