Files
aevolution/docs/infrastructure/ENVIRONMENTS.md
T
dingoandClaude Opus 5.5 2d950060db Add M8.3 dev/stage stores: Retain volumes, free-space reserve, store check
Keep the filesystem object store on a node-local Retain-class PVC per
environment (ADR-0070), with the database on the external PostgreSQL VM.
Because local-path does not enforce sizes, the object store now refuses
new objects below a free-space reserve with a distinct full error, so a
full volume publishes and acknowledges nothing. evo-world-clone gains
check-store to qualify volume retention, reserve, database version/TLS
and the runtime privilege boundary from inside the cluster; it now ships
in the server image. Add the storage class, object claims, hand-applied
store Jobs, policy rules forbidding password env vars, and the runbook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:51:15 -04:00

5.9 KiB

AEvolution environments (M8.2)

The decision is recorded in ADR-0069. This runbook applies the platform layer, creates the per-environment references, and verifies isolation. Every step is manual, and none is performed by Argo CD.

Environments

dev stage world0
Namespace evolution-dev (exists) evolution-stage evolution-world0
Purpose Development Clone, migration and soak rehearsal Persistent world (launch gated by M8.8)
Protected by admission No No Yes
Cleanup tool Allowed Allowed Refused
AppProject / Application aevolution-dev (live, manual) deploy/argocd/stage/*.example deploy/argocd/world0/*.example (deny window)
Database Its own database and logins on the AEvolution PostgreSQL VM Its own Its own
World IDs Development range Clone targets (ADR-0067) World0's own; never reused elsewhere

Secret references (created by an operator, never committed)

Each namespace gets its own Secrets, with environment-specific values:

  • harbor-pull: a pull-only Harbor robot (docker-registry type).
  • evolution-postgres-migrator: the migrator/owner login for that environment's database. Keys: pg_service.conf, pgpass, ca.crt.
  • evolution-postgres-runtime: the runtime login, a member of that environment's NOLOGIN runtime group (ADR-0060). Keys: user, pgpass, ca.crt.
  • aevolution-db: connection settings. Only PGHOST, PGHOSTADDR (numeric), PGPORT and PGDATABASE are ever read, as individual variables. PGPASSWORD and PGUSER are never mounted; the policy check forbids it.

A World0 credential must never exist in the dev or stage namespaces, and no dev or stage credential may reach the World0 database. Rotate by replacing the Secret and restarting the consumer.

Apply (cluster administrator)

# 1. Platform guardrails, per environment (dev adds labels, quota and policies to
#    the existing namespace; its completed proof Job already complies).
kubectl apply -k deploy/kubernetes/platform/dev
kubectl apply -k deploy/kubernetes/platform/stage
kubectl apply -k deploy/kubernetes/platform/world0   # an empty, protected namespace
# 2. World0 admission protection (validate first, then apply).
kubectl apply --dry-run=server -f deploy/kubernetes/platform/admission/
kubectl apply -f deploy/kubernetes/platform/admission/
# 3. Projects (resolve the Application paths only when M8.4 adds workloads).
kubectl apply -f deploy/argocd/stage/project.yaml.example
kubectl apply -f deploy/argocd/world0/project.yaml.example
# 4. Verify: reads, can-i and server dry runs only.
bash tools/ops/verify-isolation.sh

Create the Secrets listed above in each namespace before any workload syncs.

Stores (M8.3, ADR-0070)

Run this once per disposable environment (dev, then stage).

1. Label the store node, and apply the storage class and claim.

# Label exactly one worker (worker-1 or worker-2; never worker-3 or the control plane).
kubectl label node rke2-worker-1 aevolution.io/stores=true
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
kubectl apply -k deploy/kubernetes/platform/dev    # adds the aevolution-objects claim

2. Prepare the store Jobs. Copy deploy/kubernetes/stores/stores.yaml.example, replace its REPLACE_* markers (namespace, database address and port, runtime role, image digests), and create the Secrets above.

3. Run and qualify.

kubectl apply -f stores-dev.yaml                              # policy + both Jobs
kubectl logs -n evolution-dev job/aevolution-db-migrate       # schema and audit
kubectl logs -n evolution-dev job/aevolution-store-check      # expect exit 0 and "probe created"

Evidence to record in docs/engineering/VALIDATION.md:

  • Pod restart: delete the check Job and apply it again. It must print probe verified, with the same digest.
  • Missing volume: run the check with the objects volume removed. It must exit 4 with objects refused missing-or-not-a-directory.
  • Full volume: add --reserve-mib 1048576 and use a new probe region, such as --region 2. It must exit 4 with probe refused full.
  • Retention: kubectl delete pvc aevolution-objects -n evolution-dev. The PV must become Released, not deleted. To rebind:
    • clear the PV's spec.claimRef;
    • recreate the claim with volumeName set to that PV;
    • rerun the check, which must print probe verified.
  • Credential rotation:
    • Change the runtime login's password on the database VM, and run the check before updating the Secret. It must exit 3 with database refused connect.
    • Update the Secret's pgpass and rerun. It must exit 0.
  • Privilege boundary: every run must print database tables=denied functions=reachable.

Break-glass

How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is, nobody holds the group, so World0 deletions are impossible short of removing the policy.

To delete World0 data or identity on purpose:

  1. Record the operation and its approval.
  2. Use a credential whose groups include aevolution:world0-break-glass, for example a short-lived client certificate with that organization.
  3. Take and verify a backup first (M8.7).

Removing the admission policy itself is the break-glass of last resort. It needs cluster-admin and must be recorded the same way. Never test by deleting World0: the verify script uses server-side dry runs.

Residual risks

  • default AppProject. It still allows every destination, so its applications (other projects, two with automated prune) can create objects in AEvolution namespaces. Admission denies only World0 deletions. The maintainer accepted this risk on 2026-09-29 and is leaving default as it is (docs/BACKLOG.md).
  • World0 volumes. local-path provisions volumes with Delete. Patch each World0 PV to Retain when it is bound, and the policy then keeps it there. Volumes stay node-local until M8.3.