Files
aevolution/docs/infrastructure/ENVIRONMENTS.md
T
dingoandClaude Opus 5.5 8d594f28af Connect from libpq service files and match the real Secret layout
The maintainer's login Secrets carry pg_service.conf ([evolution]),
pgpass and ca.crt. Add a strict read_pg_service (known keys only,
verify-full unless a local socket, bounded size) behind --pg-service in
evo-server and evo-world-clone, mount each login Secret whole in the
store Jobs instead of reading aevolution-db, and fix the migrate Job's
PGSERVICE to the [evolution] section. Record the store node choice,
per-environment databases and the backup plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 14:12:26 -04:00

6.7 KiB

AEvolution environments (M8.2)

The decision is recorded in ADR-0069. This runbook applies the platform layer, creates the per-environment references, and verifies isolation. Every step is manual, and none is performed by Argo CD.

Environments

dev stage world0
Namespace evolution-dev (exists) evolution-stage evolution-world0
Purpose Development Clone, migration and soak rehearsal Persistent world (launch gated by M8.8)
Protected by admission No No Yes
Cleanup tool Allowed Allowed Refused
AppProject / Application aevolution-dev (live, manual) deploy/argocd/stage/*.example deploy/argocd/world0/*.example (deny window)
Database Its own database and logins on the AEvolution PostgreSQL VM Its own Its own
World IDs Development range Clone targets (ADR-0067) World0's own; never reused elsewhere

Secret references (created by an operator, never committed)

Each environment has its own database: dev uses aevolution on the PostgreSQL VM pg-forge-01. Stage needs a separate database, with its own migrator login and runtime group. Sharing one database would let one environment's runtime login act on the other's regions, because the fenced functions are per database, not per environment.

Each namespace gets its own Secrets, with environment-specific values:

  • harbor-pull: a pull-only Harbor robot (docker-registry type).
  • evolution-postgres-migrator: the migrator/owner login for that environment's database.
  • evolution-postgres-runtime: the runtime login, a member of that environment's NOLOGIN runtime group (ADR-0060).

Both login Secrets have the keys pg_service.conf, pgpass and ca.crt:

  • pg_service.conf has an [evolution] section with host, a numeric hostaddr, port, dbname, user, sslmode=verify-full and sslrootcert=/run/postgres/ca.crt. connect_timeout is allowed. evo-server and evo-world-clone read it with --pg-service and refuse any other key.
  • pgpass is mounted, then copied to private memory before use, because libpq ignores group-readable password files.

aevolution-db (the PG*/POSTGRES_* connection variables) is not used by AEvolution workloads. PGPASSWORD and PGUSER are never placed in a container environment, and the policy check forbids it.

A World0 credential must never exist in the dev or stage namespaces, and no dev or stage credential may reach the World0 database. Rotate by replacing the Secret and restarting the consumer.

Apply (cluster administrator)

# 1. Platform guardrails, per environment (dev adds labels, quota and policies to
#    the existing namespace; its completed proof Job already complies).
kubectl apply -k deploy/kubernetes/platform/dev
kubectl apply -k deploy/kubernetes/platform/stage
kubectl apply -k deploy/kubernetes/platform/world0   # an empty, protected namespace
# 2. World0 admission protection (validate first, then apply).
kubectl apply --dry-run=server -f deploy/kubernetes/platform/admission/
kubectl apply -f deploy/kubernetes/platform/admission/
# 3. Projects (resolve the Application paths only when M8.4 adds workloads).
kubectl apply -f deploy/argocd/stage/project.yaml.example
kubectl apply -f deploy/argocd/world0/project.yaml.example
# 4. Verify: reads, can-i and server dry runs only.
bash tools/ops/verify-isolation.sh

Create the Secrets listed above in each namespace before any workload syncs.

Stores (M8.3, ADR-0070)

Run this once per disposable environment (dev, then stage).

1. Label the store node, and apply the storage class and claim.

# Label exactly one worker. The maintainer allows any worker (never the control plane);
# worker-1 is chosen (lowest measured latency; worker-3's disk is 87 % full).
kubectl label node rke2-worker-1 aevolution.io/stores=true
kubectl apply -f deploy/kubernetes/platform/storage/storageclass.yaml
kubectl apply -k deploy/kubernetes/platform/dev    # adds the aevolution-objects claim

2. Prepare the store Jobs. Copy deploy/kubernetes/stores/stores.yaml.example, replace its REPLACE_* markers (namespace, database address and port, runtime role, image digests), and create the Secrets above.

3. Run and qualify.

kubectl apply -f stores-dev.yaml                              # policy + both Jobs
kubectl logs -n evolution-dev job/aevolution-db-migrate       # schema and audit
kubectl logs -n evolution-dev job/aevolution-store-check      # expect exit 0 and "probe created"

Evidence to record in docs/engineering/VALIDATION.md:

  • Pod restart: delete the check Job and apply it again. It must print probe verified, with the same digest.
  • Missing volume: run the check with the objects volume removed. It must exit 4 with objects refused missing-or-not-a-directory.
  • Full volume: add --reserve-mib 1048576 and use a new probe region, such as --region 2. It must exit 4 with probe refused full.
  • Retention: kubectl delete pvc aevolution-objects -n evolution-dev. The PV must become Released, not deleted. To rebind:
    • clear the PV's spec.claimRef;
    • recreate the claim with volumeName set to that PV;
    • rerun the check, which must print probe verified.
  • Credential rotation:
    • Change the runtime login's password on the database VM, and run the check before updating the Secret. It must exit 3 with database refused connect.
    • Update the Secret's pgpass and rerun. It must exit 0.
  • Privilege boundary: every run must print database tables=denied functions=reachable.

Break-glass

How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is, nobody holds the group, so World0 deletions are impossible short of removing the policy.

To delete World0 data or identity on purpose:

  1. Record the operation and its approval.
  2. Use a credential whose groups include aevolution:world0-break-glass, for example a short-lived client certificate with that organization.
  3. Take and verify a backup first (M8.7).

Removing the admission policy itself is the break-glass of last resort. It needs cluster-admin and must be recorded the same way. Never test by deleting World0: the verify script uses server-side dry runs.

Residual risks

  • default AppProject. It still allows every destination, so its applications (other projects, two with automated prune) can create objects in AEvolution namespaces. Admission denies only World0 deletions. The maintainer accepted this risk on 2026-09-29 and is leaving default as it is (docs/BACKLOG.md).
  • World0 volumes. local-path provisions volumes with Delete. Patch each World0 PV to Retain when it is bound, and the policy then keeps it there. Volumes stay node-local until M8.3.