Deploy evo-region per environment (ADR-0071): one replica with the Recreate strategy (a rolling update would deadlock on the lease), /readyz readiness, /livez liveness, no CPU limit and a drain inside the grace period, with the image digest doubling as the build identity. A new idempotent evo-world-clone provision subcommand creates genesis from development_opening (byte-identical to the qualification fixture) as a Sync hook after the migration. Dev and stage paths include their region; a dev AppProject allows Deployment and Service; region-drill.sh times rollout and forced-deletion recovery; the policy check enforces the host's invariants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
166 lines
8.0 KiB
Markdown
166 lines
8.0 KiB
Markdown
# AEvolution environments (M8.2)
|
||
|
||
The decision is recorded in ADR-0069. This runbook applies the platform layer,
|
||
creates the per-environment references, and verifies isolation. Every step is
|
||
manual, and none is performed by Argo CD.
|
||
|
||
## Environments
|
||
|
||
| | dev | stage | world0 |
|
||
|---|---|---|---|
|
||
| Namespace | `evolution-dev` (exists) | `evolution-stage` | `evolution-world0` |
|
||
| Purpose | Development | Clone, migration and soak rehearsal | Persistent world (launch gated by M8.8) |
|
||
| Protected by admission | No | No | Yes |
|
||
| Cleanup tool | Allowed | Allowed | Refused |
|
||
| AppProject / Application | `aevolution-dev` (live, manual) | `deploy/argocd/stage/*.example` | `deploy/argocd/world0/*.example` (deny window) |
|
||
| Database | Its own database and logins on the AEvolution PostgreSQL VM | Its own | Its own |
|
||
| World IDs | Development range | Clone targets (ADR-0067) | World0's own; never reused elsewhere |
|
||
|
||
## Secret references (created by an operator, never committed)
|
||
|
||
Each environment has its own **database**: dev uses `aevolution` on the PostgreSQL
|
||
VM `pg-forge-01`. Stage needs a separate database, with its own migrator login and
|
||
runtime group. Sharing one database would let one environment's runtime login act
|
||
on the other's regions, because the fenced functions are per database, not per
|
||
environment.
|
||
|
||
Each namespace gets its own Secrets, with environment-specific values:
|
||
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
|
||
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
|
||
database.
|
||
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
|
||
NOLOGIN runtime group (ADR-0060).
|
||
|
||
Both login Secrets have the keys `pg_service.conf`, `pgpass` and `ca.crt`:
|
||
- `pg_service.conf` has an `[evolution]` section with `host`, a numeric
|
||
`hostaddr`, `port`, `dbname`, `user`, `sslmode=verify-full` and
|
||
`sslrootcert=/run/postgres/ca.crt`. `connect_timeout` is allowed. `evo-server`
|
||
and `evo-world-clone` read it with `--pg-service` and refuse any other key.
|
||
- `pgpass` is mounted, then copied to private memory before use, because libpq
|
||
ignores group-readable password files.
|
||
|
||
`aevolution-db` (the `PG*`/`POSTGRES_*` connection variables) is not used by
|
||
AEvolution workloads. `PGPASSWORD` and `PGUSER` are never placed in a container
|
||
environment, and the policy check forbids it.
|
||
|
||
A World0 credential must never exist in the dev or stage namespaces, and no dev or
|
||
stage credential may reach the World0 database. Rotate by replacing the Secret and
|
||
restarting the consumer.
|
||
|
||
## Delivery through Argo CD (maintainer preference, 2026-09-29)
|
||
|
||
Everything except Secrets and one node label is delivered from Git by Argo CD.
|
||
|
||
**Bootstrap once.** Argo CD cannot create its own first project and application:
|
||
```sh
|
||
kubectl apply -f deploy/argocd/platform/project.yaml -f deploy/argocd/platform/application.yaml
|
||
```
|
||
`aevolution-platform` then syncs `deploy/kubernetes/platform/cluster`:
|
||
- the World0 admission policy;
|
||
- the `aevolution-local-retain` storage class;
|
||
- each environment's namespace, quota, limit range, default-deny policies,
|
||
service accounts and object claim (dev and stage).
|
||
|
||
It self-heals, so a deleted quota, policy or label comes back. It never prunes,
|
||
and every object carries `Prune=false,Delete=false`, so removing a file or
|
||
deleting the Application never deletes a namespace, claim, class or policy. The
|
||
admission policy also denies deleting the platform Application or project. Objects
|
||
applied by hand earlier are adopted (server-side apply).
|
||
|
||
**Workloads.** Each environment's Application syncs its path, currently manually:
|
||
- `aevolution-dev` syncs `deploy/kubernetes/dev`, which is live.
|
||
- Stage uses `deploy/argocd/stage`, whose path is resolved when its values are
|
||
supplied.
|
||
|
||
The store Jobs (`deploy/kubernetes/stores/<env>`) are included in the environment
|
||
path as sync hooks:
|
||
- the database migration runs as a PreSync hook, so a failed migration stops the
|
||
sync;
|
||
- `check-store` runs as a PostSync hook.
|
||
|
||
An environment's stores overlay is included only once it has its published image
|
||
digests and its runtime group role. The policy check refuses any `REPLACE_` marker
|
||
in a synced path.
|
||
|
||
**Outside Argo (by design, for now).**
|
||
- Secrets are created by an operator; a secret manager is in docs/BACKLOG.md.
|
||
- One node label: `kubectl label node rke2-worker-1 aevolution.io/stores=true`.
|
||
The maintainer allows any worker; worker-1 is chosen because it had the lowest
|
||
measured latency, and worker-3's disk is 87 % full.
|
||
|
||
**Verify:** `bash tools/ops/verify-isolation.sh` makes only reads, can-i queries and
|
||
server dry runs.
|
||
|
||
## Stores (M8.3, ADR-0070)
|
||
|
||
Once `deploy/kubernetes/stores/dev` has its values and is included in
|
||
`deploy/kubernetes/dev`, a manual sync of `aevolution-dev` runs the migration and
|
||
then the store check. The hook Jobs keep their logs until the next sync. Record
|
||
this evidence in docs/engineering/VALIDATION.md:
|
||
- **Pod restart:** sync again. The check must print `probe verified` with the same
|
||
digest.
|
||
- **Privilege boundary:** every run prints
|
||
`database tables=denied functions=reachable`.
|
||
- **Missing volume:** temporarily remove the `objects` volume in a branch, or run the
|
||
image by hand without it. The check must exit 4 with
|
||
`objects refused missing-or-not-a-directory`.
|
||
- **Full volume:** add `--reserve-mib 1048576` and a new probe region, such as
|
||
`--region 2`. The check must exit 4 with `probe refused full`.
|
||
- **Retention:** delete the claim by hand, since Argo never prunes it. The PV must
|
||
become `Released`, not deleted. To rebind:
|
||
- clear the PV's `spec.claimRef`;
|
||
- let the platform app recreate the claim, with `volumeName` pinned in Git;
|
||
- sync, and check that it prints `probe verified`.
|
||
- **Credential rotation:**
|
||
- Change the runtime login's password on the database VM and sync before updating
|
||
the Secret. The check must exit 3 with `database refused connect`.
|
||
- Update the Secret's `pgpass` and sync again. It must exit 0.
|
||
|
||
## Region host (M8.4, ADR-0071)
|
||
|
||
`deploy/kubernetes/<env>` includes `stores/<env>` and `region/<env>`. A sync runs:
|
||
1. the database egress policy (wave −2);
|
||
2. the migration hook (wave −1);
|
||
3. the provisioning hook (wave 0, idempotent);
|
||
4. the `evo-region` Deployment and `evo-region-ops` Service (wave 1);
|
||
5. the store check (PostSync).
|
||
|
||
Dev is world 100, region 1; stage is world 200, region 1. The server image digest
|
||
in `region/<env>` is also the build identity.
|
||
|
||
Before the first region sync, apply `deploy/argocd/dev/project.yaml` (it allows
|
||
Deployment and Service), and label the store node.
|
||
|
||
Evidence: `bash tools/ops/region-drill.sh dev` (dev or stage only; World0 is
|
||
refused). It records the time to Ready after a rollout restart and after a forced
|
||
pod deletion, and checks that the new epoch resumed at the committed tick. Also
|
||
record:
|
||
- the Argo sync result;
|
||
- the first pod's `phase running epoch=... committed=...` line;
|
||
- that `/readyz` stays 503 while a replacement waits out the lease.
|
||
|
||
## Break-glass
|
||
|
||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||
nobody holds the group, so World0 deletions are impossible short of removing the
|
||
policy.
|
||
|
||
To delete World0 data or identity on purpose:
|
||
1. Record the operation and its approval.
|
||
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
|
||
example a short-lived client certificate with that organization.
|
||
3. Take and verify a backup first (M8.7).
|
||
|
||
Removing the admission policy itself is the break-glass of last resort. It needs
|
||
cluster-admin and must be recorded the same way. Never test by deleting World0: the
|
||
verify script uses server-side dry runs.
|
||
|
||
## Residual risks
|
||
- **`default` AppProject.** It still allows every destination, so its applications
|
||
(other projects, two with automated prune) can create objects in AEvolution
|
||
namespaces. Admission denies only World0 deletions. The maintainer accepted this
|
||
risk on 2026-09-29 and is leaving `default` as it is (docs/BACKLOG.md).
|
||
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
|
||
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
|
||
node-local until M8.3.
|