Drill-only --synthetic-until/--accelerate and evo-world-clone compare (compare_region_histories) prove a candidate build runs the same world on clones of one cut before any barrier. upgrade-check.sh rehearses a failed migration under load and the version barrier with real hosts; upgrade-barrier.sh retires an old build in dev/stage only once the new digest is Ready. Nodes: aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is preference-only (Alertmanager first), checked by policy and scheduling-check.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
252 lines
12 KiB
Markdown
252 lines
12 KiB
Markdown
# AEvolution environments (M8.2)
|
||
|
||
The decision is recorded in ADR-0069. This runbook applies the platform layer,
|
||
creates the per-environment references, and verifies isolation. Every step is
|
||
manual, and none is performed by Argo CD.
|
||
|
||
## Environments
|
||
|
||
| | dev | stage | world0 |
|
||
|---|---|---|---|
|
||
| Namespace | `evolution-dev` (exists) | `evolution-stage` | `evolution-world0` |
|
||
| Purpose | Development | Clone, migration and soak rehearsal | Persistent world (launch gated by M8.8) |
|
||
| Protected by admission | No | No | Yes |
|
||
| Cleanup tool | Allowed | Allowed | Refused |
|
||
| AppProject / Application | `aevolution-dev` (live, manual) | `deploy/argocd/stage/*.example` | `deploy/argocd/world0/*.example` (deny window) |
|
||
| Database | Its own database and logins on the AEvolution PostgreSQL VM | Its own | Its own |
|
||
| World IDs | Development range | Clone targets (ADR-0067) | World0's own; never reused elsewhere |
|
||
|
||
## Secret references (created by an operator, never committed)
|
||
|
||
Each environment has its own **database**: dev uses `aevolution` on the PostgreSQL
|
||
VM `pg-forge-01`. Stage needs a separate database, with its own migrator login and
|
||
runtime group. Sharing one database would let one environment's runtime login act
|
||
on the other's regions, because the fenced functions are per database, not per
|
||
environment.
|
||
|
||
Each namespace gets its own Secrets, with environment-specific values:
|
||
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
|
||
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
|
||
database.
|
||
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
|
||
NOLOGIN runtime group (ADR-0060).
|
||
|
||
Both login Secrets have the keys `pg_service.conf`, `pgpass` and `ca.crt`:
|
||
- `pg_service.conf` has an `[evolution]` section with `host`, a numeric
|
||
`hostaddr`, `port`, `dbname`, `user`, `sslmode=verify-full` and
|
||
`sslrootcert=/run/postgres/ca.crt`. `connect_timeout` is allowed. `evo-server`
|
||
and `evo-world-clone` read it with `--pg-service` and refuse any other key.
|
||
- `pgpass` is mounted, then copied to private memory before use, because libpq
|
||
ignores group-readable password files.
|
||
|
||
`aevolution-db` (the `PG*`/`POSTGRES_*` connection variables) is not used by
|
||
AEvolution workloads. `PGPASSWORD` and `PGUSER` are never placed in a container
|
||
environment, and the policy check forbids it.
|
||
|
||
A World0 credential must never exist in the dev or stage namespaces, and no dev or
|
||
stage credential may reach the World0 database. Rotate by replacing the Secret and
|
||
restarting the consumer.
|
||
|
||
## Delivery through Argo CD (maintainer preference, 2026-09-29)
|
||
|
||
Everything except Secrets and one node label is delivered from Git by Argo CD.
|
||
|
||
**Bootstrap once.** Argo CD cannot create its own first project and application:
|
||
```sh
|
||
kubectl apply -f deploy/argocd/platform/project.yaml -f deploy/argocd/platform/application.yaml
|
||
```
|
||
`aevolution-platform` then syncs `deploy/kubernetes/platform/cluster`:
|
||
- the World0 admission policy;
|
||
- the `aevolution-local-retain` storage class;
|
||
- each environment's namespace, quota, limit range, default-deny policies,
|
||
service accounts and object claim (dev and stage).
|
||
|
||
It self-heals, so a deleted quota, policy or label comes back. It never prunes,
|
||
and every object carries `Prune=false,Delete=false`, so removing a file or
|
||
deleting the Application never deletes a namespace, claim, class or policy. The
|
||
admission policy also denies deleting the platform Application or project. Objects
|
||
applied by hand earlier are adopted (server-side apply).
|
||
|
||
**Workloads.** Each environment's Application syncs its path, currently manually:
|
||
- `aevolution-dev` syncs `deploy/kubernetes/dev`, which is live.
|
||
- Stage uses `deploy/argocd/stage`, whose path is resolved when its values are
|
||
supplied.
|
||
|
||
The store Jobs (`deploy/kubernetes/stores/<env>`) are included in the environment
|
||
path as sync hooks:
|
||
- the database migration runs as a PreSync hook, so a failed migration stops the
|
||
sync;
|
||
- `check-store` runs as a PostSync hook.
|
||
|
||
An environment's stores overlay is included only once it has its published image
|
||
digests and its runtime group role. The policy check refuses any `REPLACE_` marker
|
||
in a synced path.
|
||
|
||
**Outside Argo (by design, for now).**
|
||
- Secrets are created by an operator; a secret manager is in docs/BACKLOG.md.
|
||
- One node label: `kubectl label node rke2-worker-1 aevolution.io/stores=true`.
|
||
The maintainer allows any worker; worker-1 is chosen because it had the lowest
|
||
measured latency, and worker-3's disk is 87 % full.
|
||
|
||
**Verify:** `bash tools/ops/verify-isolation.sh` makes only reads, can-i queries and
|
||
server dry runs.
|
||
|
||
## Stores (M8.3, ADR-0070)
|
||
|
||
Once `deploy/kubernetes/stores/dev` has its values and is included in
|
||
`deploy/kubernetes/dev`, a manual sync of `aevolution-dev` runs the migration and
|
||
then the store check. The hook Jobs keep their logs until the next sync. Record
|
||
this evidence in docs/engineering/VALIDATION.md:
|
||
- **Pod restart:** sync again. The check must print `probe verified` with the same
|
||
digest.
|
||
- **Privilege boundary:** every run prints
|
||
`database tables=denied functions=reachable`.
|
||
- **Missing volume:** temporarily remove the `objects` volume in a branch, or run the
|
||
image by hand without it. The check must exit 4 with
|
||
`objects refused missing-or-not-a-directory`.
|
||
- **Full volume:** add `--reserve-mib 1048576` and a new probe region, such as
|
||
`--region 2`. The check must exit 4 with `probe refused full`.
|
||
- **Retention:** delete the claim by hand, since Argo never prunes it. The PV must
|
||
become `Released`, not deleted. To rebind:
|
||
- clear the PV's `spec.claimRef`;
|
||
- let the platform app recreate the claim, with `volumeName` pinned in Git;
|
||
- sync, and check that it prints `probe verified`.
|
||
- **Credential rotation:**
|
||
- Change the runtime login's password on the database VM and sync before updating
|
||
the Secret. The check must exit 3 with `database refused connect`.
|
||
- Update the Secret's `pgpass` and sync again. It must exit 0.
|
||
|
||
## Region host (M8.4, ADR-0071)
|
||
|
||
`deploy/kubernetes/<env>` includes `stores/<env>` and `region/<env>`. A sync runs:
|
||
1. the database egress policy (wave −2);
|
||
2. the migration hook (wave −1);
|
||
3. the provisioning hook (wave 0, idempotent);
|
||
4. the `evo-region` Deployment and `evo-region-ops` Service (wave 1);
|
||
5. the store check (PostSync).
|
||
|
||
Dev is world 100, region 1; stage is world 200, region 1. The server image digest
|
||
in `region/<env>` is also the build identity.
|
||
|
||
Before the first region sync, apply `deploy/argocd/dev/project.yaml` (it allows
|
||
Deployment and Service), and label the store node.
|
||
|
||
Evidence: `bash tools/ops/region-drill.sh dev` (dev or stage only; World0 is
|
||
refused). It records the time to Ready after a rollout restart and after a forced
|
||
pod deletion, and checks that the new epoch resumed at the committed tick. Also
|
||
record:
|
||
- the Argo sync result;
|
||
- the first pod's `phase running epoch=... committed=...` line;
|
||
- that `/readyz` stays 503 while a replacement waits out the lease.
|
||
|
||
## Monitoring (M8.5, ADR-0072)
|
||
|
||
Bootstrap once: `kubectl apply -f deploy/argocd/monitoring/`, then sync
|
||
`aevolution-platform` (it adds the `aevolution-monitoring` namespace and its
|
||
guardrails) and `aevolution-monitoring`. Re-sync dev and stage, which adds the
|
||
`evo-region-scrape` policy.
|
||
|
||
```sh
|
||
kubectl -n aevolution-monitoring port-forward svc/prometheus 9090 &
|
||
curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"' # both regions up
|
||
curl -s localhost:9090/api/v1/alerts # Watchdog firing only
|
||
```
|
||
|
||
**Injection in the cluster** (dev only):
|
||
- `kubectl -n evolution-dev scale deploy/evo-region --replicas=0` makes
|
||
`EvoRegionDown` pending, then firing after 2 minutes, in Alertmanager (null
|
||
receiver).
|
||
- Scale back to 1 and it resolves.
|
||
|
||
Record the timestamps. The rule tests in CI already prove the thresholds.
|
||
|
||
## Fault drills (M8.6, ADR-0073)
|
||
|
||
`bash tools/ops/fault-drill.sh dev all` enables synthetic load, then runs three
|
||
forced kills, a full volume and an empty volume. It restores the Git settings on
|
||
exit.
|
||
|
||
For the database outage, run `bash tools/ops/fault-drill.sh dev db-outage` and
|
||
follow its prompt. On `pg-forge-01`, block the store node for at least 30 s, then
|
||
lift the block:
|
||
```sh
|
||
sudo iptables -I INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP # rke2-worker-1
|
||
sleep 40
|
||
sudo iptables -D INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP
|
||
```
|
||
Record each `kill`, `full`, `empty` and `db-outage` line in VALIDATION.md. During
|
||
the drill, Prometheus should show:
|
||
- `EvoRegionObjectsFull` (full);
|
||
- `EvoRegionQuarantined` or `EvoRegionNotReady` (empty);
|
||
- `EvoRegionFenced` (db-outage).
|
||
|
||
Keep drills short: synthetic load leaves permanent history.
|
||
|
||
## Break-glass
|
||
|
||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||
nobody holds the group, so World0 deletions are impossible short of removing the
|
||
policy.
|
||
|
||
To delete World0 data or identity on purpose:
|
||
1. Record the operation and its approval.
|
||
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
|
||
example a short-lived client certificate with that organization.
|
||
3. Take and verify a backup first (M8.7; see "Backups" below).
|
||
|
||
Removing the admission policy itself is the break-glass of last resort. It needs
|
||
cluster-admin and must be recorded the same way. Never test by deleting World0: the
|
||
verify script uses server-side dry runs.
|
||
|
||
## Residual risks
|
||
- **`default` AppProject.** It still allows every destination, so its applications
|
||
(other projects, two with automated prune) can create objects in AEvolution
|
||
namespaces. Admission denies only World0 deletions. The maintainer accepted this
|
||
risk on 2026-09-29 and is leaving `default` as it is (docs/BACKLOG.md).
|
||
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
|
||
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
|
||
node-local until M8.3.
|
||
|
||
## Backups (M8.7, ADR-0074)
|
||
|
||
Dev and stage only; the tools refuse World0. From a machine with kubectl access
|
||
that is not the hypervisor host:
|
||
|
||
```bash
|
||
# 1. Sync the environment's application once so the egress policy admits the
|
||
# backup component and the store overlay pins a postgres-tools image that
|
||
# contains pg_dump (any image built after ADR-0074).
|
||
# 2. Back up to a local directory (the destination until one is chosen):
|
||
tools/backup/cluster-backup.sh dev --to /path/to/backups
|
||
# 3. Restore-test it on independent storage and rebuild it as isolated world 900
|
||
# (WSL: the pinned PostgreSQL build and the Linux PostgreSQL build of the tools).
|
||
PATH="$PWD/.cache/postgres/install/bin:$PATH" \
|
||
LD_LIBRARY_PATH="$PWD/.cache/postgres/install/lib" \
|
||
EVO_WORLD_CLONE_BIN="$PWD/build/postgres-linux/evo-world-clone" \
|
||
EVO_SERVER_BIN="$PWD/build/postgres-linux/evo-server" \
|
||
tools/backup/verify-backup.sh /path/to/backups/dev-<time> --isolate 900
|
||
```
|
||
|
||
Only `verify-backup.sh` output that ends in `PASS restore verified and world ...
|
||
rebuilt as isolated world ...` counts as restore evidence. The restore uses fresh
|
||
roles on a throwaway cluster; nothing reaches the source database. A restored copy
|
||
keeps the source world identity and is never served; restoring in place after host
|
||
loss is M8.8.
|
||
|
||
## Upgrades and nodes (M8.9, ADR-0076)
|
||
|
||
**Build upgrade (same rules).**
|
||
1. Rehearse on clones first: clone the cut twice into an isolated database, run the
|
||
current and candidate builds with `--synthetic-inputs development-v1
|
||
--synthetic-until T --accelerate 20`, then `evo-world-clone compare` must exit 0.
|
||
2. Pin the new server digest in `deploy/kubernetes/region/<env>` (image and
|
||
`REGION_BUILD`), add the old digest to `REGION_RETAINED`, and sync. The migrate
|
||
hook (wave -1) stops the sync on failure, leaving the old host running.
|
||
3. When the new pod is Ready: `tools/ops/upgrade-barrier.sh <env> --retire OLD`.
|
||
Keep OLD in `REGION_RETAINED`: its history stays recoverable.
|
||
|
||
**Adding a node.** Label it `aevolution.io/compute=true` and run
|
||
`tools/ops/scheduling-check.sh <env>`. Only stateless pods (Alertmanager today) may
|
||
move there. Never move `aevolution.io/stores` to rebalance: the objects volume stays
|
||
on its node; a region moves only by restore or clone onto the new node's storage.
|