Files
aevolution/docs/infrastructure/ENVIRONMENTS.md
T
dingoandClaude Opus 5.5 b4493b3d0f Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 06:16:36 -04:00

187 lines
8.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# AEvolution environments (M8.2)
The decision is recorded in ADR-0069. This runbook applies the platform layer,
creates the per-environment references, and verifies isolation. Every step is
manual, and none is performed by Argo CD.
## Environments
| | dev | stage | world0 |
|---|---|---|---|
| Namespace | `evolution-dev` (exists) | `evolution-stage` | `evolution-world0` |
| Purpose | Development | Clone, migration and soak rehearsal | Persistent world (launch gated by M8.8) |
| Protected by admission | No | No | Yes |
| Cleanup tool | Allowed | Allowed | Refused |
| AppProject / Application | `aevolution-dev` (live, manual) | `deploy/argocd/stage/*.example` | `deploy/argocd/world0/*.example` (deny window) |
| Database | Its own database and logins on the AEvolution PostgreSQL VM | Its own | Its own |
| World IDs | Development range | Clone targets (ADR-0067) | World0's own; never reused elsewhere |
## Secret references (created by an operator, never committed)
Each environment has its own **database**: dev uses `aevolution` on the PostgreSQL
VM `pg-forge-01`. Stage needs a separate database, with its own migrator login and
runtime group. Sharing one database would let one environment's runtime login act
on the other's regions, because the fenced functions are per database, not per
environment.
Each namespace gets its own Secrets, with environment-specific values:
- `harbor-pull`: a pull-only Harbor robot (docker-registry type).
- `evolution-postgres-migrator`: the migrator/owner login for that environment's
database.
- `evolution-postgres-runtime`: the runtime login, a member of that environment's
NOLOGIN runtime group (ADR-0060).
Both login Secrets have the keys `pg_service.conf`, `pgpass` and `ca.crt`:
- `pg_service.conf` has an `[evolution]` section with `host`, a numeric
`hostaddr`, `port`, `dbname`, `user`, `sslmode=verify-full` and
`sslrootcert=/run/postgres/ca.crt`. `connect_timeout` is allowed. `evo-server`
and `evo-world-clone` read it with `--pg-service` and refuse any other key.
- `pgpass` is mounted, then copied to private memory before use, because libpq
ignores group-readable password files.
`aevolution-db` (the `PG*`/`POSTGRES_*` connection variables) is not used by
AEvolution workloads. `PGPASSWORD` and `PGUSER` are never placed in a container
environment, and the policy check forbids it.
A World0 credential must never exist in the dev or stage namespaces, and no dev or
stage credential may reach the World0 database. Rotate by replacing the Secret and
restarting the consumer.
## Delivery through Argo CD (maintainer preference, 2026-09-29)
Everything except Secrets and one node label is delivered from Git by Argo CD.
**Bootstrap once.** Argo CD cannot create its own first project and application:
```sh
kubectl apply -f deploy/argocd/platform/project.yaml -f deploy/argocd/platform/application.yaml
```
`aevolution-platform` then syncs `deploy/kubernetes/platform/cluster`:
- the World0 admission policy;
- the `aevolution-local-retain` storage class;
- each environment's namespace, quota, limit range, default-deny policies,
service accounts and object claim (dev and stage).
It self-heals, so a deleted quota, policy or label comes back. It never prunes,
and every object carries `Prune=false,Delete=false`, so removing a file or
deleting the Application never deletes a namespace, claim, class or policy. The
admission policy also denies deleting the platform Application or project. Objects
applied by hand earlier are adopted (server-side apply).
**Workloads.** Each environment's Application syncs its path, currently manually:
- `aevolution-dev` syncs `deploy/kubernetes/dev`, which is live.
- Stage uses `deploy/argocd/stage`, whose path is resolved when its values are
supplied.
The store Jobs (`deploy/kubernetes/stores/<env>`) are included in the environment
path as sync hooks:
- the database migration runs as a PreSync hook, so a failed migration stops the
sync;
- `check-store` runs as a PostSync hook.
An environment's stores overlay is included only once it has its published image
digests and its runtime group role. The policy check refuses any `REPLACE_` marker
in a synced path.
**Outside Argo (by design, for now).**
- Secrets are created by an operator; a secret manager is in docs/BACKLOG.md.
- One node label: `kubectl label node rke2-worker-1 aevolution.io/stores=true`.
The maintainer allows any worker; worker-1 is chosen because it had the lowest
measured latency, and worker-3's disk is 87 % full.
**Verify:** `bash tools/ops/verify-isolation.sh` makes only reads, can-i queries and
server dry runs.
## Stores (M8.3, ADR-0070)
Once `deploy/kubernetes/stores/dev` has its values and is included in
`deploy/kubernetes/dev`, a manual sync of `aevolution-dev` runs the migration and
then the store check. The hook Jobs keep their logs until the next sync. Record
this evidence in docs/engineering/VALIDATION.md:
- **Pod restart:** sync again. The check must print `probe verified` with the same
digest.
- **Privilege boundary:** every run prints
`database tables=denied functions=reachable`.
- **Missing volume:** temporarily remove the `objects` volume in a branch, or run the
image by hand without it. The check must exit 4 with
`objects refused missing-or-not-a-directory`.
- **Full volume:** add `--reserve-mib 1048576` and a new probe region, such as
`--region 2`. The check must exit 4 with `probe refused full`.
- **Retention:** delete the claim by hand, since Argo never prunes it. The PV must
become `Released`, not deleted. To rebind:
- clear the PV's `spec.claimRef`;
- let the platform app recreate the claim, with `volumeName` pinned in Git;
- sync, and check that it prints `probe verified`.
- **Credential rotation:**
- Change the runtime login's password on the database VM and sync before updating
the Secret. The check must exit 3 with `database refused connect`.
- Update the Secret's `pgpass` and sync again. It must exit 0.
## Region host (M8.4, ADR-0071)
`deploy/kubernetes/<env>` includes `stores/<env>` and `region/<env>`. A sync runs:
1. the database egress policy (wave −2);
2. the migration hook (wave −1);
3. the provisioning hook (wave 0, idempotent);
4. the `evo-region` Deployment and `evo-region-ops` Service (wave 1);
5. the store check (PostSync).
Dev is world 100, region 1; stage is world 200, region 1. The server image digest
in `region/<env>` is also the build identity.
Before the first region sync, apply `deploy/argocd/dev/project.yaml` (it allows
Deployment and Service), and label the store node.
Evidence: `bash tools/ops/region-drill.sh dev` (dev or stage only; World0 is
refused). It records the time to Ready after a rollout restart and after a forced
pod deletion, and checks that the new epoch resumed at the committed tick. Also
record:
- the Argo sync result;
- the first pod's `phase running epoch=... committed=...` line;
- that `/readyz` stays 503 while a replacement waits out the lease.
## Monitoring (M8.5, ADR-0072)
Bootstrap once: `kubectl apply -f deploy/argocd/monitoring/`, then sync
`aevolution-platform` (it adds the `aevolution-monitoring` namespace and its
guardrails) and `aevolution-monitoring`. Re-sync dev and stage, which adds the
`evo-region-scrape` policy.
```sh
kubectl -n aevolution-monitoring port-forward svc/prometheus 9090 &
curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"' # both regions up
curl -s localhost:9090/api/v1/alerts # Watchdog firing only
```
**Injection in the cluster** (dev only):
- `kubectl -n evolution-dev scale deploy/evo-region --replicas=0` makes
`EvoRegionDown` pending, then firing after 2 minutes, in Alertmanager (null
receiver).
- Scale back to 1 and it resolves.
Record the timestamps. The rule tests in CI already prove the thresholds.
## Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
nobody holds the group, so World0 deletions are impossible short of removing the
policy.
To delete World0 data or identity on purpose:
1. Record the operation and its approval.
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
example a short-lived client certificate with that organization.
3. Take and verify a backup first (M8.7).
Removing the admission policy itself is the break-glass of last resort. It needs
cluster-admin and must be recorded the same way. Never test by deleting World0: the
verify script uses server-side dry runs.
## Residual risks
- **`default` AppProject.** It still allows every destination, so its applications
(other projects, two with automated prune) can create objects in AEvolution
namespaces. Admission denies only World0 deletions. The maintainer accepted this
risk on 2026-09-29 and is leaving `default` as it is (docs/BACKLOG.md).
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
node-local until M8.3.