Files
dingoandClaude Opus 5.5 5b482ad6a3 feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:51:17 -04:00

12 KiB
Raw Permalink Blame History

AEvolution environments (M8.2)

The decision is recorded in ADR-0069. This runbook applies the platform layer, creates the per-environment references, and verifies isolation. Every step is manual, and none is performed by Argo CD.

Environments

dev stage world0
Namespace evolution-dev (exists) evolution-stage evolution-world0
Purpose Development Clone, migration and soak rehearsal Persistent world (launch gated by M8.8)
Protected by admission No No Yes
Cleanup tool Allowed Allowed Refused
AppProject / Application aevolution-dev (live, manual) deploy/argocd/stage/*.example deploy/argocd/world0/*.example (deny window)
Database Its own database and logins on the AEvolution PostgreSQL VM Its own Its own
World IDs Development range Clone targets (ADR-0067) World0's own; never reused elsewhere

Secret references (created by an operator, never committed)

Each environment has its own database: dev uses aevolution on the PostgreSQL VM pg-forge-01. Stage needs a separate database, with its own migrator login and runtime group. Sharing one database would let one environment's runtime login act on the other's regions, because the fenced functions are per database, not per environment.

Each namespace gets its own Secrets, with environment-specific values:

  • harbor-pull: a pull-only Harbor robot (docker-registry type).
  • evolution-postgres-migrator: the migrator/owner login for that environment's database.
  • evolution-postgres-runtime: the runtime login, a member of that environment's NOLOGIN runtime group (ADR-0060).

Both login Secrets have the keys pg_service.conf, pgpass and ca.crt:

  • pg_service.conf has an [evolution] section with host, a numeric hostaddr, port, dbname, user, sslmode=verify-full and sslrootcert=/run/postgres/ca.crt. connect_timeout is allowed. evo-server and evo-world-clone read it with --pg-service and refuse any other key.
  • pgpass is mounted, then copied to private memory before use, because libpq ignores group-readable password files.

aevolution-db (the PG*/POSTGRES_* connection variables) is not used by AEvolution workloads. PGPASSWORD and PGUSER are never placed in a container environment, and the policy check forbids it.

A World0 credential must never exist in the dev or stage namespaces, and no dev or stage credential may reach the World0 database. Rotate by replacing the Secret and restarting the consumer.

Delivery through Argo CD (maintainer preference, 2026-09-29)

Everything except Secrets and one node label is delivered from Git by Argo CD.

Bootstrap once. Argo CD cannot create its own first project and application:

kubectl apply -f deploy/argocd/platform/project.yaml -f deploy/argocd/platform/application.yaml

aevolution-platform then syncs deploy/kubernetes/platform/cluster:

  • the World0 admission policy;
  • the aevolution-local-retain storage class;
  • each environment's namespace, quota, limit range, default-deny policies, service accounts and object claim (dev and stage).

It self-heals, so a deleted quota, policy or label comes back. It never prunes, and every object carries Prune=false,Delete=false, so removing a file or deleting the Application never deletes a namespace, claim, class or policy. The admission policy also denies deleting the platform Application or project. Objects applied by hand earlier are adopted (server-side apply).

Workloads. Each environment's Application syncs its path, currently manually:

  • aevolution-dev syncs deploy/kubernetes/dev, which is live.
  • Stage uses deploy/argocd/stage, whose path is resolved when its values are supplied.

The store Jobs (deploy/kubernetes/stores/<env>) are included in the environment path as sync hooks:

  • the database migration runs as a PreSync hook, so a failed migration stops the sync;
  • check-store runs as a PostSync hook.

An environment's stores overlay is included only once it has its published image digests and its runtime group role. The policy check refuses any REPLACE_ marker in a synced path.

Outside Argo (by design, for now).

  • Secrets are created by an operator; a secret manager is in docs/BACKLOG.md.
  • One node label: kubectl label node rke2-worker-1 aevolution.io/stores=true. The maintainer allows any worker; worker-1 is chosen because it had the lowest measured latency, and worker-3's disk is 87 % full.

Verify: bash tools/ops/verify-isolation.sh makes only reads, can-i queries and server dry runs.

Stores (M8.3, ADR-0070)

Once deploy/kubernetes/stores/dev has its values and is included in deploy/kubernetes/dev, a manual sync of aevolution-dev runs the migration and then the store check. The hook Jobs keep their logs until the next sync. Record this evidence in docs/engineering/VALIDATION.md:

  • Pod restart: sync again. The check must print probe verified with the same digest.
  • Privilege boundary: every run prints database tables=denied functions=reachable.
  • Missing volume: temporarily remove the objects volume in a branch, or run the image by hand without it. The check must exit 4 with objects refused missing-or-not-a-directory.
  • Full volume: add --reserve-mib 1048576 and a new probe region, such as --region 2. The check must exit 4 with probe refused full.
  • Retention: delete the claim by hand, since Argo never prunes it. The PV must become Released, not deleted. To rebind:
    • clear the PV's spec.claimRef;
    • let the platform app recreate the claim, with volumeName pinned in Git;
    • sync, and check that it prints probe verified.
  • Credential rotation:
    • Change the runtime login's password on the database VM and sync before updating the Secret. The check must exit 3 with database refused connect.
    • Update the Secret's pgpass and sync again. It must exit 0.

Region host (M8.4, ADR-0071)

deploy/kubernetes/<env> includes stores/<env> and region/<env>. A sync runs:

  1. the database egress policy (wave −2);
  2. the migration hook (wave −1);
  3. the provisioning hook (wave 0, idempotent);
  4. the evo-region Deployment and evo-region-ops Service (wave 1);
  5. the store check (PostSync).

Dev is world 100, region 1; stage is world 200, region 1. The server image digest in region/<env> is also the build identity.

Before the first region sync, apply deploy/argocd/dev/project.yaml (it allows Deployment and Service), and label the store node.

Evidence: bash tools/ops/region-drill.sh dev (dev or stage only; World0 is refused). It records the time to Ready after a rollout restart and after a forced pod deletion, and checks that the new epoch resumed at the committed tick. Also record:

  • the Argo sync result;
  • the first pod's phase running epoch=... committed=... line;
  • that /readyz stays 503 while a replacement waits out the lease.

Monitoring (M8.5, ADR-0072)

Bootstrap once: kubectl apply -f deploy/argocd/monitoring/, then sync aevolution-platform (it adds the aevolution-monitoring namespace and its guardrails) and aevolution-monitoring. Re-sync dev and stage, which adds the evo-region-scrape policy.

kubectl -n aevolution-monitoring port-forward svc/prometheus 9090 &
curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"'  # both regions up
curl -s localhost:9090/api/v1/alerts                                  # Watchdog firing only

Injection in the cluster (dev only):

  • kubectl -n evolution-dev scale deploy/evo-region --replicas=0 makes EvoRegionDown pending, then firing after 2 minutes, in Alertmanager (null receiver).
  • Scale back to 1 and it resolves.

Record the timestamps. The rule tests in CI already prove the thresholds.

Fault drills (M8.6, ADR-0073)

bash tools/ops/fault-drill.sh dev all enables synthetic load, then runs three forced kills, a full volume and an empty volume. It restores the Git settings on exit.

For the database outage, run bash tools/ops/fault-drill.sh dev db-outage and follow its prompt. On pg-forge-01, block the store node for at least 30 s, then lift the block:

sudo iptables -I INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP   # rke2-worker-1
sleep 40
sudo iptables -D INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP

Record each kill, full, empty and db-outage line in VALIDATION.md. During the drill, Prometheus should show:

  • EvoRegionObjectsFull (full);
  • EvoRegionQuarantined or EvoRegionNotReady (empty);
  • EvoRegionFenced (db-outage).

Keep drills short: synthetic load leaves permanent history.

Break-glass

How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is, nobody holds the group, so World0 deletions are impossible short of removing the policy.

To delete World0 data or identity on purpose:

  1. Record the operation and its approval.
  2. Use a credential whose groups include aevolution:world0-break-glass, for example a short-lived client certificate with that organization.
  3. Take and verify a backup first (M8.7; see "Backups" below).

Removing the admission policy itself is the break-glass of last resort. It needs cluster-admin and must be recorded the same way. Never test by deleting World0: the verify script uses server-side dry runs.

Residual risks

  • default AppProject. It still allows every destination, so its applications (other projects, two with automated prune) can create objects in AEvolution namespaces. Admission denies only World0 deletions. The maintainer accepted this risk on 2026-09-29 and is leaving default as it is (docs/BACKLOG.md).
  • World0 volumes. local-path provisions volumes with Delete. Patch each World0 PV to Retain when it is bound, and the policy then keeps it there. Volumes stay node-local until M8.3.

Backups (M8.7, ADR-0074)

Dev and stage only; the tools refuse World0. From a machine with kubectl access that is not the hypervisor host:

# 1. Sync the environment's application once so the egress policy admits the
#    backup component and the store overlay pins a postgres-tools image that
#    contains pg_dump (any image built after ADR-0074).
# 2. Back up to a local directory (the destination until one is chosen):
tools/backup/cluster-backup.sh dev --to /path/to/backups
# 3. Restore-test it on independent storage and rebuild it as isolated world 900
#    (WSL: the pinned PostgreSQL build and the Linux PostgreSQL build of the tools).
PATH="$PWD/.cache/postgres/install/bin:$PATH" \
LD_LIBRARY_PATH="$PWD/.cache/postgres/install/lib" \
EVO_WORLD_CLONE_BIN="$PWD/build/postgres-linux/evo-world-clone" \
EVO_SERVER_BIN="$PWD/build/postgres-linux/evo-server" \
  tools/backup/verify-backup.sh /path/to/backups/dev-<time> --isolate 900

Only verify-backup.sh output that ends in PASS restore verified and world ... rebuilt as isolated world ... counts as restore evidence. The restore uses fresh roles on a throwaway cluster; nothing reaches the source database. A restored copy keeps the source world identity and is never served; restoring in place after host loss is M8.8.

Upgrades and nodes (M8.9, ADR-0076)

Build upgrade (same rules).

  1. Rehearse on clones first: clone the cut twice into an isolated database, run the current and candidate builds with --synthetic-inputs development-v1 --synthetic-until T --accelerate 20, then evo-world-clone compare must exit 0.
  2. Pin the new server digest in deploy/kubernetes/region/<env> (image and REGION_BUILD), add the old digest to REGION_RETAINED, and sync. The migrate hook (wave -1) stops the sync on failure, leaving the old host running.
  3. When the new pod is Ready: tools/ops/upgrade-barrier.sh <env> --retire OLD. Keep OLD in REGION_RETAINED: its history stays recoverable.

Adding a node. Label it aevolution.io/compute=true and run tools/ops/scheduling-check.sh <env>. Only stateless pods (Alertmanager today) may move there. Never move aevolution.io/stores to rebalance: the objects volume stays on its node; a region moves only by restore or clone onto the new node's storage.