Add M8.4 region host deployment, provisioning hook and drill

Deploy evo-region per environment (ADR-0071): one replica with the
Recreate strategy (a rolling update would deadlock on the lease),
/readyz readiness, /livez liveness, no CPU limit and a drain inside the
grace period, with the image digest doubling as the build identity. A
new idempotent evo-world-clone provision subcommand creates genesis from
development_opening (byte-identical to the qualification fixture) as a
Sync hook after the migration. Dev and stage paths include their region;
a dev AppProject allows Deployment and Service; region-drill.sh times
rollout and forced-deletion recovery; the policy check enforces the
host's invariants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 21:32:56 -04:00
co-authored by Claude Opus 5.5
parent 98240ef96e
commit 8ef69fab1d
27 changed files with 879 additions and 159 deletions
+11
View File
@@ -624,3 +624,14 @@ M8.3 implemented and locally qualified (2026-09-29), under ADR-0070:
- `evo-world-clone check-store` for in-cluster qualification.
The in-cluster evidence and the backup destination wait on the maintainer
(docs/BACKLOG.md).
M8.3 delivered through Argo CD by the maintainer (2026-09-29): the dev, stage and
platform applications are synced and connected to PostgreSQL.
M8.4 implemented and locally qualified (2026-09-29), under ADR-0071:
- one `evo-region` host per environment, with the Recreate strategy and the image
digest as its build identity;
- an idempotent provisioning hook from the development opening;
- `region-drill.sh` for rollout and forced-deletion evidence.
The in-cluster sync and drill wait on the maintainer. No director yet: assignment
is static and lives in Git.
+20
View File
@@ -1044,3 +1044,23 @@ M8.3 delivery moved to Argo CD, at the maintainer's request:
- **Store Jobs:** they become sync hooks in each environment's path once their
digests and runtime roles are supplied.
- **Outside Argo:** Secrets and the store node label.
## M8.4 region host (2026-09-29)
The maintainer synced the dev, stage and platform applications through Argo CD,
and they are connected to PostgreSQL, with digests `661e433e…` (server) and
`9abb06ef…` (tools).
ADR-0071 deploys `evo-region` as one replica per environment:
- the Recreate strategy, since a rolling update would deadlock on the lease;
- readiness `/readyz`, liveness `/livez`;
- no CPU limit;
- a 20 s drain inside a 45 s grace period.
Each sync runs, in order: egress, migration, provisioning (idempotent, from the
new `development_opening`, which is byte-identical to the qualification fixture),
the host, then the store check. The image digest is also the build identity, and
the policy check enforces this. Dev is world 100 and stage world 200, both region 1.
Next: apply `deploy/argocd/dev/project.yaml`, sync, and run
`tools/ops/region-drill.sh dev`.
+92
View File
@@ -0,0 +1,92 @@
# ADR-0071: Region host deployment in dev and stage
## Status
Accepted for M8.4, 2026-09-29. It builds on:
- ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
- ADR-0068 (build-bound authority);
- ADR-0069 and ADR-0070 (environments, stores and Argo delivery).
It amends the M8.4 roadmap row: there is **no director** yet.
## Context
M8.4 asks for:
- the host deployed through the existing GitOps path, with an immutable digest;
- measured resources, probes, a security context and a drain budget;
- evidence that a region is ready only after lease and recovery;
- a graceful rollout, and forced deletion of a worker.
The named risk is a successful rollout hiding a failed region recovery. Regions
could so far only be provisioned from test code.
M7.3 deferred the director because it would hold no authority, and would be a
placeholder. With one region per environment, the assignment is static and lives
in Git.
## Decision
**Deployment `evo-region`** (`deploy/kubernetes/region`), one per environment.
- **Rollout.** `replicas: 1` with the **Recreate** strategy. A rolling update would
start a pod that cannot become ready while the old one holds the lease, and the
old one is never removed. Recreate drains the old pod (`--drain-ms 20000`,
inside a 45 s grace period) before the replacement starts. The replacement then
acquires, recovers, verifies the tail, and only then becomes ready.
- **Probes.** Readiness is `/readyz` (current lease and verified head). Liveness is
`/livez` only, so a process waiting out a lease is never restarted for being
unready.
- **Resources.** These come from the measured profile: a 64 Mi / 50m request and a
256 Mi memory limit. There is **no CPU limit**, because throttling would delay
renewals. The production lease policy and the 1024 MiB object reserve apply.
- **Security.** Pod security `restricted`, a read-only root filesystem, no token,
and no capabilities.
- **Placement and access.** Pinned to the store node (`aevolution.io/stores=true`).
Its object claim is node-local, and the replacement lands where the data is.
- **Credentials.** The runtime login comes from its service file. The password file
is copied to private memory, and `exec` makes `evo-server` PID 1, so SIGTERM
reaches it.
- **Service.** A ClusterIP Service `evo-region-ops` on port 9100. Scraping is M8.5.
- **Identity.** Dev is world 100, region 1; stage is world 200, region 1. Both use
frame 9 and the development opening.
**Build identity is the image digest.** Each overlay sets the server image digest
twice: as the image, and as `--build`, the identity recorded in every commit. The
policy check requires them to be equal. An image upgrade is therefore also a
visible build change, handled by the ADR-0068 barrier and `--retained`.
**Provisioning.** `evo-world-clone provision` runs as a Sync hook (wave 0) with the
migrator login. The order is:
1. the database egress policy (wave −2);
2. the migration (wave −1);
3. provisioning (wave 0);
4. the Deployment and Service (wave 1);
5. the PostSync store check.
Provisioning records the digest in `retained_builds`, then creates genesis from
`development-v1` only when the region does not exist. A region that exists, in any
state, is reported and never touched, so every sync can run it. The opening is
`evo::processes::development_opening`, byte-identical to the qualification fixture
(proved by `lineage-region-contracts`). World0's opening remains a separate,
reviewed decision.
**Evidence tooling.** `tools/ops/region-drill.sh <dev|stage>` (World0 refused)
reads epoch and committed tick from the pod's own `/metrics`, then:
- performs `rollout restart`, and times the replacement to Ready;
- force-deletes the pod, and times the replacement to Ready.
After each step, the replacement must report a newer epoch at a committed tick
that is no lower than before.
## Consequences
- No inputs reach the region until a gateway exists. The host runs idle, holding
and renewing its lease, and tick debt accrues visibly. Commit-level fault
evidence under load is M8.6.
- A forced deletion's replacement waits out up to 20 s of lease. That is correct,
not a fault.
- Losing the store node leaves the region unavailable until the node returns
(ADR-0070); it is never re-created empty.
- Each environment's AppProject must allow Deployment and Service.
`deploy/argocd/dev/project.yaml` does, and supersedes the old example.
## Alternatives rejected
- **RollingUpdate with maxSurge.** It deadlocks on the lease, as described above.
- **A StatefulSet.** It adds no guarantee the lease does not already provide.
- **A director.** With static single-region assignment, it would be a placeholder
with no authority.
+1
View File
@@ -81,3 +81,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
- [ADR-0069: Environment isolation and World0 protection](0069-environment-isolation.md) - accepted for M8.2 per-environment guardrails, native World0 admission protection, scoped Argo projects and verification.
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
+30
View File
@@ -2781,3 +2781,33 @@ admission policy and the stage and World0 AppProjects, then ran
- The digests the maintainer pasted are local image IDs from the build step, not
registry digests. The registry digest is printed by the Publish step.
- Not executed: bootstrapping the platform Application, or any Argo sync.
## M8.4 region host - 2026-09-29
- Windows MSVC Release CTest 59/59, including the new opening code.
`lineage-region-contracts` proves `development_opening(7)` is byte-identical
to the fixture, and that world 100 differs only by identity.
- WSL Linux `checks` set: 59/59 in each configuration. The format check passed.
- PostgreSQL suites: Release, analysis and ASan/UBSan passed. Analysis passed after
one optional-access fix in `provision`.
- Provisioning test:
- A runtime login cannot provision.
- The owner provisions `world=7 region=86 sequence=0 tick=0`, and a rerun reports
`already-provisioned`.
- A server then became ready and acknowledged ticks 1–5, after which the tool
reported `sequence=1 tick=5` and left the region untouched.
- Mutation: removing the already-provisioned early return was caught.
- `deploy-policy-check.py`: 50 objects, 0 failures. Four new mutations were each
caught:
- a build identity differing from the image digest;
- a RollingUpdate strategy;
- a CPU limit;
- a grace period shorter than the drain.
The checker first misread overlay patches as whole Deployments; it now checks
only the base.
- Kustomize renders dev and stage in order: egress policy (−2), migration hook
(−1), provisioning hook (0), Deployment and Service (1), then the store check
(PostSync).
- `region-drill.sh` passed `bash -n`, and refuses World0 before kubectl (also
enforced by the policy check).
- Not executed: the in-cluster region sync and the drill.
+23
View File
@@ -116,6 +116,29 @@ this evidence in docs/engineering/VALIDATION.md:
the Secret. The check must exit 3 with `database refused connect`.
- Update the Secret's `pgpass` and sync again. It must exit 0.
## Region host (M8.4, ADR-0071)
`deploy/kubernetes/<env>` includes `stores/<env>` and `region/<env>`. A sync runs:
1. the database egress policy (wave −2);
2. the migration hook (wave −1);
3. the provisioning hook (wave 0, idempotent);
4. the `evo-region` Deployment and `evo-region-ops` Service (wave 1);
5. the store check (PostSync).
Dev is world 100, region 1; stage is world 200, region 1. The server image digest
in `region/<env>` is also the build identity.
Before the first region sync, apply `deploy/argocd/dev/project.yaml` (it allows
Deployment and Service), and label the store node.
Evidence: `bash tools/ops/region-drill.sh dev` (dev or stage only; World0 is
refused). It records the time to Ready after a rollout restart and after a forced
pod deletion, and checks that the new epoch resumed at the committed tick. Also
record:
- the Argo sync result;
- the first pod's `phase running epoch=... committed=...` line;
- that `/readyz` stays 503 while a replacement waits out the lease.
## Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,