5.3 KiB
[Reviewed]
ADR-0071: Region host deployment in dev and stage
Status
Accepted for M8.4, 2026-09-29. It builds on:
- ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
- ADR-0068 (build-bound authority);
- ADR-0069 and ADR-0070 (environments, stores and Argo delivery).
It amends the M8.4 roadmap row: there is no director yet.
Context
M8.4 asks for:
- the host deployed through the existing GitOps path, with an immutable digest;
- measured resources, probes, a security context and a drain budget;
- evidence that a region is ready only after lease and recovery;
- a graceful rollout, and forced deletion of a worker.
The named risk is a successful rollout hiding a failed region recovery. Regions could so far only be provisioned from test code.
M7.3 deferred the director because it would hold no authority, and would be a placeholder. With one region per environment, the assignment is static and lives in Git.
Decision
Deployment evo-region (deploy/kubernetes/region), one per environment.
- Rollout.
replicas: 1with the Recreate strategy. A rolling update would start a pod that cannot become ready while the old one holds the lease, and the old one is never removed. Recreate drains the old pod (--drain-ms 20000, inside a 45 s grace period) before the replacement starts. The replacement then acquires, recovers, verifies the tail, and only then becomes ready. - Probes. Readiness is
/readyz(current lease and verified head). Liveness is/livezonly, so a process waiting out a lease is never restarted for being unready. - Resources. These come from the measured profile: a 64 Mi / 50m request and a 256 Mi memory limit. There is no CPU limit, because throttling would delay renewals. The production lease policy and the 1024 MiB object reserve apply.
- Security. Pod security
restricted, a read-only root filesystem, no token, and no capabilities. - Placement and access. Pinned to the store node (
aevolution.io/stores=true). Its object claim is node-local, and the replacement lands where the data is. - Credentials. The runtime login comes from its service file. The password file
is copied to private memory, and
execmakesevo-serverPID 1, so SIGTERM reaches it. - Service. A ClusterIP Service
evo-region-opson port 9100. Scraping is M8.5. - Identity. Dev is world 100, region 1; stage is world 200, region 1. Both use frame 9 and the development opening.
Build identity is the image digest. Each overlay sets the server image digest
twice: as the image, and as --build, the identity recorded in every commit. The
policy check requires them to be equal. An image upgrade is therefore also a
visible build change, handled by the ADR-0068 barrier and --retained.
Provisioning. evo-world-clone provision runs as a Sync hook (wave 0) with the
migrator login. The order is:
- the database egress policy (wave −2);
- the migration (wave −1);
- provisioning (wave 0);
- the Deployment and Service (wave 1);
- the PostSync store check.
Provisioning records the digest in retained_builds, then creates genesis from
development-v1 only when the region does not exist. A region that exists, in any
state, is reported and never touched, so every sync can run it. The opening is
evo::processes::development_opening, byte-identical to the qualification fixture
(proved by lineage-region-contracts). World0's opening remains a separate,
reviewed decision.
Evidence tooling. tools/ops/region-drill.sh <dev|stage> (World0 refused)
reads epoch and committed tick from the pod's own /metrics, then:
- performs
rollout restart, and times the replacement to Ready; - force-deletes the pod, and times the replacement to Ready.
After each step, the replacement must report a newer epoch at a committed tick that is no lower than before.
Consequences
- No inputs reach the region until a gateway exists. The host runs idle, holding and renewing its lease, and tick debt accrues visibly. Commit-level fault evidence under load is M8.6.
- A forced deletion's replacement waits out up to 20 s of lease. That is correct, not a fault.
- Losing the store node leaves the region unavailable until the node returns (ADR-0070); it is never re-created empty.
- Each environment's AppProject must allow Deployment and Service.
deploy/argocd/dev/project.yamldoes, and supersedes the old example.
Alternatives rejected
- RollingUpdate with maxSurge. It deadlocks on the lease, as described above.
- A StatefulSet. It adds no guarantee the lease does not already provide.
- A director. With static single-region assignment, it would be a placeholder with no authority.
Amendment: retained builds across image upgrades (2026-09-30)
Each commit records the build that wrote it, and recovery accepts only builds on
the host's allowlist. After an image upgrade, a region's existing head was
therefore written by the previous build. Without --retained <old build> the new
host would refuse its own history as unsupported and quarantine. region/<env>
now lists earlier builds in REGION_RETAINED, passed as --retained flags; the
policy check requires 64-hex entries distinct from the current build. The
provisioning hook records each new build in retained_builds before the host
starts. Retiring an old build from publication is a separate, deliberate ADR-0068
barrier.