Files
aevolution/docs/adr/0071-region-host-deployment.md
T
2026-10-08 12:08:53 -04:00

105 lines
5.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
[Reviewed]
# ADR-0071: Region host deployment in dev and stage
## Status
Accepted for M8.4, 2026-09-29. It builds on:
- ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
- ADR-0068 (build-bound authority);
- ADR-0069 and ADR-0070 (environments, stores and Argo delivery).
It amends the M8.4 roadmap row: there is **no director** yet.
## Context
M8.4 asks for:
- the host deployed through the existing GitOps path, with an immutable digest;
- measured resources, probes, a security context and a drain budget;
- evidence that a region is ready only after lease and recovery;
- a graceful rollout, and forced deletion of a worker.
The named risk is a successful rollout hiding a failed region recovery. Regions
could so far only be provisioned from test code.
M7.3 deferred the director because it would hold no authority, and would be a
placeholder. With one region per environment, the assignment is static and lives
in Git.
## Decision
**Deployment `evo-region`** (`deploy/kubernetes/region`), one per environment.
- **Rollout.** `replicas: 1` with the **Recreate** strategy. A rolling update would
start a pod that cannot become ready while the old one holds the lease, and the
old one is never removed. Recreate drains the old pod (`--drain-ms 20000`,
inside a 45 s grace period) before the replacement starts. The replacement then
acquires, recovers, verifies the tail, and only then becomes ready.
- **Probes.** Readiness is `/readyz` (current lease and verified head). Liveness is
`/livez` only, so a process waiting out a lease is never restarted for being
unready.
- **Resources.** These come from the measured profile: a 64 Mi / 50m request and a
256 Mi memory limit. There is **no CPU limit**, because throttling would delay
renewals. The production lease policy and the 1024 MiB object reserve apply.
- **Security.** Pod security `restricted`, a read-only root filesystem, no token,
and no capabilities.
- **Placement and access.** Pinned to the store node (`aevolution.io/stores=true`).
Its object claim is node-local, and the replacement lands where the data is.
- **Credentials.** The runtime login comes from its service file. The password file
is copied to private memory, and `exec` makes `evo-server` PID 1, so SIGTERM
reaches it.
- **Service.** A ClusterIP Service `evo-region-ops` on port 9100. Scraping is M8.5.
- **Identity.** Dev is world 100, region 1; stage is world 200, region 1. Both use
frame 9 and the development opening.
**Build identity is the image digest.** Each overlay sets the server image digest
twice: as the image, and as `--build`, the identity recorded in every commit. The
policy check requires them to be equal. An image upgrade is therefore also a
visible build change, handled by the ADR-0068 barrier and `--retained`.
**Provisioning.** `evo-world-clone provision` runs as a Sync hook (wave 0) with the
migrator login. The order is:
1. the database egress policy (wave −2);
2. the migration (wave −1);
3. provisioning (wave 0);
4. the Deployment and Service (wave 1);
5. the PostSync store check.
Provisioning records the digest in `retained_builds`, then creates genesis from
`development-v1` only when the region does not exist. A region that exists, in any
state, is reported and never touched, so every sync can run it. The opening is
`evo::processes::development_opening`, byte-identical to the qualification fixture
(proved by `lineage-region-contracts`). World0's opening remains a separate,
reviewed decision.
**Evidence tooling.** `tools/ops/region-drill.sh <dev|stage>` (World0 refused)
reads epoch and committed tick from the pod's own `/metrics`, then:
- performs `rollout restart`, and times the replacement to Ready;
- force-deletes the pod, and times the replacement to Ready.
After each step, the replacement must report a newer epoch at a committed tick
that is no lower than before.
## Consequences
- No inputs reach the region until a gateway exists. The host runs idle, holding
and renewing its lease, and tick debt accrues visibly. Commit-level fault
evidence under load is M8.6.
- A forced deletion's replacement waits out up to 20 s of lease. That is correct,
not a fault.
- Losing the store node leaves the region unavailable until the node returns
(ADR-0070); it is never re-created empty.
- Each environment's AppProject must allow Deployment and Service.
`deploy/argocd/dev/project.yaml` does, and supersedes the old example.
## Alternatives rejected
- **RollingUpdate with maxSurge.** It deadlocks on the lease, as described above.
- **A StatefulSet.** It adds no guarantee the lease does not already provide.
- **A director.** With static single-region assignment, it would be a placeholder
with no authority.
## Amendment: retained builds across image upgrades (2026-09-30)
Each commit records the build that wrote it, and recovery accepts only builds on
the host's allowlist. After an image upgrade, a region's existing head was
therefore written by the previous build. Without `--retained <old build>` the new
host would refuse its own history as unsupported and quarantine. `region/<env>`
now lists earlier builds in `REGION_RETAINED`, passed as `--retained` flags; the
policy check requires 64-hex entries distinct from the current build. The
provisioning hook records each new build in `retained_builds` before the host
starts. Retiring an old build from publication is a separate, deliberate ADR-0068
barrier.