105 lines
5.3 KiB
Markdown
105 lines
5.3 KiB
Markdown
[Reviewed]
|
||
# ADR-0071: Region host deployment in dev and stage
|
||
|
||
## Status
|
||
Accepted for M8.4, 2026-09-29. It builds on:
|
||
- ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
|
||
- ADR-0068 (build-bound authority);
|
||
- ADR-0069 and ADR-0070 (environments, stores and Argo delivery).
|
||
|
||
It amends the M8.4 roadmap row: there is **no director** yet.
|
||
|
||
## Context
|
||
M8.4 asks for:
|
||
- the host deployed through the existing GitOps path, with an immutable digest;
|
||
- measured resources, probes, a security context and a drain budget;
|
||
- evidence that a region is ready only after lease and recovery;
|
||
- a graceful rollout, and forced deletion of a worker.
|
||
|
||
The named risk is a successful rollout hiding a failed region recovery. Regions
|
||
could so far only be provisioned from test code.
|
||
|
||
M7.3 deferred the director because it would hold no authority, and would be a
|
||
placeholder. With one region per environment, the assignment is static and lives
|
||
in Git.
|
||
|
||
## Decision
|
||
**Deployment `evo-region`** (`deploy/kubernetes/region`), one per environment.
|
||
- **Rollout.** `replicas: 1` with the **Recreate** strategy. A rolling update would
|
||
start a pod that cannot become ready while the old one holds the lease, and the
|
||
old one is never removed. Recreate drains the old pod (`--drain-ms 20000`,
|
||
inside a 45 s grace period) before the replacement starts. The replacement then
|
||
acquires, recovers, verifies the tail, and only then becomes ready.
|
||
- **Probes.** Readiness is `/readyz` (current lease and verified head). Liveness is
|
||
`/livez` only, so a process waiting out a lease is never restarted for being
|
||
unready.
|
||
- **Resources.** These come from the measured profile: a 64 Mi / 50m request and a
|
||
256 Mi memory limit. There is **no CPU limit**, because throttling would delay
|
||
renewals. The production lease policy and the 1024 MiB object reserve apply.
|
||
- **Security.** Pod security `restricted`, a read-only root filesystem, no token,
|
||
and no capabilities.
|
||
- **Placement and access.** Pinned to the store node (`aevolution.io/stores=true`).
|
||
Its object claim is node-local, and the replacement lands where the data is.
|
||
- **Credentials.** The runtime login comes from its service file. The password file
|
||
is copied to private memory, and `exec` makes `evo-server` PID 1, so SIGTERM
|
||
reaches it.
|
||
- **Service.** A ClusterIP Service `evo-region-ops` on port 9100. Scraping is M8.5.
|
||
- **Identity.** Dev is world 100, region 1; stage is world 200, region 1. Both use
|
||
frame 9 and the development opening.
|
||
|
||
**Build identity is the image digest.** Each overlay sets the server image digest
|
||
twice: as the image, and as `--build`, the identity recorded in every commit. The
|
||
policy check requires them to be equal. An image upgrade is therefore also a
|
||
visible build change, handled by the ADR-0068 barrier and `--retained`.
|
||
|
||
**Provisioning.** `evo-world-clone provision` runs as a Sync hook (wave 0) with the
|
||
migrator login. The order is:
|
||
1. the database egress policy (wave −2);
|
||
2. the migration (wave −1);
|
||
3. provisioning (wave 0);
|
||
4. the Deployment and Service (wave 1);
|
||
5. the PostSync store check.
|
||
|
||
Provisioning records the digest in `retained_builds`, then creates genesis from
|
||
`development-v1` only when the region does not exist. A region that exists, in any
|
||
state, is reported and never touched, so every sync can run it. The opening is
|
||
`evo::processes::development_opening`, byte-identical to the qualification fixture
|
||
(proved by `lineage-region-contracts`). World0's opening remains a separate,
|
||
reviewed decision.
|
||
|
||
**Evidence tooling.** `tools/ops/region-drill.sh <dev|stage>` (World0 refused)
|
||
reads epoch and committed tick from the pod's own `/metrics`, then:
|
||
- performs `rollout restart`, and times the replacement to Ready;
|
||
- force-deletes the pod, and times the replacement to Ready.
|
||
|
||
After each step, the replacement must report a newer epoch at a committed tick
|
||
that is no lower than before.
|
||
|
||
## Consequences
|
||
- No inputs reach the region until a gateway exists. The host runs idle, holding
|
||
and renewing its lease, and tick debt accrues visibly. Commit-level fault
|
||
evidence under load is M8.6.
|
||
- A forced deletion's replacement waits out up to 20 s of lease. That is correct,
|
||
not a fault.
|
||
- Losing the store node leaves the region unavailable until the node returns
|
||
(ADR-0070); it is never re-created empty.
|
||
- Each environment's AppProject must allow Deployment and Service.
|
||
`deploy/argocd/dev/project.yaml` does, and supersedes the old example.
|
||
|
||
## Alternatives rejected
|
||
- **RollingUpdate with maxSurge.** It deadlocks on the lease, as described above.
|
||
- **A StatefulSet.** It adds no guarantee the lease does not already provide.
|
||
- **A director.** With static single-region assignment, it would be a placeholder
|
||
with no authority.
|
||
|
||
## Amendment: retained builds across image upgrades (2026-09-30)
|
||
Each commit records the build that wrote it, and recovery accepts only builds on
|
||
the host's allowlist. After an image upgrade, a region's existing head was
|
||
therefore written by the previous build. Without `--retained <old build>` the new
|
||
host would refuse its own history as unsupported and quarantine. `region/<env>`
|
||
now lists earlier builds in `REGION_RETAINED`, passed as `--retained` flags; the
|
||
policy check requires 64-hex entries distinct from the current build. The
|
||
provisioning hook records each new build in `retained_builds` before the host
|
||
starts. Retiring an old build from publication is a separate, deliberate ADR-0068
|
||
barrier.
|