Files
aevolution/docs/adr/0071-region-host-deployment.md
T
dingoandClaude Opus 5.5 4ef0fe8167 Keep earlier builds recoverable across region image upgrades
Complete the maintainer's server image bump to edb39f by also setting
the region build identity, and add REGION_RETAINED so the host keeps
the builds that wrote its history on its recovery allowlist; without it
the upgraded host would quarantine its own head as unsupported. Record
the M8.4 drill (rollout 3.0 s, forced deletion 19.3 s to Ready, epochs
3 to 5) and the maintainer's M8.5 sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:34:03 -04:00

5.3 KiB
Raw Blame History

ADR-0071: Region host deployment in dev and stage

Status

Accepted for M8.4, 2026-09-29. It builds on:

  • ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
  • ADR-0068 (build-bound authority);
  • ADR-0069 and ADR-0070 (environments, stores and Argo delivery).

It amends the M8.4 roadmap row: there is no director yet.

Context

M8.4 asks for:

  • the host deployed through the existing GitOps path, with an immutable digest;
  • measured resources, probes, a security context and a drain budget;
  • evidence that a region is ready only after lease and recovery;
  • a graceful rollout, and forced deletion of a worker.

The named risk is a successful rollout hiding a failed region recovery. Regions could so far only be provisioned from test code.

M7.3 deferred the director because it would hold no authority, and would be a placeholder. With one region per environment, the assignment is static and lives in Git.

Decision

Deployment evo-region (deploy/kubernetes/region), one per environment.

  • Rollout. replicas: 1 with the Recreate strategy. A rolling update would start a pod that cannot become ready while the old one holds the lease, and the old one is never removed. Recreate drains the old pod (--drain-ms 20000, inside a 45 s grace period) before the replacement starts. The replacement then acquires, recovers, verifies the tail, and only then becomes ready.
  • Probes. Readiness is /readyz (current lease and verified head). Liveness is /livez only, so a process waiting out a lease is never restarted for being unready.
  • Resources. These come from the measured profile: a 64 Mi / 50m request and a 256 Mi memory limit. There is no CPU limit, because throttling would delay renewals. The production lease policy and the 1024 MiB object reserve apply.
  • Security. Pod security restricted, a read-only root filesystem, no token, and no capabilities.
  • Placement and access. Pinned to the store node (aevolution.io/stores=true). Its object claim is node-local, and the replacement lands where the data is.
  • Credentials. The runtime login comes from its service file. The password file is copied to private memory, and exec makes evo-server PID 1, so SIGTERM reaches it.
  • Service. A ClusterIP Service evo-region-ops on port 9100. Scraping is M8.5.
  • Identity. Dev is world 100, region 1; stage is world 200, region 1. Both use frame 9 and the development opening.

Build identity is the image digest. Each overlay sets the server image digest twice: as the image, and as --build, the identity recorded in every commit. The policy check requires them to be equal. An image upgrade is therefore also a visible build change, handled by the ADR-0068 barrier and --retained.

Provisioning. evo-world-clone provision runs as a Sync hook (wave 0) with the migrator login. The order is:

  1. the database egress policy (wave −2);
  2. the migration (wave −1);
  3. provisioning (wave 0);
  4. the Deployment and Service (wave 1);
  5. the PostSync store check.

Provisioning records the digest in retained_builds, then creates genesis from development-v1 only when the region does not exist. A region that exists, in any state, is reported and never touched, so every sync can run it. The opening is evo::processes::development_opening, byte-identical to the qualification fixture (proved by lineage-region-contracts). World0's opening remains a separate, reviewed decision.

Evidence tooling. tools/ops/region-drill.sh <dev|stage> (World0 refused) reads epoch and committed tick from the pod's own /metrics, then:

  • performs rollout restart, and times the replacement to Ready;
  • force-deletes the pod, and times the replacement to Ready.

After each step, the replacement must report a newer epoch at a committed tick that is no lower than before.

Consequences

  • No inputs reach the region until a gateway exists. The host runs idle, holding and renewing its lease, and tick debt accrues visibly. Commit-level fault evidence under load is M8.6.
  • A forced deletion's replacement waits out up to 20 s of lease. That is correct, not a fault.
  • Losing the store node leaves the region unavailable until the node returns (ADR-0070); it is never re-created empty.
  • Each environment's AppProject must allow Deployment and Service. deploy/argocd/dev/project.yaml does, and supersedes the old example.

Alternatives rejected

  • RollingUpdate with maxSurge. It deadlocks on the lease, as described above.
  • A StatefulSet. It adds no guarantee the lease does not already provide.
  • A director. With static single-region assignment, it would be a placeholder with no authority.

Amendment: retained builds across image upgrades (2026-09-30)

Each commit records the build that wrote it, and recovery accepts only builds on the host's allowlist. After an image upgrade, a region's existing head was therefore written by the previous build. Without --retained <old build> the new host would refuse its own history as unsupported and quarantine. region/<env> now lists earlier builds in REGION_RETAINED, passed as --retained flags; the policy check requires 64-hex entries distinct from the current build. The provisioning hook records each new build in retained_builds before the host starts. Retiring an old build from publication is a separate, deliberate ADR-0068 barrier.