Add M8.4 region host deployment, provisioning hook and drill
Deploy evo-region per environment (ADR-0071): one replica with the Recreate strategy (a rolling update would deadlock on the lease), /readyz readiness, /livez liveness, no CPU limit and a drain inside the grace period, with the image digest doubling as the build identity. A new idempotent evo-world-clone provision subcommand creates genesis from development_opening (byte-identical to the qualification fixture) as a Sync hook after the migration. Dev and stage paths include their region; a dev AppProject allows Deployment and Service; region-drill.sh times rollout and forced-deletion recovery; the policy check enforces the host's invariants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -624,3 +624,14 @@ M8.3 implemented and locally qualified (2026-09-29), under ADR-0070:
|
||||
- `evo-world-clone check-store` for in-cluster qualification.
|
||||
The in-cluster evidence and the backup destination wait on the maintainer
|
||||
(docs/BACKLOG.md).
|
||||
|
||||
M8.3 delivered through Argo CD by the maintainer (2026-09-29): the dev, stage and
|
||||
platform applications are synced and connected to PostgreSQL.
|
||||
|
||||
M8.4 implemented and locally qualified (2026-09-29), under ADR-0071:
|
||||
- one `evo-region` host per environment, with the Recreate strategy and the image
|
||||
digest as its build identity;
|
||||
- an idempotent provisioning hook from the development opening;
|
||||
- `region-drill.sh` for rollout and forced-deletion evidence.
|
||||
The in-cluster sync and drill wait on the maintainer. No director yet: assignment
|
||||
is static and lives in Git.
|
||||
|
||||
@@ -1044,3 +1044,23 @@ M8.3 delivery moved to Argo CD, at the maintainer's request:
|
||||
- **Store Jobs:** they become sync hooks in each environment's path once their
|
||||
digests and runtime roles are supplied.
|
||||
- **Outside Argo:** Secrets and the store node label.
|
||||
|
||||
## M8.4 region host (2026-09-29)
|
||||
|
||||
The maintainer synced the dev, stage and platform applications through Argo CD,
|
||||
and they are connected to PostgreSQL, with digests `661e433e…` (server) and
|
||||
`9abb06ef…` (tools).
|
||||
|
||||
ADR-0071 deploys `evo-region` as one replica per environment:
|
||||
- the Recreate strategy, since a rolling update would deadlock on the lease;
|
||||
- readiness `/readyz`, liveness `/livez`;
|
||||
- no CPU limit;
|
||||
- a 20 s drain inside a 45 s grace period.
|
||||
|
||||
Each sync runs, in order: egress, migration, provisioning (idempotent, from the
|
||||
new `development_opening`, which is byte-identical to the qualification fixture),
|
||||
the host, then the store check. The image digest is also the build identity, and
|
||||
the policy check enforces this. Dev is world 100 and stage world 200, both region 1.
|
||||
|
||||
Next: apply `deploy/argocd/dev/project.yaml`, sync, and run
|
||||
`tools/ops/region-drill.sh dev`.
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# ADR-0071: Region host deployment in dev and stage
|
||||
|
||||
## Status
|
||||
Accepted for M8.4, 2026-09-29. It builds on:
|
||||
- ADR-0064 to ADR-0066 (host, recovery and operations endpoints);
|
||||
- ADR-0068 (build-bound authority);
|
||||
- ADR-0069 and ADR-0070 (environments, stores and Argo delivery).
|
||||
|
||||
It amends the M8.4 roadmap row: there is **no director** yet.
|
||||
|
||||
## Context
|
||||
M8.4 asks for:
|
||||
- the host deployed through the existing GitOps path, with an immutable digest;
|
||||
- measured resources, probes, a security context and a drain budget;
|
||||
- evidence that a region is ready only after lease and recovery;
|
||||
- a graceful rollout, and forced deletion of a worker.
|
||||
|
||||
The named risk is a successful rollout hiding a failed region recovery. Regions
|
||||
could so far only be provisioned from test code.
|
||||
|
||||
M7.3 deferred the director because it would hold no authority, and would be a
|
||||
placeholder. With one region per environment, the assignment is static and lives
|
||||
in Git.
|
||||
|
||||
## Decision
|
||||
**Deployment `evo-region`** (`deploy/kubernetes/region`), one per environment.
|
||||
- **Rollout.** `replicas: 1` with the **Recreate** strategy. A rolling update would
|
||||
start a pod that cannot become ready while the old one holds the lease, and the
|
||||
old one is never removed. Recreate drains the old pod (`--drain-ms 20000`,
|
||||
inside a 45 s grace period) before the replacement starts. The replacement then
|
||||
acquires, recovers, verifies the tail, and only then becomes ready.
|
||||
- **Probes.** Readiness is `/readyz` (current lease and verified head). Liveness is
|
||||
`/livez` only, so a process waiting out a lease is never restarted for being
|
||||
unready.
|
||||
- **Resources.** These come from the measured profile: a 64 Mi / 50m request and a
|
||||
256 Mi memory limit. There is **no CPU limit**, because throttling would delay
|
||||
renewals. The production lease policy and the 1024 MiB object reserve apply.
|
||||
- **Security.** Pod security `restricted`, a read-only root filesystem, no token,
|
||||
and no capabilities.
|
||||
- **Placement and access.** Pinned to the store node (`aevolution.io/stores=true`).
|
||||
Its object claim is node-local, and the replacement lands where the data is.
|
||||
- **Credentials.** The runtime login comes from its service file. The password file
|
||||
is copied to private memory, and `exec` makes `evo-server` PID 1, so SIGTERM
|
||||
reaches it.
|
||||
- **Service.** A ClusterIP Service `evo-region-ops` on port 9100. Scraping is M8.5.
|
||||
- **Identity.** Dev is world 100, region 1; stage is world 200, region 1. Both use
|
||||
frame 9 and the development opening.
|
||||
|
||||
**Build identity is the image digest.** Each overlay sets the server image digest
|
||||
twice: as the image, and as `--build`, the identity recorded in every commit. The
|
||||
policy check requires them to be equal. An image upgrade is therefore also a
|
||||
visible build change, handled by the ADR-0068 barrier and `--retained`.
|
||||
|
||||
**Provisioning.** `evo-world-clone provision` runs as a Sync hook (wave 0) with the
|
||||
migrator login. The order is:
|
||||
1. the database egress policy (wave −2);
|
||||
2. the migration (wave −1);
|
||||
3. provisioning (wave 0);
|
||||
4. the Deployment and Service (wave 1);
|
||||
5. the PostSync store check.
|
||||
|
||||
Provisioning records the digest in `retained_builds`, then creates genesis from
|
||||
`development-v1` only when the region does not exist. A region that exists, in any
|
||||
state, is reported and never touched, so every sync can run it. The opening is
|
||||
`evo::processes::development_opening`, byte-identical to the qualification fixture
|
||||
(proved by `lineage-region-contracts`). World0's opening remains a separate,
|
||||
reviewed decision.
|
||||
|
||||
**Evidence tooling.** `tools/ops/region-drill.sh <dev|stage>` (World0 refused)
|
||||
reads epoch and committed tick from the pod's own `/metrics`, then:
|
||||
- performs `rollout restart`, and times the replacement to Ready;
|
||||
- force-deletes the pod, and times the replacement to Ready.
|
||||
|
||||
After each step, the replacement must report a newer epoch at a committed tick
|
||||
that is no lower than before.
|
||||
|
||||
## Consequences
|
||||
- No inputs reach the region until a gateway exists. The host runs idle, holding
|
||||
and renewing its lease, and tick debt accrues visibly. Commit-level fault
|
||||
evidence under load is M8.6.
|
||||
- A forced deletion's replacement waits out up to 20 s of lease. That is correct,
|
||||
not a fault.
|
||||
- Losing the store node leaves the region unavailable until the node returns
|
||||
(ADR-0070); it is never re-created empty.
|
||||
- Each environment's AppProject must allow Deployment and Service.
|
||||
`deploy/argocd/dev/project.yaml` does, and supersedes the old example.
|
||||
|
||||
## Alternatives rejected
|
||||
- **RollingUpdate with maxSurge.** It deadlocks on the lease, as described above.
|
||||
- **A StatefulSet.** It adds no guarantee the lease does not already provide.
|
||||
- **A director.** With static single-region assignment, it would be a placeholder
|
||||
with no authority.
|
||||
@@ -81,3 +81,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
|
||||
- [ADR-0069: Environment isolation and World0 protection](0069-environment-isolation.md) - accepted for M8.2 per-environment guardrails, native World0 admission protection, scoped Argo projects and verification.
|
||||
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
|
||||
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
|
||||
|
||||
@@ -2781,3 +2781,33 @@ admission policy and the stage and World0 AppProjects, then ran
|
||||
- The digests the maintainer pasted are local image IDs from the build step, not
|
||||
registry digests. The registry digest is printed by the Publish step.
|
||||
- Not executed: bootstrapping the platform Application, or any Argo sync.
|
||||
|
||||
## M8.4 region host - 2026-09-29
|
||||
|
||||
- Windows MSVC Release CTest 59/59, including the new opening code.
|
||||
`lineage-region-contracts` proves `development_opening(7)` is byte-identical
|
||||
to the fixture, and that world 100 differs only by identity.
|
||||
- WSL Linux `checks` set: 59/59 in each configuration. The format check passed.
|
||||
- PostgreSQL suites: Release, analysis and ASan/UBSan passed. Analysis passed after
|
||||
one optional-access fix in `provision`.
|
||||
- Provisioning test:
|
||||
- A runtime login cannot provision.
|
||||
- The owner provisions `world=7 region=86 sequence=0 tick=0`, and a rerun reports
|
||||
`already-provisioned`.
|
||||
- A server then became ready and acknowledged ticks 1–5, after which the tool
|
||||
reported `sequence=1 tick=5` and left the region untouched.
|
||||
- Mutation: removing the already-provisioned early return was caught.
|
||||
- `deploy-policy-check.py`: 50 objects, 0 failures. Four new mutations were each
|
||||
caught:
|
||||
- a build identity differing from the image digest;
|
||||
- a RollingUpdate strategy;
|
||||
- a CPU limit;
|
||||
- a grace period shorter than the drain.
|
||||
The checker first misread overlay patches as whole Deployments; it now checks
|
||||
only the base.
|
||||
- Kustomize renders dev and stage in order: egress policy (−2), migration hook
|
||||
(−1), provisioning hook (0), Deployment and Service (1), then the store check
|
||||
(PostSync).
|
||||
- `region-drill.sh` passed `bash -n`, and refuses World0 before kubectl (also
|
||||
enforced by the policy check).
|
||||
- Not executed: the in-cluster region sync and the drill.
|
||||
|
||||
@@ -116,6 +116,29 @@ this evidence in docs/engineering/VALIDATION.md:
|
||||
the Secret. The check must exit 3 with `database refused connect`.
|
||||
- Update the Secret's `pgpass` and sync again. It must exit 0.
|
||||
|
||||
## Region host (M8.4, ADR-0071)
|
||||
|
||||
`deploy/kubernetes/<env>` includes `stores/<env>` and `region/<env>`. A sync runs:
|
||||
1. the database egress policy (wave −2);
|
||||
2. the migration hook (wave −1);
|
||||
3. the provisioning hook (wave 0, idempotent);
|
||||
4. the `evo-region` Deployment and `evo-region-ops` Service (wave 1);
|
||||
5. the store check (PostSync).
|
||||
|
||||
Dev is world 100, region 1; stage is world 200, region 1. The server image digest
|
||||
in `region/<env>` is also the build identity.
|
||||
|
||||
Before the first region sync, apply `deploy/argocd/dev/project.yaml` (it allows
|
||||
Deployment and Service), and label the store node.
|
||||
|
||||
Evidence: `bash tools/ops/region-drill.sh dev` (dev or stage only; World0 is
|
||||
refused). It records the time to Ready after a rollout restart and after a forced
|
||||
pod deletion, and checks that the new epoch resumed at the committed tick. Also
|
||||
record:
|
||||
- the Argo sync result;
|
||||
- the first pod's `phase running epoch=... committed=...` line;
|
||||
- that `/readyz` stays 503 while a replacement waits out the lease.
|
||||
|
||||
## Break-glass
|
||||
|
||||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||||
|
||||
Reference in New Issue
Block a user