Refuse session windows beyond the server's 4-hour maximum in deploy policy
Foundation / checks (client) (push) Successful in 4s
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 4s
Foundation / checks (deploy) (push) Successful in 3m31s
Foundation / postgres-tools (push) Successful in 5s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 5s

A window ending 5.2 h after start (b2f224d) put evo-region into a
crashloop (error m10-window-too-long); the server only sees it at
startup. The deploy policy now requires REGION_SESSION_WINDOW to name its
UTC activation and end within 4 h of it, with two selftest cases (one is
the b2f224d value). DEPLOYMENT.md documents the follow-on quarantine:
syncs recorded as Deployment-only run no hooks, so a promoted build is
never retained; the provision hook is re-run from the rendered manifest.
VALIDATION records the incident and the recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 07:41:06 -04:00
co-authored by Claude Opus 5.5
parent 4a72897679
commit 47b84bce74
4 changed files with 76 additions and 0 deletions
+33
View File
@@ -5085,3 +5085,36 @@ Executed:
- **Linux GCC 14:** `postgres-check.sh` with QUIC, all suites.
- **Linux Clang 19:** clang-tidy 19 on the M10 process tests (clean); clang-format 19.
- **Windows:** not applicable (the PostgreSQL process tests are Linux only).
## 2026-10-08 — evolution-dev crashloop and recovery
Incident:
- `b2f224d` set `REGION_SESSION_WINDOW` to end about 5.2 h after the pod started. evo-server
refuses windows longer than 4 h (`SessionWindow::max_duration_seconds`), so
`evo-region` exited with `error m10-window-too-long` in a loop.
- `4a72897` set a window of 3 h 50 min. The region then reached
`phase quarantined epoch=0 … exit 4`: the bot promotion `c3b40f4` (image `56647b49…`)
had never had its build retained. The provision hook still ran the previous image
(`2ccf6074…`, created 01:58Z).
- The two syncs requested afterwards with the agent token were both recorded by Argo
as limited to `Deployment/evo-region`, even though the second request carried no
resource filter. So no hooks ran. The cause is not visible with the agent's access
(possibly a guard on that account).
Recovery, within the agent contract (no reset, no SQL):
- Rendered `Job/aevolution-region-provision` from `deploy/kubernetes/dev` at
`4a72897` (image and `REGION_BUILD` `56647b49…`), deleted the stale Job and created it.
It completed with `already-provisioned … sequence=578 tick=12084`, the build retained
first.
- `rollout restart deployment/evo-region`. The region came up `phase running epoch=54
committed=12084`, `ready 1`, with history intact. Argo reports Synced and Healthy, and a
public-endpoint login attempt with an untrusted key was answered `unauthenticated`
(TLS and protocol fine).
Prevention:
- The deploy policy now refuses a `REGION_SESSION_WINDOW` that does not name its UTC
activation or that ends more than 4 h after it (two new selftest cases, one with the
`b2f224d` value).
- DEPLOYMENT.md documents the Deployment-only sync symptom and the hook re-run.
Executed: deploy policy check (0 failures) and selftest (46 cases, 0 failures).