Fix load governor startup refusal and drill issues from the first dev run
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s
ADR-0097. The first evolution-dev load drill (image 6c96629e) did not measure capacity: the governor refused six seconds after the restart with no session (p95 over a window of only startup ticks), and the bots spent their grants retrying every 2 s. Reference numbers with one session: tick work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms. - The governor acts on work only over a full 40-tick window; tick debt still counts at once. - evo-load backs off 10 s after a load refusal and, in bundle mode, reuses the grant the region refused before using it. - The drill waits for a calm load-report before the bots start, takes the drill pod's whole log before restoring (the live follow died), and confirms the restore against a pod other than the drill pod. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -5158,3 +5158,29 @@ Executed:
|
||||
|
||||
Not done: the reference run on evolution-dev. It needs this image promoted, a load bundle
|
||||
and the maintainer's go-ahead.
|
||||
|
||||
## 2026-10-08 — M10.9d first evolution-dev load drill (ADR-0097)
|
||||
|
||||
Ran `tools/ops/load-drill.sh` on evolution-dev, with the image `6c96629e…` promoted and the
|
||||
provision Job run with it. Steps 20/24/28/32, 60 s holds, 4 grants per bot, from the
|
||||
maintainer workstation. Restore verified: `REGION_INPUTS` empty, epoch 60, `ready 1`, Argo
|
||||
Synced and Healthy.
|
||||
|
||||
Result: not a capacity qualification.
|
||||
- The governor refused from six seconds after the restart, with no session (p95 24.4 ms
|
||||
over a partial window of startup ticks).
|
||||
- 19 of 20 bots spent all 4 grants on refusals within about 10 s.
|
||||
- The live log follow ended early, so only 4 server reports were captured.
|
||||
|
||||
Reference numbers with 1 session (`m10 load-report` at 14:33:58):
|
||||
- tick work p95 4.0 ms, p99 and maximum 33 ms;
|
||||
- session layer p95 0.68 ms, p99 3.0 ms;
|
||||
- client response p95 111 ms, outcome p95 1.13 s, snapshots at 20 Hz.
|
||||
|
||||
Fixes (in the following commit):
|
||||
- the governor acts on work only over a full window;
|
||||
- evo-load backs off 10 s after a load refusal and reuses the unused bundle grant;
|
||||
- the drill waits for a calm report, takes the pod log before restoring, and has a
|
||||
stricter restore check.
|
||||
|
||||
The next run needs a new image and a new bundle: some of this bundle's grants are spent.
|
||||
|
||||
Reference in New Issue
Block a user