Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s

ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent 47b84bce74
commit ada77fd761
16 changed files with 609 additions and 70 deletions
+74
View File
@@ -0,0 +1,74 @@
# M10 load drill (evolution-dev reference worker)
ADR-0097 qualifies M10 load only on the evolution-dev reference worker.
`tools/ops/load-drill.sh` performs that run. It is a drill: it changes the live dev region
for the run's duration and restores it on exit, including on failure. It is never part of
CI.
## What it changes
- `REGION_INPUTS=development-v1` on `Deployment/evo-region` while it runs, the drill-time
override of ADR-0073. Synthetic inputs stay off in Git (deploy policy). On exit the
value goes back to empty, and the region restarts twice: once to enable, once to
restore.
- Nothing else: no Secrets, migrations, Jobs, SQL or Argo operations. While it runs, Argo
shows `Deployment/evo-region` OutOfSync, and Synced again after the restore.
## Inputs
In a tokens directory:
- `dev-ca.pem`: the current dev session CA, from the renewal.
- `load-grants.bin`: one bundle with grants for every bot, made where the dev issuer key
lives (the key never leaves that machine). For the default drill (up to 32 bots, 4
logins each):
```bash
evo-session-probe grants --issuer-key issuer.key --world 100 --frame 9 --region 1 \
--account <HEX32 base account> --accounts 32 --count 4 --lifetime 14400 \
--out load-grants.bin
```
Account i is the base account with its last two bytes set to i + 1. Each bot uses its
own 4 grants in order, one per login, and a bot that has used all 4 stays out
(`exhausted=` in the report). Grants are valid for at most 4 hours.
The session window (`REGION_SESSION_WINDOW`) must stay open until the drill ends; the
script refuses to start otherwise. The agent's kubectl access is enough: patch
`Deployment/evo-region`, read pods and logs.
## Running
```bash
EVO_LOAD=/path/to/evo-load tools/ops/load-drill.sh <tokens-dir> \
[--steps 20,24,28,32] [--hold 60] [--warmup 10] [--grants-per-bot 4]
```
Defaults: steps 20, 24, 28 and 32 bots; 60 s holds; 10 s warmups; 4 grants per bot;
`void.aeternalisgames.com:7400`. The drill takes about 6 minutes plus two region
restarts.
## Output
- **evo-load's per-step lines** (see `tools/network/load.cpp`): client distributions
(response, receipt, outcome, login), snapshot cadence, starvation, `close_reasons`
(why sessions or logins ended) and a client-side verdict, then
`load result sustainable=N saturated_at=M`.
- **One `drill step` line per step**, from the server's `m10 load-report` log lines
(every 10 s, read with `kubectl logs --timestamps` because the ops port is not
reachable for the agent) that fall inside the step's hold:
- the worst work p99 and maximum;
- the session-layer p99 and its per-session share;
- the governor's refusing and shed state;
- tick debt;
- `server_within_budget`: work p99 ≤ 25 ms, session p99 ≤ 0.5 ms × sessions, no debt.
The work directory (kept and printed at the end) holds `load.log` and `server.log`.
## Interpreting
A step qualifies when both evo-load's verdict and `server_within_budget` are 1. The
largest such step is the region's sustainable session count on the reference worker.
That number, with the first failing step and its reasons, goes into ADR-0097 and
`schemas/network/m10-budgets-v1.md`. If the governor engages (refusing or shed above 0),
that step is the overload response in action, and the thresholds are revisited
against the measured work.