Files
dingoandClaude Opus 5.5 ada77fd761
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
Add M10.9d load drill harness and serialize Mbed TLS use
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 08:38:13 -04:00

75 lines
3.3 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# M10 load drill (evolution-dev reference worker)
ADR-0097 qualifies M10 load only on the evolution-dev reference worker.
`tools/ops/load-drill.sh` performs that run. It is a drill: it changes the live dev region
for the run's duration and restores it on exit, including on failure. It is never part of
CI.
## What it changes
- `REGION_INPUTS=development-v1` on `Deployment/evo-region` while it runs, the drill-time
override of ADR-0073. Synthetic inputs stay off in Git (deploy policy). On exit the
value goes back to empty, and the region restarts twice: once to enable, once to
restore.
- Nothing else: no Secrets, migrations, Jobs, SQL or Argo operations. While it runs, Argo
shows `Deployment/evo-region` OutOfSync, and Synced again after the restore.
## Inputs
In a tokens directory:
- `dev-ca.pem`: the current dev session CA, from the renewal.
- `load-grants.bin`: one bundle with grants for every bot, made where the dev issuer key
lives (the key never leaves that machine). For the default drill (up to 32 bots, 4
logins each):
```bash
evo-session-probe grants --issuer-key issuer.key --world 100 --frame 9 --region 1 \
--account <HEX32 base account> --accounts 32 --count 4 --lifetime 14400 \
--out load-grants.bin
```
Account i is the base account with its last two bytes set to i + 1. Each bot uses its
own 4 grants in order, one per login, and a bot that has used all 4 stays out
(`exhausted=` in the report). Grants are valid for at most 4 hours.
The session window (`REGION_SESSION_WINDOW`) must stay open until the drill ends; the
script refuses to start otherwise. The agent's kubectl access is enough: patch
`Deployment/evo-region`, read pods and logs.
## Running
```bash
EVO_LOAD=/path/to/evo-load tools/ops/load-drill.sh <tokens-dir> \
[--steps 20,24,28,32] [--hold 60] [--warmup 10] [--grants-per-bot 4]
```
Defaults: steps 20, 24, 28 and 32 bots; 60 s holds; 10 s warmups; 4 grants per bot;
`void.aeternalisgames.com:7400`. The drill takes about 6 minutes plus two region
restarts.
## Output
- **evo-load's per-step lines** (see `tools/network/load.cpp`): client distributions
(response, receipt, outcome, login), snapshot cadence, starvation, `close_reasons`
(why sessions or logins ended) and a client-side verdict, then
`load result sustainable=N saturated_at=M`.
- **One `drill step` line per step**, from the server's `m10 load-report` log lines
(every 10 s, read with `kubectl logs --timestamps` because the ops port is not
reachable for the agent) that fall inside the step's hold:
- the worst work p99 and maximum;
- the session-layer p99 and its per-session share;
- the governor's refusing and shed state;
- tick debt;
- `server_within_budget`: work p99 ≤ 25 ms, session p99 ≤ 0.5 ms × sessions, no debt.
The work directory (kept and printed at the end) holds `load.log` and `server.log`.
## Interpreting
A step qualifies when both evo-load's verdict and `server_within_budget` are 1. The
largest such step is the region's sustainable session count on the reference worker.
That number, with the first failing step and its reasons, goes into ADR-0097 and
`schemas/network/m10-budgets-v1.md`. If the governor engages (refusing or shed above 0),
that step is the overload response in action, and the thresholds are revisited
against the measured work.