Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s

ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent 47b84bce74
commit ada77fd761
16 changed files with 609 additions and 70 deletions
@@ -222,3 +222,36 @@ snapshots.
millisecond apart). That cost is bounded per datagram but can add up at very high rates
from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is
recorded as a follow-up, not done.
## 10.9d drill harness (2026-10-08)
- **`tools/ops/load-drill.sh`** (runbook: `docs/networking/M10_LOAD_DRILL.md`):
1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of
ADR-0073 (never in Git);
2. runs `evo-load` through the public route;
3. matches each step's hold with the server's reports in it;
4. restores `REGION_INPUTS` on exit, including on failure.
- **`evo-server`** logs `m10 load-report …` every 10 s: sessions, work and session-work
percentiles, governor state and tick debt. The agent cannot reach the ops port (no
port-forward, and the API-server proxy is refused), but it can read logs.
- **Bundle mode:**
- `evo-load --grants FILE --grants-per-bot C` logs bots in from a pre-issued bundle, so
the drill machine never holds the issuer key;
- `evo-session-probe grants --accounts N` writes such a bundle for N accounts;
- each step line now carries `exhausted=` and `hold_utc=`.
- **Evidence:** the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the
M10 process test checks the `load-report` line. The drill's summariser was tested on
synthetic logs; that test found a window-overlap bug, now fixed.
- **Finding: Mbed TLS is not thread-safe here.** Bundle mode made logins fast enough
that two bots on different threads verified their first connections at once. Both were
refused as `untrusted` (QUIC cause 3), which surfaced as `close_reasons=259:2`. Mbed TLS
and PSA are built without `MBEDTLS_THREADING_C`. Every network-library entry point that
touches them now holds one process-wide recursive lock
(`include/evo/network/crypto_lock.hpp`): the QUIC client's pinned check and the
session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3
of 3 after. `quic-transport` gains a concurrent-connect test (8 threads, 5 rounds); it
does not reliably reproduce the race, because PSA is already initialised there.
- `evo-load` step lines carry `close_reasons=REASON:COUNT,…`: session close reasons, or
256 plus the QUIC cause when there was no session Close.
- **Not yet:** the reference run. It needs an image containing this change promoted to
dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead,
because the drill restarts the dev region twice.