Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md) runs the evolution-dev reference qualification: synthetic inputs on for the run (the ADR-0073 drill-time override, restored on exit), bundle-mode evo-load bots through the public route, and each step's hold matched with the server's reports. evo-server logs "m10 load-report" every 10 s, since the agent cannot reach the ops port. evo-load gains --grants FILE --grants-per-bot C, exhausted=, hold_utc= and close_reasons=; evo-session-probe grants gains --accounts N. Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on different threads verifying their first connections at once were refused as untrusted (two per smoke run). One process-wide recursive crypto lock now guards every network-library entry point that touches them; the smoke went from 2/2 failing to 3/3 clean. quic-transport adds a concurrent-connect test. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -222,3 +222,36 @@ snapshots.
|
||||
millisecond apart). That cost is bounded per datagram but can add up at very high rates
|
||||
from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is
|
||||
recorded as a follow-up, not done.
|
||||
|
||||
## 10.9d drill harness (2026-10-08)
|
||||
- **`tools/ops/load-drill.sh`** (runbook: `docs/networking/M10_LOAD_DRILL.md`):
|
||||
1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of
|
||||
ADR-0073 (never in Git);
|
||||
2. runs `evo-load` through the public route;
|
||||
3. matches each step's hold with the server's reports in it;
|
||||
4. restores `REGION_INPUTS` on exit, including on failure.
|
||||
- **`evo-server`** logs `m10 load-report …` every 10 s: sessions, work and session-work
|
||||
percentiles, governor state and tick debt. The agent cannot reach the ops port (no
|
||||
port-forward, and the API-server proxy is refused), but it can read logs.
|
||||
- **Bundle mode:**
|
||||
- `evo-load --grants FILE --grants-per-bot C` logs bots in from a pre-issued bundle, so
|
||||
the drill machine never holds the issuer key;
|
||||
- `evo-session-probe grants --accounts N` writes such a bundle for N accounts;
|
||||
- each step line now carries `exhausted=` and `hold_utc=`.
|
||||
- **Evidence:** the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the
|
||||
M10 process test checks the `load-report` line. The drill's summariser was tested on
|
||||
synthetic logs; that test found a window-overlap bug, now fixed.
|
||||
- **Finding: Mbed TLS is not thread-safe here.** Bundle mode made logins fast enough
|
||||
that two bots on different threads verified their first connections at once. Both were
|
||||
refused as `untrusted` (QUIC cause 3), which surfaced as `close_reasons=259:2`. Mbed TLS
|
||||
and PSA are built without `MBEDTLS_THREADING_C`. Every network-library entry point that
|
||||
touches them now holds one process-wide recursive lock
|
||||
(`include/evo/network/crypto_lock.hpp`): the QUIC client's pinned check and the
|
||||
session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3
|
||||
of 3 after. `quic-transport` gains a concurrent-connect test (8 threads, 5 rounds); it
|
||||
does not reliably reproduce the race, because PSA is already initialised there.
|
||||
- `evo-load` step lines carry `close_reasons=REASON:COUNT,…`: session close reasons, or
|
||||
256 plus the QUIC cause when there was no session Close.
|
||||
- **Not yet:** the reference run. It needs an image containing this change promoted to
|
||||
dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead,
|
||||
because the drill restarts the dev region twice.
|
||||
|
||||
Reference in New Issue
Block a user