Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s

ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent 47b84bce74
commit ada77fd761
16 changed files with 609 additions and 70 deletions
+2 -1
View File
@@ -549,7 +549,8 @@ smoke done; 10.9c-1 session fuzz target and smoke, in-process Hello and datagram
slow peer and connect/disconnect storm with honest-service checks done; 10.9c-2 SIGKILL and
drain lease loss with 24 bots (all back within 5 s of the successor) done; no standalone
raw-UDP sender (abuse stays in-process/loopback; datagram-cap eviction cost analysed).
10.9d dev-cluster drill harness and reference results follow.
10.9d drill harness (load-drill.sh, server load-report, bundle-mode evo-load, Mbed TLS
crypto lock) done; the evolution-dev reference run follows.
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
+7 -2
View File
@@ -123,8 +123,13 @@ refusing at 32 (work p95 about 14 ms), with responses about 103 ms and outcomes
and its intention admitted (the dev region does not simulate outside drills). M10.9c-2
kills and drains a real evo-server under 24 `evo-load` bots. All of them log in to the
successor within 5.0 s (SIGKILL, after a 3.5 s lease wait) or 2.4 s (drain), with no failed
logins, and responses stay at 106 ms p95. Next: 10.9d, the evolution-dev drill harness and
the reference-worker run.
logins, and responses stay at 106 ms p95. M10.9d adds the evolution-dev load drill
(`tools/ops/load-drill.sh`). It turns synthetic inputs on for the run, drives bundle-mode
`evo-load` bots through the public route, and matches each step with the server's 10 s
`m10 load-report` log lines. Building it found that Mbed TLS (built without its threading
layer) raced when bots verified connections on several threads; one process-wide crypto
lock fixed it. Next: the reference run, which needs a promoted image, a load bundle and
the maintainer's go-ahead.
## Previous gate - 2026-10-02
@@ -222,3 +222,36 @@ snapshots.
millisecond apart). That cost is bounded per datagram but can add up at very high rates
from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is
recorded as a follow-up, not done.
## 10.9d drill harness (2026-10-08)
- **`tools/ops/load-drill.sh`** (runbook: `docs/networking/M10_LOAD_DRILL.md`):
1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of
ADR-0073 (never in Git);
2. runs `evo-load` through the public route;
3. matches each step's hold with the server's reports in it;
4. restores `REGION_INPUTS` on exit, including on failure.
- **`evo-server`** logs `m10 load-report …` every 10 s: sessions, work and session-work
percentiles, governor state and tick debt. The agent cannot reach the ops port (no
port-forward, and the API-server proxy is refused), but it can read logs.
- **Bundle mode:**
- `evo-load --grants FILE --grants-per-bot C` logs bots in from a pre-issued bundle, so
the drill machine never holds the issuer key;
- `evo-session-probe grants --accounts N` writes such a bundle for N accounts;
- each step line now carries `exhausted=` and `hold_utc=`.
- **Evidence:** the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the
M10 process test checks the `load-report` line. The drill's summariser was tested on
synthetic logs; that test found a window-overlap bug, now fixed.
- **Finding: Mbed TLS is not thread-safe here.** Bundle mode made logins fast enough
that two bots on different threads verified their first connections at once. Both were
refused as `untrusted` (QUIC cause 3), which surfaced as `close_reasons=259:2`. Mbed TLS
and PSA are built without `MBEDTLS_THREADING_C`. Every network-library entry point that
touches them now holds one process-wide recursive lock
(`include/evo/network/crypto_lock.hpp`): the QUIC client's pinned check and the
session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3
of 3 after. `quic-transport` gains a concurrent-connect test (8 threads, 5 rounds); it
does not reliably reproduce the race, because PSA is already initialised there.
- `evo-load` step lines carry `close_reasons=REASON:COUNT,…`: session close reasons, or
256 plus the QUIC cause when there was no session Close.
- **Not yet:** the reference run. It needs an image containing this change promoted to
dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead,
because the drill restarts the dev region twice.
+40
View File
@@ -5118,3 +5118,43 @@ Prevention:
- DEPLOYMENT.md documents the Deployment-only sync symptom and the hook re-run.
Executed: deploy policy check (0 failures) and selftest (46 cases, 0 failures).
## 2026-10-08 — M10.9d load drill harness (ADR-0097)
Implemented:
- `tools/ops/load-drill.sh` and its runbook `docs/networking/M10_LOAD_DRILL.md`:
synthetic inputs on for the run (restored on exit), `evo-load` through the public route,
and server reports matched to each hold.
- evo-server `m10 load-report` every 10 s. The agent cannot reach the ops port: no
port-forward, and the API-server proxy is refused.
- `evo-load --grants FILE --grants-per-bot C` with `exhausted=`, `hold_utc=` and
`close_reasons=`; `evo-session-probe grants --accounts N`.
- `include/evo/network/crypto_lock.hpp`: one recursive lock at every Mbed TLS/PSA entry
point of the network library (`MBEDTLS_THREADING_C` is off).
Findings:
- Concurrent first verifications on different threads were refused as `untrusted`, two
per smoke run in bundle mode (2 of 2 runs). With the lock, 3 of 3 runs had no failures.
- The drill summariser, tested on synthetic logs, counted reports from after the hold
(the next step's ramp). Fixed: only reports wholly inside the hold count.
Tests:
- M10 load smoke in bundle mode (8 accounts × 2 grants) with `exhausted=0`.
- The M10 process test checks `m10 load-report`.
- `quic-transport` `concurrent_clients`: 8 threads × 5 rounds all connect. It does not
reliably reproduce the race; the smoke does.
- The lease-loss test's close reasons read 24 × `shutdown` (drain) and 24 × idle
(SIGKILL).
Executed:
- **Linux GCC 14:** presets `quic` and `network` 5/5; `postgres-check.sh` with QUIC, all 18
suites.
- **Linux Clang 19:** clang-tidy 19 on `load.cpp`, the probe, `session_grant.cpp`,
`handoff_ticket.cpp`, `quic_endpoint.cpp`, `quic_transport.cpp`, evo-server `main.cpp`
with the bridge, and the M10 process tests (3 findings fixed); clang-format 19.
- **Windows MSVC 19.44:** presets `quic` 5/5 and `client-network` 15/15.
- **Drill script:** `bash -n`, plus the summariser on synthetic logs; shellcheck is not
installed here.
Not done: the reference run on evolution-dev. It needs this image promoted, a load bundle
and the maintainer's go-ahead.
+74
View File
@@ -0,0 +1,74 @@
# M10 load drill (evolution-dev reference worker)
ADR-0097 qualifies M10 load only on the evolution-dev reference worker.
`tools/ops/load-drill.sh` performs that run. It is a drill: it changes the live dev region
for the run's duration and restores it on exit, including on failure. It is never part of
CI.
## What it changes
- `REGION_INPUTS=development-v1` on `Deployment/evo-region` while it runs, the drill-time
override of ADR-0073. Synthetic inputs stay off in Git (deploy policy). On exit the
value goes back to empty, and the region restarts twice: once to enable, once to
restore.
- Nothing else: no Secrets, migrations, Jobs, SQL or Argo operations. While it runs, Argo
shows `Deployment/evo-region` OutOfSync, and Synced again after the restore.
## Inputs
In a tokens directory:
- `dev-ca.pem`: the current dev session CA, from the renewal.
- `load-grants.bin`: one bundle with grants for every bot, made where the dev issuer key
lives (the key never leaves that machine). For the default drill (up to 32 bots, 4
logins each):
```bash
evo-session-probe grants --issuer-key issuer.key --world 100 --frame 9 --region 1 \
--account <HEX32 base account> --accounts 32 --count 4 --lifetime 14400 \
--out load-grants.bin
```
Account i is the base account with its last two bytes set to i + 1. Each bot uses its
own 4 grants in order, one per login, and a bot that has used all 4 stays out
(`exhausted=` in the report). Grants are valid for at most 4 hours.
The session window (`REGION_SESSION_WINDOW`) must stay open until the drill ends; the
script refuses to start otherwise. The agent's kubectl access is enough: patch
`Deployment/evo-region`, read pods and logs.
## Running
```bash
EVO_LOAD=/path/to/evo-load tools/ops/load-drill.sh <tokens-dir> \
[--steps 20,24,28,32] [--hold 60] [--warmup 10] [--grants-per-bot 4]
```
Defaults: steps 20, 24, 28 and 32 bots; 60 s holds; 10 s warmups; 4 grants per bot;
`void.aeternalisgames.com:7400`. The drill takes about 6 minutes plus two region
restarts.
## Output
- **evo-load's per-step lines** (see `tools/network/load.cpp`): client distributions
(response, receipt, outcome, login), snapshot cadence, starvation, `close_reasons`
(why sessions or logins ended) and a client-side verdict, then
`load result sustainable=N saturated_at=M`.
- **One `drill step` line per step**, from the server's `m10 load-report` log lines
(every 10 s, read with `kubectl logs --timestamps` because the ops port is not
reachable for the agent) that fall inside the step's hold:
- the worst work p99 and maximum;
- the session-layer p99 and its per-session share;
- the governor's refusing and shed state;
- tick debt;
- `server_within_budget`: work p99 ≤ 25 ms, session p99 ≤ 0.5 ms × sessions, no debt.
The work directory (kept and printed at the end) holds `load.log` and `server.log`.
## Interpreting
A step qualifies when both evo-load's verdict and `server_within_budget` are 1. The
largest such step is the region's sustainable session count on the reference worker.
That number, with the first failing step and its reasons, goes into ADR-0097 and
`schemas/network/m10-budgets-v1.md`. If the governor engages (refusing or shed above 0),
that step is the overload response in action, and the thresholds are revisited
against the measured work.