Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md) runs the evolution-dev reference qualification: synthetic inputs on for the run (the ADR-0073 drill-time override, restored on exit), bundle-mode evo-load bots through the public route, and each step's hold matched with the server's reports. evo-server logs "m10 load-report" every 10 s, since the agent cannot reach the ops port. evo-load gains --grants FILE --grants-per-bot C, exhausted=, hold_utc= and close_reasons=; evo-session-probe grants gains --accounts N. Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on different threads verifying their first connections at once were refused as untrusted (two per smoke run). One process-wide recursive crypto lock now guards every network-library entry point that touches them; the smoke went from 2/2 failing to 3/3 clean. quic-transport adds a concurrent-connect test. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+2
-1
@@ -549,7 +549,8 @@ smoke done; 10.9c-1 session fuzz target and smoke, in-process Hello and datagram
|
||||
slow peer and connect/disconnect storm with honest-service checks done; 10.9c-2 SIGKILL and
|
||||
drain lease loss with 24 bots (all back within 5 s of the successor) done; no standalone
|
||||
raw-UDP sender (abuse stays in-process/loopback; datagram-cap eviction cost analysed).
|
||||
10.9d dev-cluster drill harness and reference results follow.
|
||||
10.9d drill harness (load-drill.sh, server load-report, bundle-mode evo-load, Mbed TLS
|
||||
crypto lock) done; the evolution-dev reference run follows.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+7
-2
@@ -123,8 +123,13 @@ refusing at 32 (work p95 about 14 ms), with responses about 103 ms and outcomes
|
||||
and its intention admitted (the dev region does not simulate outside drills). M10.9c-2
|
||||
kills and drains a real evo-server under 24 `evo-load` bots. All of them log in to the
|
||||
successor within 5.0 s (SIGKILL, after a 3.5 s lease wait) or 2.4 s (drain), with no failed
|
||||
logins, and responses stay at 106 ms p95. Next: 10.9d, the evolution-dev drill harness and
|
||||
the reference-worker run.
|
||||
logins, and responses stay at 106 ms p95. M10.9d adds the evolution-dev load drill
|
||||
(`tools/ops/load-drill.sh`). It turns synthetic inputs on for the run, drives bundle-mode
|
||||
`evo-load` bots through the public route, and matches each step with the server's 10 s
|
||||
`m10 load-report` log lines. Building it found that Mbed TLS (built without its threading
|
||||
layer) raced when bots verified connections on several threads; one process-wide crypto
|
||||
lock fixed it. Next: the reference run, which needs a promoted image, a load bundle and
|
||||
the maintainer's go-ahead.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -222,3 +222,36 @@ snapshots.
|
||||
millisecond apart). That cost is bounded per datagram but can add up at very high rates
|
||||
from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is
|
||||
recorded as a follow-up, not done.
|
||||
|
||||
## 10.9d drill harness (2026-10-08)
|
||||
- **`tools/ops/load-drill.sh`** (runbook: `docs/networking/M10_LOAD_DRILL.md`):
|
||||
1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of
|
||||
ADR-0073 (never in Git);
|
||||
2. runs `evo-load` through the public route;
|
||||
3. matches each step's hold with the server's reports in it;
|
||||
4. restores `REGION_INPUTS` on exit, including on failure.
|
||||
- **`evo-server`** logs `m10 load-report …` every 10 s: sessions, work and session-work
|
||||
percentiles, governor state and tick debt. The agent cannot reach the ops port (no
|
||||
port-forward, and the API-server proxy is refused), but it can read logs.
|
||||
- **Bundle mode:**
|
||||
- `evo-load --grants FILE --grants-per-bot C` logs bots in from a pre-issued bundle, so
|
||||
the drill machine never holds the issuer key;
|
||||
- `evo-session-probe grants --accounts N` writes such a bundle for N accounts;
|
||||
- each step line now carries `exhausted=` and `hold_utc=`.
|
||||
- **Evidence:** the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the
|
||||
M10 process test checks the `load-report` line. The drill's summariser was tested on
|
||||
synthetic logs; that test found a window-overlap bug, now fixed.
|
||||
- **Finding: Mbed TLS is not thread-safe here.** Bundle mode made logins fast enough
|
||||
that two bots on different threads verified their first connections at once. Both were
|
||||
refused as `untrusted` (QUIC cause 3), which surfaced as `close_reasons=259:2`. Mbed TLS
|
||||
and PSA are built without `MBEDTLS_THREADING_C`. Every network-library entry point that
|
||||
touches them now holds one process-wide recursive lock
|
||||
(`include/evo/network/crypto_lock.hpp`): the QUIC client's pinned check and the
|
||||
session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3
|
||||
of 3 after. `quic-transport` gains a concurrent-connect test (8 threads, 5 rounds); it
|
||||
does not reliably reproduce the race, because PSA is already initialised there.
|
||||
- `evo-load` step lines carry `close_reasons=REASON:COUNT,…`: session close reasons, or
|
||||
256 plus the QUIC cause when there was no session Close.
|
||||
- **Not yet:** the reference run. It needs an image containing this change promoted to
|
||||
dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead,
|
||||
because the drill restarts the dev region twice.
|
||||
|
||||
@@ -5118,3 +5118,43 @@ Prevention:
|
||||
- DEPLOYMENT.md documents the Deployment-only sync symptom and the hook re-run.
|
||||
|
||||
Executed: deploy policy check (0 failures) and selftest (46 cases, 0 failures).
|
||||
|
||||
## 2026-10-08 — M10.9d load drill harness (ADR-0097)
|
||||
|
||||
Implemented:
|
||||
- `tools/ops/load-drill.sh` and its runbook `docs/networking/M10_LOAD_DRILL.md`:
|
||||
synthetic inputs on for the run (restored on exit), `evo-load` through the public route,
|
||||
and server reports matched to each hold.
|
||||
- evo-server `m10 load-report` every 10 s. The agent cannot reach the ops port: no
|
||||
port-forward, and the API-server proxy is refused.
|
||||
- `evo-load --grants FILE --grants-per-bot C` with `exhausted=`, `hold_utc=` and
|
||||
`close_reasons=`; `evo-session-probe grants --accounts N`.
|
||||
- `include/evo/network/crypto_lock.hpp`: one recursive lock at every Mbed TLS/PSA entry
|
||||
point of the network library (`MBEDTLS_THREADING_C` is off).
|
||||
|
||||
Findings:
|
||||
- Concurrent first verifications on different threads were refused as `untrusted`, two
|
||||
per smoke run in bundle mode (2 of 2 runs). With the lock, 3 of 3 runs had no failures.
|
||||
- The drill summariser, tested on synthetic logs, counted reports from after the hold
|
||||
(the next step's ramp). Fixed: only reports wholly inside the hold count.
|
||||
|
||||
Tests:
|
||||
- M10 load smoke in bundle mode (8 accounts × 2 grants) with `exhausted=0`.
|
||||
- The M10 process test checks `m10 load-report`.
|
||||
- `quic-transport` `concurrent_clients`: 8 threads × 5 rounds all connect. It does not
|
||||
reliably reproduce the race; the smoke does.
|
||||
- The lease-loss test's close reasons read 24 × `shutdown` (drain) and 24 × idle
|
||||
(SIGKILL).
|
||||
|
||||
Executed:
|
||||
- **Linux GCC 14:** presets `quic` and `network` 5/5; `postgres-check.sh` with QUIC, all 18
|
||||
suites.
|
||||
- **Linux Clang 19:** clang-tidy 19 on `load.cpp`, the probe, `session_grant.cpp`,
|
||||
`handoff_ticket.cpp`, `quic_endpoint.cpp`, `quic_transport.cpp`, evo-server `main.cpp`
|
||||
with the bridge, and the M10 process tests (3 findings fixed); clang-format 19.
|
||||
- **Windows MSVC 19.44:** presets `quic` 5/5 and `client-network` 15/15.
|
||||
- **Drill script:** `bash -n`, plus the summariser on synthetic logs; shellcheck is not
|
||||
installed here.
|
||||
|
||||
Not done: the reference run on evolution-dev. It needs this image promoted, a load bundle
|
||||
and the maintainer's go-ahead.
|
||||
|
||||
@@ -0,0 +1,74 @@
|
||||
# M10 load drill (evolution-dev reference worker)
|
||||
|
||||
ADR-0097 qualifies M10 load only on the evolution-dev reference worker.
|
||||
`tools/ops/load-drill.sh` performs that run. It is a drill: it changes the live dev region
|
||||
for the run's duration and restores it on exit, including on failure. It is never part of
|
||||
CI.
|
||||
|
||||
## What it changes
|
||||
|
||||
- `REGION_INPUTS=development-v1` on `Deployment/evo-region` while it runs, the drill-time
|
||||
override of ADR-0073. Synthetic inputs stay off in Git (deploy policy). On exit the
|
||||
value goes back to empty, and the region restarts twice: once to enable, once to
|
||||
restore.
|
||||
- Nothing else: no Secrets, migrations, Jobs, SQL or Argo operations. While it runs, Argo
|
||||
shows `Deployment/evo-region` OutOfSync, and Synced again after the restore.
|
||||
|
||||
## Inputs
|
||||
|
||||
In a tokens directory:
|
||||
- `dev-ca.pem`: the current dev session CA, from the renewal.
|
||||
- `load-grants.bin`: one bundle with grants for every bot, made where the dev issuer key
|
||||
lives (the key never leaves that machine). For the default drill (up to 32 bots, 4
|
||||
logins each):
|
||||
|
||||
```bash
|
||||
evo-session-probe grants --issuer-key issuer.key --world 100 --frame 9 --region 1 \
|
||||
--account <HEX32 base account> --accounts 32 --count 4 --lifetime 14400 \
|
||||
--out load-grants.bin
|
||||
```
|
||||
|
||||
Account i is the base account with its last two bytes set to i + 1. Each bot uses its
|
||||
own 4 grants in order, one per login, and a bot that has used all 4 stays out
|
||||
(`exhausted=` in the report). Grants are valid for at most 4 hours.
|
||||
|
||||
The session window (`REGION_SESSION_WINDOW`) must stay open until the drill ends; the
|
||||
script refuses to start otherwise. The agent's kubectl access is enough: patch
|
||||
`Deployment/evo-region`, read pods and logs.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
EVO_LOAD=/path/to/evo-load tools/ops/load-drill.sh <tokens-dir> \
|
||||
[--steps 20,24,28,32] [--hold 60] [--warmup 10] [--grants-per-bot 4]
|
||||
```
|
||||
|
||||
Defaults: steps 20, 24, 28 and 32 bots; 60 s holds; 10 s warmups; 4 grants per bot;
|
||||
`void.aeternalisgames.com:7400`. The drill takes about 6 minutes plus two region
|
||||
restarts.
|
||||
|
||||
## Output
|
||||
|
||||
- **evo-load's per-step lines** (see `tools/network/load.cpp`): client distributions
|
||||
(response, receipt, outcome, login), snapshot cadence, starvation, `close_reasons`
|
||||
(why sessions or logins ended) and a client-side verdict, then
|
||||
`load result sustainable=N saturated_at=M`.
|
||||
- **One `drill step` line per step**, from the server's `m10 load-report` log lines
|
||||
(every 10 s, read with `kubectl logs --timestamps` because the ops port is not
|
||||
reachable for the agent) that fall inside the step's hold:
|
||||
- the worst work p99 and maximum;
|
||||
- the session-layer p99 and its per-session share;
|
||||
- the governor's refusing and shed state;
|
||||
- tick debt;
|
||||
- `server_within_budget`: work p99 ≤ 25 ms, session p99 ≤ 0.5 ms × sessions, no debt.
|
||||
|
||||
The work directory (kept and printed at the end) holds `load.log` and `server.log`.
|
||||
|
||||
## Interpreting
|
||||
|
||||
A step qualifies when both evo-load's verdict and `server_within_budget` are 1. The
|
||||
largest such step is the region's sustainable session count on the reference worker.
|
||||
That number, with the first failing step and its reasons, goes into ADR-0097 and
|
||||
`schemas/network/m10-budgets-v1.md`. If the governor engages (refusing or shed above 0),
|
||||
that step is the overload response in action, and the thresholds are revisited
|
||||
against the measured work.
|
||||
Reference in New Issue
Block a user