Files
aevolution/docs/adr/0097-m10-load-and-abuse-qualification.md
T
dingoandClaude Opus 5.5 ada77fd761
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
Add M10.9d load drill harness and serialize Mbed TLS use
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 08:38:13 -04:00

15 KiB
Raw Blame History

ADR-0097: M10 load and abuse qualification and overload response

Status

Accepted for M10.9 (2026-10-07). Decisions confirmed with the maintainer on 2026-10-07:

  • Venue. Only measurements on the reference hardware count as qualification: one RKE2 worker per region host on evolution-dev (ADR-0089). Local runs are smoke tests and development references, never evidence. The gate stays open until the dev run is done.
  • Load tool. One multi-client tool, evo-load: N session clients in one process, each a bot, reporting per-client distributions so aggregate throughput cannot hide a starved client.
  • Overload response: refuse, then shed. Measured per-tick work drives two thresholds. Above the first, new logins are refused. Above the second, the interest and far-update cadence of existing sessions shrink step by step. Both are restored with hysteresis, so ticks do not fall behind for admitted sessions.
  • Abuse coverage: packet fuzz, floods, slow peers and connect/disconnect storms, and region lease loss with 20 or more sessions attached.

Delivery:

  • 10.9a: the work measurement, the load governor and its effects (this text).
  • 10.9b: evo-load and a local smoke test.
  • 10.9c: the abuse suite.
  • 10.9d: the evolution-dev run harness, then the reference results.

Each segment amends this ADR. Every threshold here is an engineering budget under schemas/network/m10-budgets-v1.md, not a game rule.

Context

ADR-0089 budgets:

  • per-tick region work: p99 ≤ 25 ms on the reference worker;
  • session handling: ≤ 0.5 ms per session per tick;
  • at least 20 sessions per region (32 by default);
  • overload response: shed interest and refuse new sessions before tick debt.

Until now nothing measured the work, and the only bound was the session count. A region that falls behind accrues tick debt, which delays every admitted session's outcomes and snapshots.

Decision (10.9a)

  • Work measurement (evo-server loop). Busy time is the host's step plus the session bridge's work (session poll, replication, action carry and acknowledgements). Waits for input and the loop's sleeps are excluded. When the simulated tick advances by k, the busy time since the previous advance counts as k equal per-tick samples. The host's tick debt is read with every sample.

  • Load governor (include/evo/server/load_governor.hpp, portable, no clock). It keeps a window of the last 40 samples (2 s). It reports the window's p95, p99 and maximum, plus two outputs:

    • refusing turns on when p95 ≥ 15 ms or tick debt ≥ 2;
    • shed level (0-3) rises by one when p95 ≥ 20 ms or tick debt ≥ 4, at most once per second.

    One tick of debt is jitter (a commit flush or a scheduler hiccup) and only interrupts calm. As first written, any debt refused logins for at least 5 s, which made a login just after a restart's catch-up fail intermittently in a process test (amended in 10.9b).

    Calm means p95 < 10 ms and no debt. After 5 s of calm the level drops by one. At level 0, 5 s more of calm turns refusing off. Any non-calm sample restarts the calm timer. Thresholds are configurable and validated (relax < refuse ≤ shed ≤ tick budget).

  • Effects (RegionSessions::set_load):

    • While refusing, a Hello carrying a grant is closed with capacity before signature work. Handoff tickets are still admitted: a crossing moves an existing player between regions, and refusing it would strand the probe until the source aborts the transfer (up to 40 s).

    • The shed level limits replication (ReplicationServer::set_shed):

      Level Radius cap Near radius Far updates
      0 7 3 every 4 ticks
      1 7 3 every 8 ticks
      2 5 3 every 8 ticks
      3 3 2 every 16 ticks

      The stored interest is kept, so shedding forces no fresh baseline. Entities beyond the cap are removed from the client's view and return when the level drops. Each session's own marker is still sent every tick it changes.

  • Exposition. /metrics adds evo_m10_* gauges:

    • sessions;
    • refusing;
    • shed level;
    • work p95, p99 and maximum in microseconds;
    • the session layer's share, p95 and p99 (the ADR-0089 per-session budget is checked against it);
    • refused logins (counter).

    evo-server prints m10 load refusing=R shed=L p95_us=P on every change. No labels are added.

Consequences

  • Under overload, honest new players are refused (capacity, retry later) and existing sessions see a smaller, slower-updating view rather than delayed ticks. Crossings continue.
  • The thresholds are provisional until 10.9d measures the reference worker. A change is made by amending this ADR and the budgets table.
  • No change to the session protocol: the client already handles capacity and removals.

10.9a evidence (2026-10-07)

  • Unit tests:

    • the governor's thresholds, one step per second, hysteresis, the calm timer restarting on a non-calm window, tick debt alone, and policy validation;
    • the metrics families, present only with the listener and bounded at worst-case values;
    • replication shedding: radius caps 5 and 3 without a fresh baseline or a change of the stored interest, a far update every 16 ticks at level 3, and the full square back at level 0;
    • sessions under load: grant logins refused with capacity before signature work (the grant stays usable), tickets admitted.
  • Process test (region_m10_session_tests.cpp): a real evo-server with --ops-listen exposes the evo_m10_* gauges.

    Local development reference (WSL, Release, one session; not evidence): work p95 about 7 ms and p99 about 9 ms per tick. Of that, the session layer's p99 was about 4.5 ms. That is far above 0.5 ms per session, so 10.9b examines it with evo-load. Likely contributors are the per-tick snapshot source query and synchronous control renewals.

10.9b load tool (2026-10-07)

  • evo-load (tools/network/load.cpp, quic preset; usage in its header). Bots run on worker threads, each a RegionSessionClient with its own account and a dev grant signed in-process. A bot:

    • sends one intention per server tick at its InputLead (a random walk turning every 2 s) and predicts its marker with MotionPredictor;
    • sends a probe action every 2 s and moves its interest every 10 s (a fresh baseline);
    • logs in again 2 s after any close.

    Steps raise the bot count (--ramp-ms apart), warm up, then hold. Each step reports:

    • pooled p50, p95, p99 and maximum, plus the per-client p95 spread, for responsiveness (input to the first own-marker snapshot applying it), receipt RTT, final-outcome latency, login time and prediction error;
    • snapshot rate, gaps and bandwidth per client;
    • drops, failed logins and refusals;
    • the generator's own CPU;
    • with --metrics, the region's evo_m10_* gauges.

    Starvation is judged among the bots connected during the hold: a snapshot rate under 80 % of the median, or a response p95 over both twice the median and the budget. A step's verdict is within budget only with every bot established, no refusals or drops, no starvation, response p95 within budget (150 ms by default), outcome p95 ≤ 1.5 s, server work p99 ≤ 25 ms, no shedding or refusing, and no tick debt. The run ends with load result sustainable=N saturated_at=M.

  • Smoke (Postgres region_m10_session_tests): 4 then 8 bots against a real evo-server with --metrics. Every bot logs in once with no drops or refusals, both steps measure responses, outcomes and snapshots, and the report is complete.

  • Found while ramping:

    • The process-test fixture's Child::pump(1ms) rounded its wait down to 0 ms and never read. A chatty evo-server then blocked on its full stdout pipe, which looked like a server stall at 32 bots. The wait is now rounded up.
    • The governor's any-debt refusal was too twitchy (see the amended decision above).
  • Local development reference (WSL, Release, one region; not evidence):

    Bots Verdict Work p95 / max Session p95 (per session) Response p95 Outcome p95 Snapshots
    8 within 9.0 / 9.6 ms 1.7 ms (0.22) 106 ms 659 ms 20 Hz, no gaps
    16 within 11.0 / 11.2 ms 3.5 ms (0.22) 102 ms 661 ms 20 Hz
    24 within 13.1 / 13.3 ms 5.8 ms (0.24) 103 ms 664 ms 20 Hz
    32 refusing 14.3 / 17.0 ms 8.4 ms (0.26) 103 ms 665 ms 20 Hz

    At 48 and 64 bots, the bots beyond the 32-session cap fail at the transport connection limit, and the 32 established sessions are unaffected. Prediction errors were 0 (no loss on loopback), and no client starved at any step. The host step dominates the work; the session layer stays near 0.25 ms per session (budget 0.5).

10.9c-1 abuse suite: fuzzing and in-process abuse (2026-10-07)

  • Fuzzing (tests/fuzz/session_fuzz.cpp). For any input:

    • decode_session never crashes, and an accepted message has a stable canonical form (decode, encode, decode, encode gives the same bytes);
    • an accepted snapshot chunk reaches the client's assembler without crashing;
    • GrantVerifier and HandoffVerifier refuse the bytes, as they do the grant slot of a decoded Hello. Each holds a real key, so an acceptance would mean a claim byte was trusted before the signature.

    Two builds:

    • evo-session-fuzz: libFuzzer with ASan and UBSan behind EVO_BUILD_FUZZER, with the session codec, replication, motion, control and verifier sources compiled in so coverage instruments them;
    • session-fuzz-smoke: CTest, label network, on every platform. It feeds 68,000 seeded inputs: random bytes, and bit flips, truncations, extensions and edge-byte overwrites of an encoding of every v4 message type.
  • In-process abuse (quic-sessions). Each test measures an honest session tick by tick while the abuse runs. It must keep at least 90 % of its snapshots and admitted receipts, and the server's connections and retained state must stay bounded:

    • Hello flood: 160 raw connections over 40 ticks, sending garbage (closed with protocol), well-formed Hellos signed by an untrusted key (unauthenticated), or nothing (timeout). Connections stay within the transport bound of 32, and the region returns to the honest connection alone.
    • Datagram flood: an authenticated peer sends hundreds of intentions per tick for any tick. They are throttled per session (one per tick, refused receipts), and garbage then closes the peer (protocol).
    • Slow peer: an established client that never reads or acknowledges anything. The server retains at most its baselines for it (392 entities against a bound of 490).
    • Storm: 120 logins across 12 accounts, half closed politely and half vanishing. The region returns to one session and one connection.
  • Probe: evo-session-probe run --grants FILE [--grant N] logs in with grant N of an existing bundle, such as one from the dev renewal, without an issuer key. It is covered by the M10 process test and was used against evolution-dev, where the login was welcomed and its intention admitted. That region runs without synthetic inputs, as the Git policy requires, so nothing is simulated there outside a drill.

  • Not yet (10.9c-2): lease loss and SIGKILL with 20 or more evo-load bots attached, and a live raw-UDP sender of mutated datagrams. Our client endpoint caps its own send rate, so it cannot load the server's per-connection datagram cap (64, newest wins, with an O(queue) eviction scan); the raw sender measures that.

10.9c-2 authority loss under load (2026-10-08)

  • Lease loss under load (region_m10_session_tests.cpp): 24 evo-load bots on a real evo-server (loopback, fixed port). First the server is SIGKILLed, so its lease must expire before a successor runs. Then it is drained with SIGTERM: sessions close with shutdown and the lease is released. Each time the successor takes the same port under a new epoch, and every bot must log in again within 15 s of it being ready.

    Development reference:

    Loss Successor ready All 24 bots back after ready
    SIGKILL 3.5 s 5.0 s
    SIGTERM drain 0.18 s 2.4 s

    Over the 100 s hold the bots made 72 logins (24 + 2 × 24) and counted 48 drops, with no failed or refused login, and response p95 stayed at 106 ms. After a SIGKILL, clients only notice the loss through the QUIC idle timeout (5 s), which dominates the SIGKILL figure.

  • Scope narrowed: no standalone tool sends raw UDP traffic at a network endpoint. Abuse testing stays in-process or on loopback inside the test suite (10.9c-1). The server's per-connection datagram cap is analysed here instead of being measured with a flood tool. Each connection may hold 64 queued datagrams (newest wins). Evicting the oldest scans the endpoint's event queue (at most 4096 events), which happens only when one connection queues more than 64 datagrams between two server polls (normally under a millisecond apart). That cost is bounded per datagram but can add up at very high rates from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is recorded as a follow-up, not done.

10.9d drill harness (2026-10-08)

  • tools/ops/load-drill.sh (runbook: docs/networking/M10_LOAD_DRILL.md):
    1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of ADR-0073 (never in Git);
    2. runs evo-load through the public route;
    3. matches each step's hold with the server's reports in it;
    4. restores REGION_INPUTS on exit, including on failure.
  • evo-server logs m10 load-report … every 10 s: sessions, work and session-work percentiles, governor state and tick debt. The agent cannot reach the ops port (no port-forward, and the API-server proxy is refused), but it can read logs.
  • Bundle mode:
    • evo-load --grants FILE --grants-per-bot C logs bots in from a pre-issued bundle, so the drill machine never holds the issuer key;
    • evo-session-probe grants --accounts N writes such a bundle for N accounts;
    • each step line now carries exhausted= and hold_utc=.
  • Evidence: the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the M10 process test checks the load-report line. The drill's summariser was tested on synthetic logs; that test found a window-overlap bug, now fixed.
  • Finding: Mbed TLS is not thread-safe here. Bundle mode made logins fast enough that two bots on different threads verified their first connections at once. Both were refused as untrusted (QUIC cause 3), which surfaced as close_reasons=259:2. Mbed TLS and PSA are built without MBEDTLS_THREADING_C. Every network-library entry point that touches them now holds one process-wide recursive lock (include/evo/network/crypto_lock.hpp): the QUIC client's pinned check and the session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3 of 3 after. quic-transport gains a concurrent-connect test (8 threads, 5 rounds); it does not reliably reproduce the race, because PSA is already initialised there.
  • evo-load step lines carry close_reasons=REASON:COUNT,…: session close reasons, or 256 plus the QUIC cause when there was no session Close.
  • Not yet: the reference run. It needs an image containing this change promoted to dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead, because the drill restarts the dev region twice.