Files
aevolution/docs/adr/0097-m10-load-and-abuse-qualification.md
T
dingoandClaude Opus 5.5 1daa62cf6d
Foundation / checks (deploy) (push) Successful in 3m26s
Foundation / checks (ecs) (push) Successful in 1m55s
Foundation / checks (client) (push) Successful in 12m32s
Foundation / checks (network) (push) Successful in 13m34s
Foundation / checks (tsan) (push) Successful in 7m42s
Foundation / checks (checks) (push) Successful in 32m43s
Foundation / checks (postgres) (push) Successful in 46m18s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m28s
Foundation / server-image (push) Successful in 4m5s
M10.9: raise the evo-server session cap to 2000 and close 10.9 for now
Maintainer decision (2026-10-09): the region session cap becomes a safety bound of 2000
(plus 64 connections of headroom); the load governor is the working limit. Library bounds
go from 1024 to 4096 in SessionControl, IntentAdmission and ReplicationServer (defaults
stay 32). Drills stop for now: saturation and native-client observation move to a later
2k drill, whose prerequisites ADR-0097 lists (single-thread CPU, region login rate,
256 MiB pod limit, linear lookups). Declared: >= 32 sessions within every budget on the
reference worker. Budgets v2, STATUS, ROADMAP, VALIDATION. Refresh the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-09 08:22:04 -04:00

29 KiB
Raw Blame History

[Reviewed]

ADR-0097: M10 load and abuse qualification and overload response

Status

Accepted for M10.9 (2026-10-07). Decisions confirmed with the maintainer on 2026-10-07:

  • Venue. Only measurements on the reference hardware count as qualification: one RKE2 worker per region host on evolution-dev (ADR-0089). Local runs are smoke tests and development references, never evidence. The gate stays open until the dev run is done.
  • Load tool. One multi-client tool, evo-load: N session clients in one process, each a bot, reporting per-client distributions so aggregate throughput cannot hide a starved client.
  • Overload response: refuse, then shed. Measured per-tick work drives two thresholds. Above the first, new logins are refused. Above the second, the interest and far-update cadence of existing sessions shrink step by step. Both are restored with hysteresis, so ticks do not fall behind for admitted sessions.
  • Abuse coverage: packet fuzz, floods, slow peers and connect/disconnect storms, and region lease loss with 20 or more sessions attached.

Delivery:

  • 10.9a: the work measurement, the load governor and its effects (this text).
  • 10.9b: evo-load and a local smoke test.
  • 10.9c: the abuse suite.
  • 10.9d: the evolution-dev run harness, then the reference results.

Each segment amends this ADR. Every threshold here is an engineering budget under schemas/network/m10-budgets-v1.md, not a game rule.

Amendment 2026-10-08: Separate qualification gates on the target environment

Maintainer selected option A during reassessment (D-004), with the explicit requirement that qualification runs use the expected deployment environment.

  • Steady-state capacity: measure sustained load after a bounded, recorded warm-up. Failure to reach the required stable state within that bound is a failed run; do not wait indefinitely or selectively discard slow samples.
  • Startup and recovery: separately measure cold starts, reconnect storms, fault recovery, client recovery and durable outcomes from the start of each event. Retain startup spikes and failures in this report. Both gates must pass; warm-up exclusion from capacity does not excuse a recovery failure.
  • Venue for both gates: use the intended evolution-dev Kubernetes deployment on the ADR-0089 reference RKE2 worker, with deployed image, resource limits, network route, PostgreSQL and object-storage configuration recorded. Capture hardware and placement for each run; reconcile reference changes before comparing results. A workstation or WSL-hosted server is not a substitute.
  • Local evidence: WSL, loopback and developer-host runs remain useful for correctness, regression and smoke tests. They do not qualify capacity or deployment recovery, even if their numbers meet the budgets. A workstation may generate test clients against the actual cluster; report generator limits.

The first evolution-dev reference run documented below remains a failed capacity qualification. Earlier local tests and that cluster attempt must be labelled separately. Implement explicit warm-up bounds, separate acceptance reports and target-environment evidence before declaring M10.9 qualified. Existing budgets and durability requirements are not relaxed by this decision.

Context

ADR-0089 budgets:

  • per-tick region work: p99 ≤ 25 ms on the reference worker;
  • session handling: ≤ 0.5 ms per session per tick;
  • at least 20 sessions per region (32 by default);
  • overload response: shed interest and refuse new sessions before tick debt.

Until now nothing measured the work, and the only bound was the session count. A region that falls behind accrues tick debt, which delays every admitted session's outcomes and snapshots.

Decision (10.9a)

  • Work measurement (evo-server loop). Busy time is the host's step plus the session bridge's work (session poll, replication, action carry and acknowledgements). Waits for input and the loop's sleeps are excluded. When the simulated tick advances by k, the busy time since the previous advance counts as k equal per-tick samples. The host's tick debt is read with every sample.

  • Load governor (include/evo/server/load_governor.hpp, portable, no clock). It keeps a window of the last 40 samples (2 s). It reports the window's p95, p99 and maximum, plus two outputs:

    • refusing turns on when p95 ≥ 15 ms or tick debt ≥ 2;
    • shed level (0-3) rises by one when p95 ≥ 20 ms or tick debt ≥ 4, at most once per second.

    One tick of debt is jitter (a commit flush or a scheduler hiccup) and only interrupts calm. As first written, any debt refused logins for at least 5 s, which made a login just after a restart's catch-up fail intermittently in a process test (amended in 10.9b).

    Calm means p95 < 10 ms and no debt. After 5 s of calm the level drops by one. At level 0, 5 s more of calm turns refusing off. Any non-calm sample restarts the calm timer. Thresholds are configurable and validated (relax < refuse ≤ shed ≤ tick budget).

  • Effects (RegionSessions::set_load):

    • While refusing, a Hello carrying a grant is closed with capacity before signature work. Handoff tickets are still admitted: a crossing moves an existing player between regions, and refusing it would strand the probe until the source aborts the transfer (up to 40 s).

    • The shed level limits replication (ReplicationServer::set_shed):

      Level Radius cap Near radius Far updates
      0 7 3 every 4 ticks
      1 7 3 every 8 ticks
      2 5 3 every 8 ticks
      3 3 2 every 16 ticks

      The stored interest is kept, so shedding forces no fresh baseline. Entities beyond the cap are removed from the client's view and return when the level drops. Each session's own marker is still sent every tick it changes.

  • Exposition. /metrics adds evo_m10_* gauges:

    • sessions;
    • refusing;
    • shed level;
    • work p95, p99 and maximum in microseconds;
    • the session layer's share, p95 and p99 (the ADR-0089 per-session budget is checked against it);
    • refused logins (counter).

    evo-server prints m10 load refusing=R shed=L p95_us=P on every change. No labels are added.

Consequences

  • Under overload, honest new players are refused (capacity, retry later) and existing sessions see a smaller, slower-updating view rather than delayed ticks. Crossings continue.
  • The thresholds are provisional until 10.9d measures the reference worker. A change is made by amending this ADR and the budgets table.
  • No change to the session protocol: the client already handles capacity and removals.

10.9a evidence (2026-10-07)

  • Unit tests:

    • the governor's thresholds, one step per second, hysteresis, the calm timer restarting on a non-calm window, tick debt alone, and policy validation;
    • the metrics families, present only with the listener and bounded at worst-case values;
    • replication shedding: radius caps 5 and 3 without a fresh baseline or a change of the stored interest, a far update every 16 ticks at level 3, and the full square back at level 0;
    • sessions under load: grant logins refused with capacity before signature work (the grant stays usable), tickets admitted.
  • Process test (region_m10_session_tests.cpp): a real evo-server with --ops-listen exposes the evo_m10_* gauges.

    Local development reference (WSL, Release, one session; not evidence): work p95 about 7 ms and p99 about 9 ms per tick. Of that, the session layer's p99 was about 4.5 ms. That is far above 0.5 ms per session, so 10.9b examines it with evo-load. Likely contributors are the per-tick snapshot source query and synchronous control renewals.

10.9b load tool (2026-10-07)

  • evo-load (tools/network/load.cpp, quic preset; usage in its header). Bots run on worker threads, each a RegionSessionClient with its own account and a dev grant signed in-process. A bot:

    • sends one intention per server tick at its InputLead (a random walk turning every 2 s) and predicts its marker with MotionPredictor;
    • sends a probe action every 2 s and moves its interest every 10 s (a fresh baseline);
    • logs in again 2 s after any close.

    Steps raise the bot count (--ramp-ms apart), warm up, then hold. Each step reports:

    • pooled p50, p95, p99 and maximum, plus the per-client p95 spread, for responsiveness (input to the first own-marker snapshot applying it), receipt RTT, final-outcome latency, login time and prediction error;
    • snapshot rate, gaps and bandwidth per client;
    • drops, failed logins and refusals;
    • the generator's own CPU;
    • with --metrics, the region's evo_m10_* gauges.

    Starvation is judged among the bots connected during the hold: a snapshot rate under 80 % of the median, or a response p95 over both twice the median and the budget. A step's verdict is within budget only with every bot established, no refusals or drops, no starvation, response p95 within budget (150 ms by default), outcome p95 ≤ 1.5 s, server work p99 ≤ 25 ms, no shedding or refusing, and no tick debt. The run ends with load result sustainable=N saturated_at=M.

  • Smoke (Postgres region_m10_session_tests): 4 then 8 bots against a real evo-server with --metrics. Every bot logs in once with no drops or refusals, both steps measure responses, outcomes and snapshots, and the report is complete.

  • Found while ramping:

    • The process-test fixture's Child::pump(1ms) rounded its wait down to 0 ms and never read. A chatty evo-server then blocked on its full stdout pipe, which looked like a server stall at 32 bots. The wait is now rounded up.
    • The governor's any-debt refusal was too twitchy (see the amended decision above).
  • Local development reference (WSL, Release, one region; not evidence):

    Bots Verdict Work p95 / max Session p95 (per session) Response p95 Outcome p95 Snapshots
    8 within 9.0 / 9.6 ms 1.7 ms (0.22) 106 ms 659 ms 20 Hz, no gaps
    16 within 11.0 / 11.2 ms 3.5 ms (0.22) 102 ms 661 ms 20 Hz
    24 within 13.1 / 13.3 ms 5.8 ms (0.24) 103 ms 664 ms 20 Hz
    32 refusing 14.3 / 17.0 ms 8.4 ms (0.26) 103 ms 665 ms 20 Hz

    At 48 and 64 bots, the bots beyond the 32-session cap fail at the transport connection limit, and the 32 established sessions are unaffected. Prediction errors were 0 (no loss on loopback), and no client starved at any step. The host step dominates the work; the session layer stays near 0.25 ms per session (budget 0.5).

10.9c-1 abuse suite: fuzzing and in-process abuse (2026-10-07)

  • Fuzzing (tests/fuzz/session_fuzz.cpp). For any input:

    • decode_session never crashes, and an accepted message has a stable canonical form (decode, encode, decode, encode gives the same bytes);
    • an accepted snapshot chunk reaches the client's assembler without crashing;
    • GrantVerifier and HandoffVerifier refuse the bytes, as they do the grant slot of a decoded Hello. Each holds a real key, so an acceptance would mean a claim byte was trusted before the signature.

    Two builds:

    • evo-session-fuzz: libFuzzer with ASan and UBSan behind EVO_BUILD_FUZZER, with the session codec, replication, motion, control and verifier sources compiled in so coverage instruments them;
    • session-fuzz-smoke: CTest, label network, on every platform. It feeds 68,000 seeded inputs: random bytes, and bit flips, truncations, extensions and edge-byte overwrites of an encoding of every v4 message type.
  • In-process abuse (quic-sessions). Each test measures an honest session tick by tick while the abuse runs. It must keep at least 90 % of its snapshots and admitted receipts, and the server's connections and retained state must stay bounded:

    • Hello flood: 160 raw connections over 40 ticks, sending garbage (closed with protocol), well-formed Hellos signed by an untrusted key (unauthenticated), or nothing (timeout). Connections stay within the transport bound of 32, and the region returns to the honest connection alone.
    • Datagram flood: an authenticated peer sends hundreds of intentions per tick for any tick. They are throttled per session (one per tick, refused receipts), and garbage then closes the peer (protocol).
    • Slow peer: an established client that never reads or acknowledges anything. The server retains at most its baselines for it (392 entities against a bound of 490).
    • Storm: 120 logins across 12 accounts, half closed politely and half vanishing. The region returns to one session and one connection.
  • Probe: evo-session-probe run --grants FILE [--grant N] logs in with grant N of an existing bundle, such as one from the dev renewal, without an issuer key. It is covered by the M10 process test and was used against evolution-dev, where the login was welcomed and its intention admitted. That region runs without synthetic inputs, as the Git policy requires, so nothing is simulated there outside a drill.

  • Not yet (10.9c-2): lease loss and SIGKILL with 20 or more evo-load bots attached, and a live raw-UDP sender of mutated datagrams. Our client endpoint caps its own send rate, so it cannot load the server's per-connection datagram cap (64, newest wins, with an O(queue) eviction scan); the raw sender measures that.

10.9c-2 authority loss under load (2026-10-08)

  • Lease loss under load (region_m10_session_tests.cpp): 24 evo-load bots on a real evo-server (loopback, fixed port). First the server is SIGKILLed, so its lease must expire before a successor runs. Then it is drained with SIGTERM: sessions close with shutdown and the lease is released. Each time the successor takes the same port under a new epoch, and every bot must log in again within 15 s of it being ready.

    Development reference:

    Loss Successor ready All 24 bots back after ready
    SIGKILL 3.5 s 5.0 s
    SIGTERM drain 0.18 s 2.4 s

    Over the 100 s hold the bots made 72 logins (24 + 2 × 24) and counted 48 drops, with no failed or refused login, and response p95 stayed at 106 ms. After a SIGKILL, clients only notice the loss through the QUIC idle timeout (5 s), which dominates the SIGKILL figure.

  • Scope narrowed: no standalone tool sends raw UDP traffic at a network endpoint. Abuse testing stays in-process or on loopback inside the test suite (10.9c-1). The server's per-connection datagram cap is analysed here instead of being measured with a flood tool. Each connection may hold 64 queued datagrams (newest wins). Evicting the oldest scans the endpoint's event queue (at most 4096 events), which happens only when one connection queues more than 64 datagrams between two server polls (normally under a millisecond apart). That cost is bounded per datagram but can add up at very high rates from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is recorded as a follow-up, not done.

10.9d drill harness (2026-10-08)

  • tools/ops/load-drill.sh (runbook: docs/networking/M10_LOAD_DRILL.md):
    1. turns synthetic inputs on in evolution-dev for the run, the drill-time override of ADR-0073 (never in Git);
    2. runs evo-load through the public route;
    3. matches each step's hold with the server's reports in it;
    4. restores REGION_INPUTS on exit, including on failure.
  • evo-server logs m10 load-report … every 10 s: sessions, work and session-work percentiles, governor state and tick debt. The agent cannot reach the ops port (no port-forward, and the API-server proxy is refused), but it can read logs.
  • Bundle mode:
    • evo-load --grants FILE --grants-per-bot C logs bots in from a pre-issued bundle, so the drill machine never holds the issuer key;
    • evo-session-probe grants --accounts N writes such a bundle for N accounts;
    • each step line now carries exhausted= and hold_utc=.
  • Evidence: the M10 load smoke runs in bundle mode (8 accounts × 2 grants), and the M10 process test checks the load-report line. The drill's summariser was tested on synthetic logs; that test found a window-overlap bug, now fixed.
  • Finding: Mbed TLS is not thread-safe here. Bundle mode made logins fast enough that two bots on different threads verified their first connections at once. Both were refused as untrusted (QUIC cause 3), which surfaced as close_reasons=259:2. Mbed TLS and PSA are built without MBEDTLS_THREADING_C. Every network-library entry point that touches them now holds one process-wide recursive lock (include/evo/network/crypto_lock.hpp): the QUIC client's pinned check and the session-grant and handoff-ticket keys. The smoke failed 2 of 2 runs before and passed 3 of 3 after. quic-transport gains a concurrent-connect test (8 threads, 5 rounds); it does not reliably reproduce the race, because PSA is already initialised there.
  • evo-load step lines carry close_reasons=REASON:COUNT,…: session close reasons, or 256 plus the QUIC cause when there was no session Close.
  • Not yet: the reference run. It needs an image containing this change promoted to dev, a load bundle made where the issuer key lives, and the maintainer's go-ahead, because the drill restarts the dev region twice.

10.9d first reference run (2026-10-08, evolution-dev, image 6c96629e…)

Drill with steps 20, 24, 28 and 32 bots, 60 s holds and 4 grants per bot, run from the maintainer's workstation over the LAN route. Synthetic inputs were on from 14:33:46 to 14:38:51 UTC; the region was restored (REGION_INPUTS empty, epoch 60, Synced and Healthy), and committed history advanced from 12,084 to 18,142 ticks.

It did not measure capacity:

  • The governor tripped six seconds after the restart, with no session. It went refusing=1 shed=1 at p95 24.4 ms over a window holding only the restart's first ticks (cold caches, first commits).
  • The bots burned their grants. They retried every 2–2.5 s, so 19 of 20 used all 4 grants within about 10 s (76 refusals: 63 capacity, 13 rate_limited). Only one session was measured, and the later steps added nobody.
  • The drill's live log follow ended at 14:34:19, so it holds 4 server reports.

Measured with 1 session, the first reference numbers:

  • tick work: p95 4.0 ms, p99 and maximum 33 ms (isolated spikes);
  • session layer: p95 0.68 ms, p99 3.0 ms;
  • client side: response p95 111 ms, outcome p95 1.13 s, snapshots 20 Hz.

The spikes exceed the 25 ms p99 budget and need to be attributed (commit flush or snapshot rebuild) in the next run.

Fixes before the next run:

  • Governor: work counts only over a full 40-tick window; tick debt still counts at once (unit test added).
  • evo-load: a login refused for load backs off 10 s, and in bundle mode reuses its grant, which the region refused before using it.
  • Drill:
    • waits for a calm load-report before the bots start;
    • takes the drill pod's whole log before restoring;
    • waits after restoring for a Ready pod other than the drill pod (the old check could accept a pod still running during the rollout);
    • checks that REGION_INPUTS is empty again.

10.9d second reference run (2026-10-08, image 7952185d…)

Same drill (steps 20/24/28/32, 60 s holds), 18:21–18:26 UTC, restored and verified.

  • Client side, 20 sessions: all established, no drops, no starvation; response p95 111 ms (LAN budget 150); outcome p95 1.18–1.21 s (budget 1.5); snapshots at a median of 19.8–20 Hz. No login beyond 20 succeeded.

  • Server side, 20 sessions:

    Measure Value Budget
    Tick work p95 8–16 ms (one report 27 ms) —
    Tick work p99 48–70 ms 25 ms
    Session layer p95 about the whole tick work —
    Session layer p99 23–41 ms, 1.2–2.0 ms per session 0.5 ms per session

    There was no tick debt. The governor refused from 18:22:15 and shed to level 3 from 18:22:42, which barely lowered the cost.

  • The startup trip happened again, briefly. The first calm report came 4 s after the restart, and the governor (now over a full window of startup ticks) refused the first logins until 18:21:36. The drill now waits for two consecutive calm reports.

  • Cause: ReplicationServer::tick encoded every environment cell and organism again for each session, about 225 records × 20 sessions each tick.

  • Fix: sessions with the same requested interest and no perception predicate share one encoding per tick. Markers, and everything when perception filters, stay per session. Behaviour is unchanged: the existing replication tests pass, and a shared-interest test checks 20 sessions on one square and one on another. Measured locally, 21 sessions went from about 970 µs to about 360 µs per tick (2.7×).

  • Next: a third run with the fix. The remaining per-session cost is the comparison with each session's acknowledged baseline.

10.9d third reference run and decisions (2026-10-08, image f378a43d…)

The renewal had rotated the dev CA, but the running pod still held the old server certificate. A rollout restart loaded the new one first (a throwaway-key login checked the trust), then the drill ran at 21:00–21:05 UTC and was restored.

  • 24 sessions held with every client budget met: response p95 111 ms; 20 Hz; no drops or starvation. The governor (refusing at p95 ≥ 15 ms) refused logins beyond 24.
  • Server at 24 sessions:
    • tick work p95 9–16 ms, p99 25–34 ms;
    • with 0 sessions the p99 is already 23–25 ms: the synchronous commit flush;
    • session layer 0.62–0.69 ms per session at p99, 0.35–0.5 at p95;
    • no tick debt.
  • Outcome latency in this run is invalid. The bundle reused the earlier runs' accounts and the bots numbered actions from 1, so every action was a duplicate answered at once with its recorded outcome (16 ms). evo-load now numbers actions from the current time and counts duplicate_actions apart from outcome latency.

Maintainer decisions (2026-10-08):

  1. Budget amended. Per-tick region work is p95 ≤ 25 ms over all ticks, with ticks that commit a batch ≤ 50 ms, measured separately (schemas/network/m10-budgets-v2.md). Moving the commit off the simulation thread is a recorded follow-up.
  2. Optimize first, then declare the measured sustainable session count.

Profile and second optimisation (local, 24 bots):

  • The load-report now splits the session layer into poll and replicate and gives commit-tick and other-tick maxima.
  • poll cost about 1.3 ms per tick against 0.45 ms for replicate. Probes inside poll (temporary, not committed) attributed it to datagram sends of about 15 µs each, almost all in msquic's DatagramSend: a receipt and a watermark per session per tick, plus snapshot chunks.
  • Snapshots carry the server tick every tick (10.5), so the per-tick watermark is no longer needed for the client's clock. An advancing watermark is now sent at most every 4 ticks (IntentLimits::watermark_coalesce; evo-server uses 4, the library default stays 1). Watermarks dropped from 4,800 to 1,200 per 10 s and poll from about 1.36 to 1.05 ms per tick. Receipts stay one per intention (ADR-0092).
  • Further gains need fewer datagrams per tick, which is a protocol change and out of scope here.
  • The drill summary now judges by budgets v2 and prints commit and other-tick maxima and the poll and replicate averages.

10.9d fourth reference run (2026-10-09, image 754446fe…)

Drill at 01:00–01:06 UTC, restored. It is the first run whose actions were all new (no duplicates), so the first in which actions actually committed (about 12 per commit, one commit about every 1.05 s).

  • Client side: 24 sessions within every client budget (response p95 111 ms; 20 Hz; no starvation; duplicate_actions=0). Outcome p95 about 1.2 s: the action waits for the commit of the batch it rides. The governor refused logins beyond 24.
  • Server side, not within budgets v2:
    • commit ticks 58–70 ms (budget 50);
    • session-layer p99 30–39 ms (1.5–1.6 ms per session), and 3–5.6 ms with a single session;
    • other ticks at most 12–19 ms; no tick debt.
  • Cause: a synchronous PostgreSQL round trip per action on the simulation thread. Admission looked up each new action's durable record (uniqueness across epochs, processes and regions), and each commit looked up every committed action again, so the first-record rule could hold. About 12 lookups land on each commit tick (≈ 35 ms on the dev database), and each incoming action costs one round trip on whichever tick it arrives. Locally a round trip is a fraction of a millisecond, which is why the profile missed it. Drill 3 never reached the commit path.
  • Fix (no schema change):
    • ActionLedger::find_all answers many (account, action) pairs in one statement: the schema v11 find_region_action, lease check included, applied to each pair through unnest … CROSS JOIN LATERAL.
    • IntentAdmission::admit holds an action that only the ledger can still refuse, and one verify per poll confirms every waiting action with one request. Its receipt is still sent in the same poll.
    • committed asks once per commit.
    • The store cost is now at most one round trip per poll plus one per commit, whatever the number of sessions or actions.
  • Next: a fifth run with this fix, then the declaration.

10.9d fifth reference run (2026-10-09, image c597c0ba…)

Drill at 11:28–11:33 UTC, restored. All actions were new, and about 12 committed per commit.

Sessions Client budgets Work p95 (≤ 25) Commit ticks (≤ 50) Other ticks Session p99 Per session p99 (≤ 0.5) Governor
20 met (4 ramp refusals) 11.1 ms 36.4 ms 11.1 ms 11.1 ms 0.554 ms refused during the login ramp only
24 met 10.4 ms 36.9 ms 12.9 ms 11.3 ms 0.471 ms calm
28 met 12.5 ms 44.3 ms 12.8 ms 13.3 ms 0.473 ms calm
32 met 12.9 ms 41.5 ms 14.5 ms 11.2 ms 0.351 ms calm

Client budgets throughout:

  • response p95 111 ms;
  • snapshots 20 Hz with no gaps;
  • outcome p95 1.13–1.14 s (the action waits for the commit of its batch);
  • no starvation and no duplicate actions.

Notes on the run:

  • Batching worked. Against the fourth run, commit ticks fell from 58–70 ms to 36–44 ms and session p99 from 30–39 ms to 11–13 ms. Tick debt stayed 0 throughout.
  • The N=20 refusals were the login ramp, not steady state. Sixteen bots logged in within 8 s, and each login does synchronous store work (control claim, probe open). The window's p95 reached 15.2 ms and the governor refused 4 logins (11:28:25–28). Those bots retried 10 s later and all 20 established. evo-load counts any refusal as saturation, so it reported sustainable=0 saturated_at=20; the per-step results above are the evidence.
  • Session teardown at the end shed briefly. All 32 sessions closed at once (each close does a release and a probe save; p95 35.6 ms), after the measured hold.
  • N=20 misses the per-session budget slightly (0.554 ms against 0.5). The session layer's fixed per-tick cost is shared by fewer sessions there.
  • 32 is the configured session cap (SessionControl::max_sessions, the budgets' "32 default"). The region reached it within every per-tick budget without saturating. Commit ticks are the tightest margin (44.3 of 50 ms).

Open for 10.9's completion criteria, a maintainer decision:

  • measured saturation beyond the cap;
  • native-client observation during a drill.

Follow-ups:

  • login and close store work off the simulation thread, or batched;
  • the commit off the tick (already recorded).

Maintainer decisions (2026-10-09), closing 10.9 for now:

  1. Session cap raised to 2000 in evo-server (m10_max_sessions, plus 64 connections of headroom). The library bounds go from 1024 to 4096 (SessionControl, IntentAdmission, ReplicationServer); the library defaults stay 32. The cap is now a safety bound, and the load governor (refuse, then shed) is the working limit.
  2. No more drills for now. Saturation and native-client observation move to a later 2k drill. The maintainer will watch that drill with the native client.

Declared, on the evolution-dev reference worker:

  • Sustainable: at least 32 sessions within every budget (fifth run). That is a measured lower bound, not a saturation point.
  • Overload response: measured as designed. The governor refused logins during a ramp burst and shed during a mass close, and no tick debt built up.

What the 2k drill has to face (estimates, not measurements):

  • Per-tick CPU. The session layer costs 0.35–0.55 ms per session per tick on one thread against a 25 ms p95 budget. The governor would refuse somewhere around 50–70 sessions, well short of 2000. 2000 sessions in one region need a different architecture (session work off the simulation thread, several regions per world, or both). That calls for its own ADR before the drill.
  • Login rate. ControlConfig::max_region_attempts is 64 per 60 s region-wide, so a 2000-session ramp would take about 31 minutes. Whether to raise it is an abuse-budget decision for that drill.
  • Memory. The evo-region pod is limited to 256 MiB, and the governor bounds CPU, not memory. Each session's baselines, msquic state and per-session queues need measuring before thousands of sessions are admitted.
  • Linear scans. Sessions and actions are found by linear search; that is fine at tens of sessions, not at thousands.