Fix load governor startup refusal and drill issues from the first dev run
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s

ADR-0097. The first evolution-dev load drill (image 6c96629e) did not
measure capacity: the governor refused six seconds after the restart with
no session (p95 over a window of only startup ticks), and the bots spent
their grants retrying every 2 s. Reference numbers with one session: tick
work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms.

- The governor acts on work only over a full 40-tick window; tick debt
  still counts at once.
- evo-load backs off 10 s after a load refusal and, in bundle mode,
  reuses the grant the region refused before using it.
- The drill waits for a calm load-report before the bots start, takes the
  drill pod's whole log before restoring (the live follow died), and
  confirms the restore against a pod other than the drill pod.

Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 11:01:02 -04:00
co-authored by Claude Opus 5.5
parent 9b868b3ae2
commit d173f076ea
7 changed files with 120 additions and 21 deletions
+15 -6
View File
@@ -13,8 +13,9 @@
// process). Bots run on T worker threads (each owns a slice). Every server tick a bot sends
// one intention for tick + its input lead (a random walk turning every 2 s) and predicts its
// probe marker; every 2 s it sends a probe action; every 10 s it moves its interest by a
// cell (a fresh baseline). A closed or refused bot logs in again after 2 s with the next
// serial.
// cell (a fresh baseline). A closed bot logs in again after 2 s with the next serial; one
// refused for load (capacity, rate_limited) waits 10 s and, in bundle mode, reuses the same
// grant (the region refused it before using it).
// M10.9d: with --grants (a bundle from `evo-session-probe grants --accounts N --count C`,
// e.g. made where the dev issuer key lives), bot i logs in with grants i*C .. i*C+C-1 in
// order instead of signing; a bot that used all C stays out (counted as exhausted).
@@ -314,12 +315,20 @@ class Load {
++b.s.closes[256U + static_cast<unsigned>(*cause)];
else
++b.s.closes[255U];
if (c.close_reason() == SessionCloseReason::capacity ||
c.close_reason() == SessionCloseReason::rate_limited)
const bool refused = c.close_reason() == SessionCloseReason::capacity ||
c.close_reason() == SessionCloseReason::rate_limited;
auto backoff = std::chrono::seconds{2};
if (refused) {
++b.s.refused;
// Refused for load before the grant was used: the region is shedding, so back
// off longer, and a bundle grant stays usable for the next try (10.9d).
backoff = std::chrono::seconds{10};
if (!key_ && b.used > 0)
--b.used;
}
b.c.reset();
b.retry_at = now + std::chrono::seconds{2} +
milliseconds{std::uniform_int_distribution<int>{0, 500}(b.rng)};
b.retry_at =
now + backoff + milliseconds{std::uniform_int_distribution<int>{0, 500}(b.rng)};
return;
}
if (c.state() != client::RegionSessionState::established)