Fix load governor startup refusal and drill issues from the first dev run
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s
ADR-0097. The first evolution-dev load drill (image 6c96629e) did not measure capacity: the governor refused six seconds after the restart with no session (p95 over a window of only startup ticks), and the bots spent their grants retrying every 2 s. Reference numbers with one session: tick work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms. - The governor acts on work only over a full 40-tick window; tick debt still counts at once. - evo-load backs off 10 s after a load refusal and, in bundle mode, reuses the grant the region refused before using it. - The drill waits for a calm load-report before the bots start, takes the drill pod's whole log before restoring (the live follow died), and confirms the restore against a pod other than the drill pod. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -42,8 +42,13 @@ bool LoadGovernor::observe(std::uint64_t work_us, std::uint64_t session_us, std:
|
||||
state_.session_p99_us = rank(sessions, n, 99);
|
||||
|
||||
const auto before = std::pair{state_.refusing, state_.shed};
|
||||
const bool hot = state_.p95_us >= policy_.refuse_us || tick_debt >= refuse_debt_ticks;
|
||||
const bool overloaded = state_.p95_us >= policy_.shed_us || tick_debt >= shed_debt_ticks;
|
||||
// Work counts only over a full window: after a (re)start the first ticks (cold caches, the
|
||||
// first commits) would otherwise be the p95 alone (seen on evolution-dev, 10.9d). Tick
|
||||
// debt is acted on at once.
|
||||
const bool full = n == window_.size();
|
||||
const bool hot = (full && state_.p95_us >= policy_.refuse_us) || tick_debt >= refuse_debt_ticks;
|
||||
const bool overloaded =
|
||||
(full && state_.p95_us >= policy_.shed_us) || tick_debt >= shed_debt_ticks;
|
||||
if (hot)
|
||||
state_.refusing = true;
|
||||
if (overloaded && state_.shed < policy_.max_shed &&
|
||||
|
||||
Reference in New Issue
Block a user