ADR-0097. evo-load runs bots on worker threads, each a RegionSessionClient
with its own account and an in-process dev grant: one intention per tick
at its input lead with marker prediction, actions every 2 s, interest
moves every 10 s, and re-login after any close. Stepped ramps report
pooled and per-client distributions (response, receipt, outcome, login,
prediction error), snapshot rate, gaps and bandwidth, drops, failed
logins and refusals, generator CPU, a starvation check among connected
bots, a budget verdict and the region's evo_m10_* gauges, ending with the
sustainable bot count. A Postgres process smoke runs 4 then 8 bots
against a real evo-server.
Fixes found while ramping: the process fixture's short pumps never read
(a chatty child blocked on its stdout pipe), and the load governor
refused on a single tick of debt (now 2 refuse, 4 shed). Refreshes the
dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record the maintainer-stated host facts (R730xd ~128 GB, 32 GB workers,
automatic Harbor publish and Argo CD deploy, existing aevolution-db
Secret and its conflicts with the pinned version and role split), the
measured evo-server profile (peak RSS ~9.7 MiB, ~0.3% of a core) and a
local storage baseline. Add a read-only cluster collector that never
fetches Secret data and a by-hand storage latency probe outside the
Argo path. Cluster facts remain uninspected pending read-only access.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0065: before any readiness, recovery now verifies the head's tail.
The trace object must exist with its manifest digest, its header must
bind profile, region, predecessor and range; the predecessor (schema v6
load_region_commit) must hash to the digest the head names; replaying
the trace from the predecessor's verified payload must reproduce every
report and the exact head bytes; and trace-derived decisions must equal
the head's durable decisions. Missing, corrupt or non-reproducing tails
quarantine (RecoveryError::tail); nothing older or empty is served.
Test-only crash seams (HostConfig::crash_hook, PublishHooks) and
evo-server --crash-at deliver real SIGKILL at before-upload,
before-transaction, after-commit and after-ack. The kill matrix,
a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as
the durability contract requires; successors verify before readiness.
Validated: PostgreSQL suites under GCC Release, clang-tidy and
ASan/UBSan; tail-skip and replay-equality mutations each caught. No
Docker/Gitea CI, production-lease kill test or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>