The fourth dev drill showed a synchronous PostgreSQL round trip per action on the
simulation thread: at admission (uniqueness) and again at commit (first-record rule).
About 12 lookups landed on each commit tick (commit ticks 58-70 ms, session p99 30-39 ms).
- ActionLedger::find_all; persistence::find_region_actions applies the schema v11
find_region_action to many pairs in one statement (no migration).
- IntentAdmission::admit/verify: actions only the ledger can refuse wait for one verify
per poll; receipts still go out in the same poll. committed() asks once per commit.
- Tests: deferred_actions (network_control), find_region_actions and find_all (Postgres).
- ADR-0097 fourth run, ADR-0092 amendment, VALIDATION; refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- IntentLimits::watermark_coalesce (evo-server: every 4, coalesce 4); ADR-0092 amendment.
- m10 load-report splits poll/replicate and commit-tick/other-tick maxima.
- Budgets v2: work p95 <= 25 ms over all ticks, commit ticks <= 50 ms (ADR-0097);
moving the commit off the tick is a recorded follow-up.
- evo-load: time-based action ids, duplicate_actions, receipts before outcomes;
drill ramps at 500 ms and judges by v2.
- Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. The second evolution-dev load drill (image 7952185d) held 20
sessions with client budgets met (response p95 111 ms, outcome p95 about
1.2 s, 20 Hz, no starvation) but the server over its own: tick work p99
48-70 ms (25 ms budget), session layer 1.2-2.0 ms per session at p99
(0.5 ms budget), governor refusing and shedding to level 3 to little
effect. ReplicationServer::tick re-encoded every environment cell and
organism for each session. Sessions with the same requested interest
(no perception predicate) now share one encoding per tick; markers stay
per session. Locally 21 sessions: about 970 us -> 360 us per tick.
The drill now waits for two consecutive calm reports before the bots.
The dev overlay's REGION_RETAINED listed the deployed build itself
(61df6a1, f88bee2), which the deploy policy refuses (CI deploy failed on
those commits); it now lists only the other builds, keeping 6c96629e,
the build of the region head. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. The first evolution-dev load drill (image 6c96629e) did not
measure capacity: the governor refused six seconds after the restart with
no session (p95 over a window of only startup ticks), and the bots spent
their grants retrying every 2 s. Reference numbers with one session: tick
work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms.
- The governor acts on work only over a full 40-tick window; tick debt
still counts at once.
- evo-load backs off 10 s after a load refusal and, in bundle mode,
reuses the grant the region refused before using it.
- The drill waits for a calm load-report before the bots start, takes the
drill pod's whole log before restoring (the live follow died), and
confirms the restore against a pod other than the drill pod.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.
Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b2f224d set REGION_SESSION_WINDOW to expire about 5.2 hours after the pod
started; evo-server refuses windows longer than 4 hours
(SessionWindow::max_duration_seconds) and exited with
"error m10-window-too-long" in a loop. The window now expires 3 h 50 min
after it was written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. 24 evo-load bots against a real evo-server on a fixed loopback
port: a SIGKILL (the successor waits out the lease) and then a SIGTERM
drain, each followed by a successor under a new epoch. Every bot logs in
again within 15 s of the successor being ready (development reference:
5.0 s after SIGKILL, 2.4 s after drain), with 48 drops and no failed
logins counted over the hold.
Scope narrowed: no standalone raw-UDP sender; abuse stays in-process or
on loopback, and the per-connection datagram cap's eviction cost is
analysed in the ADR. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
checks (postgres) failed on both M10.9 pushes: on the shared CI runner one
session measured 22 ms p95 per tick, so the governor correctly refused
and the test's assumption about host speed failed (server-image was
skipped as a result). evo-server gains --m10-load-governor on|off (dev
only): off keeps measuring and exposing the work but never refuses or
sheds. The M10 session, load-smoke and two-process handoff tests use it;
the governor stays covered by its unit tests. Refreshes the dev session
window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. evo-load runs bots on worker threads, each a RegionSessionClient
with its own account and an in-process dev grant: one intention per tick
at its input lead with marker prediction, actions every 2 s, interest
moves every 10 s, and re-login after any close. Stepped ramps report
pooled and per-client distributions (response, receipt, outcome, login,
prediction error), snapshot rate, gaps and bandwidth, drops, failed
logins and refusals, generator CPU, a starvation check among connected
bots, a budget verdict and the region's evo_m10_* gauges, ending with the
sustainable bot count. A Postgres process smoke runs 4 then 8 bots
against a real evo-server.
Fixes found while ramping: the process fixture's short pumps never read
(a chatty child blocked on its stdout pipe), and the load governor
refused on a single tick of debt (now 2 refuse, 4 shed). Refreshes the
dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097 (M10.9 load and abuse qualification; only evolution-dev
reference-worker runs qualify). evo-server measures each simulated tick's
work (host step plus the M10 bridge, with the session layer's share kept
separately) and feeds a portable LoadGovernor: grant logins are refused
at a 2 s p95 of 15 ms or any tick debt, replication is shed in up to
three levels at 20 ms or debt >= 2 (radius cap 7/7/5/3, far updates
x1/x2/x2/x4, no fresh baselines, own markers unaffected), and both relax
after 5 s of calm. Handoff tickets are still admitted. /metrics gains
evo_m10_* gauges when the listener is enabled.
Adds the missing ADR index entries for ADR-0094 to ADR-0096. Refreshes
the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.
The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
abort deadline was in memory): schema v14 overdue_probe_transfers and a
session-layer sweep abort open outgoing transfers past the ticket
lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
took control back: admission of a committed transfer now requires the
record as that commit left it.
M10.8 is complete. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. The engineering client follows a HandoffOffer into the next region
with its ticket (no grant, no pause, the view kept) and a Redirect to the
probe's owner, picking grants by region claim in rising serial order.
evo-server gains --m10-area, --m10-handoff-key and --m10-neighbour (dev
only), wiring the PostgreSQL probe and handoff ledgers; the probe tool writes
multi-region bundles and reports handoffs and redirects.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Session protocol v4 (v3 is deployed and frozen) adds HandoffOffer,
SessionRedirect, close reasons handed_off and moved, and ghost marker records.
Sessions use the durable probe record: logins open it where it was saved
(redirected when it lives elsewhere), and session ends save it. A marker
crossing into a neighbour's area is frozen, prepared, its control released
and a signed ticket offered; uncommitted transfers are aborted after the
ticket's lifetime. The destination shows arriving probes as ghosts (schema
v13 incoming_probe_transfers) and opens the marker where it crossed.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP
7400, so nothing reached it through the VIP. The route container port, Service and
NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check
requires exactly UDP 7400 for the route and counts any protocol on 7400 as session
exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and
internal DNS as separate reviewed steps, end-to-end probe as acceptance);
session protocol v1 recorded as frozen now that it is deployed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>