Movement and ingestion are stored per second in schema 2 (ranges x20). Schema 1 records
stay valid, are never rewritten, and read as exactly x20 via Genome::per_second; drafts
default to schema 1 so existing literals keep their meaning. Development version 1
converts genome rates at its own 50 ms quantum (round down), so phenotypes are
unchanged. Mutation keeps the parent's schema and step; the lineage archive header names
exactly the highest schema held (all-schema-1 archives keep their bytes); the
development-v1 opening names schema 1. No golden changed; two tests that used 2 as an
unknown schema now use the next one. Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rates are stored per second in their integer base units and converted at the tick
boundary with no carried state: costs round up, capacities down (maintainer decision),
error below one unit per tick. per_second_from_per_tick is the migration's exact inverse
(x20 at 50 ms) and refuses inexact values. New rate-contracts test. No biology uses it
yet; genome v2, development v2 and process fields follow. Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A stale_grant before Welcome spends only that grant: the session tries the next one
in the bundle and ends as exhausted when none is left (other refusals still stop).
- UI scale (ADR-0084 amendment): embedded scalable font (ProggyForever), scale from
logical window height (720 reference, 1.0-2.5) times a Ctrl+= / Ctrl+- nudge; panel
sizes and table columns in logical units; movable panels placed again per scale.
- --size W,H for engineering review screenshots.
- Tests: client_remote_session (stale grant, exhausted), client_ui_state (scale).
Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The connect form paired the dev endpoint with the local test PKI's name
(dev-region.evolution.test), so a default connection to the dev region failed the
certificate name check. client_ui_model checks the new default; the tester doc names it.
Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Maintainer decision (2026-10-09): the region session cap becomes a safety bound of 2000
(plus 64 connections of headroom); the load governor is the working limit. Library bounds
go from 1024 to 4096 in SessionControl, IntentAdmission and ReplicationServer (defaults
stay 32). Drills stop for now: saturation and native-client observation move to a later
2k drill, whose prerequisites ADR-0097 lists (single-thread CPU, region login rate,
256 MiB pod limit, linear lookups). Declared: >= 32 sessions within every budget on the
reference worker. Budgets v2, STATUS, ROADMAP, VALIDATION. Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fourth dev drill showed a synchronous PostgreSQL round trip per action on the
simulation thread: at admission (uniqueness) and again at commit (first-record rule).
About 12 lookups landed on each commit tick (commit ticks 58-70 ms, session p99 30-39 ms).
- ActionLedger::find_all; persistence::find_region_actions applies the schema v11
find_region_action to many pairs in one statement (no migration).
- IntentAdmission::admit/verify: actions only the ledger can refuse wait for one verify
per poll; receipts still go out in the same poll. committed() asks once per commit.
- Tests: deferred_actions (network_control), find_region_actions and find_all (Postgres).
- ADR-0097 fourth run, ADR-0092 amendment, VALIDATION; refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- IntentLimits::watermark_coalesce (evo-server: every 4, coalesce 4); ADR-0092 amendment.
- m10 load-report splits poll/replicate and commit-tick/other-tick maxima.
- Budgets v2: work p95 <= 25 ms over all ticks, commit ticks <= 50 ms (ADR-0097);
moving the commit off the tick is a recorded follow-up.
- evo-load: time-based action ids, duplicate_actions, receipts before outcomes;
drill ramps at 500 ms and judges by v2.
- Refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.
Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. evo-load runs bots on worker threads, each a RegionSessionClient
with its own account and an in-process dev grant: one intention per tick
at its input lead with marker prediction, actions every 2 s, interest
moves every 10 s, and re-login after any close. Stepped ramps report
pooled and per-client distributions (response, receipt, outcome, login,
prediction error), snapshot rate, gaps and bandwidth, drops, failed
logins and refusals, generator CPU, a starvation check among connected
bots, a budget verdict and the region's evo_m10_* gauges, ending with the
sustainable bot count. A Postgres process smoke runs 4 then 8 bots
against a real evo-server.
Fixes found while ramping: the process fixture's short pumps never read
(a chatty child blocked on its stdout pipe), and the load governor
refused on a single tick of debt (now 2 refuse, 4 shed). Refreshes the
dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097 (M10.9 load and abuse qualification; only evolution-dev
reference-worker runs qualify). evo-server measures each simulated tick's
work (host step plus the M10 bridge, with the session layer's share kept
separately) and feeds a portable LoadGovernor: grant logins are refused
at a 2 s p95 of 15 ms or any tick debt, replication is shed in up to
three levels at 20 ms or debt >= 2 (radius cap 7/7/5/3, far updates
x1/x2/x2/x4, no fresh baselines, own markers unaffected), and both relax
after 5 s of calm. Handoff tickets are still admitted. /metrics gains
evo_m10_* gauges when the listener is enabled.
Adds the missing ADR index entries for ADR-0094 to ADR-0096. Refreshes
the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.
The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
abort deadline was in memory): schema v14 overdue_probe_transfers and a
session-layer sweep abort open outgoing transfers past the ticket
lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
took control back: admission of a committed transfer now requires the
record as that commit left it.
M10.8 is complete. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Session protocol v4 (v3 is deployed and frozen) adds HandoffOffer,
SessionRedirect, close reasons handed_off and moved, and ghost marker records.
Sessions use the durable probe record: logins open it where it was saved
(redirected when it lives elsewhere), and session ends save it. A marker
crossing into a neighbour's area is frozen, prepared, its control released
and a signed ticket offered; uncommitted transfers are aborted after the
ticket's lifetime. The destination shows arriving probes as ghosts (schema
v13 incoming_probe_transfers) and opens the marker where it crossed.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. A source region's handoff key signs a 160-byte ticket for one probe
transfer, account and destination; it rides the Hello grant slot and is
verified against neighbour keys bound to their regions. SessionControl
admits a ticket once per region, superseding like a grant, with the session
ending at the client's original grant expiry. RegionSessions makes the
destination's durable step first: PostgresHandoffLedger stages and commits
the transfer and moves the account's control, idempotently.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable
probe records hold one owning region per account, so ownership is unique and
the test inventory is conserved by construction. A fenced transfer state
machine (prepared, staged, committed, aborted) under each caller's exact
region lease returns the recorded outcome on every retry; a commit requires
staging by the destination's current epoch; aborts are durable and only
before commit. handoff_account_control moves account control on a committed
transfer.
A PostgreSQL fault matrix covers restarts at each transition, stale epochs,
failed destinations, partitions and commit/abort races, checking one owner and
the conserved inventory after every scenario.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0095. The engineering client times presentation with the tick clock:
its own marker between ticks, other markers interpolated a few ticks behind
(grey when stale). Silence beyond 3 ticks and long-gap holds are stale. A lost
session leaves its last view frozen and labelled stale until the reconnect's
fresh baseline; nothing is predicted or sent meanwhile.
evo-motion-trace --remote measures remote-marker error between two clients
through the relay, with a 1.5 s outage (motion-trace-remote-smoke CTest).
Prediction errors are recorded only for ticks inputs are scheduled for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0095 (presentation only): a drift-corrected monotonic tick clock that
never runs more than one tick past the newest observation; remote probe
markers shown 2-6 ticks behind (adaptive to snapshot gaps), interpolated,
extrapolated for at most 2 ticks and then marked stale; appearance and
removal at their snapshot ticks; organisms never extrapolated; sub-tick
own-marker prediction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-motion-trace drives the engineering client's own session code through
the owned impairment relay against a real-time 50 ms loopback region:
exact without loss, 94-99 % exact at 60-200 ms RTT with 2-5 % loss, every
profile converging after stopping, restart resync about 2-3 s
(docs/networking/M10_PREDICTION_TRACES.md). The CTest motion-trace-smoke
checks convergence, bounds and a majority of exact predictions.
The relay lets a new sender take over after the current client has been
silent for 1 s (reconnects); the client counts refusal receipts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EXPERIMENT (ADR-0094). RegionSessions owns the probe markers: admitted
intentions schedule them, each poll advances them to the simulated tick, and
snapshots carry them. evo-server sets the motion area from the region's
habitat sites; the probe prints its own marker.
The engineering client predicts its marker with the adaptive input lead,
rebases on each snapshot, resends a lost or refused intention, smooths
corrections, draws markers and follows its own with camera and interest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EXPERIMENT (ADR-0094), not an organism movement rule: one non-biological
marker per session, moved by admitted intentions with a shared integer rule
(probe-motion-v1, 100 mm per tick, clamped to the area), held in region
memory only.
- ProbeMarkers: spawn, area, bounded per-marker schedule, catch-up bound
- session protocol v3: marker records in snapshot chunks; the own marker is
always sent with its applied sequence and area; v1 and v2 are refused
- MotionPredictor (bounded replay from the authoritative marker, corrections,
error samples), CorrectionSmoother and InputLead
- tests: exact prediction with enough lead, refusal and lead adaptation,
lost inputs, bounds, marker records through the codec
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-client's remote mode now runs over RegionSessionClient (QUIC, session
v2): snapshot-fed environment and organism views, the camera following the
server's view until the user moves it and then driving the interest, intention
records with a simulated state, and reconnects with the next grant of a bundle.
The launcher reads 160-byte M10 grants and defaults to the dev route.
- quic::resolve_ipv4 shared by the client and evo-session-probe
- evo-session-probe grants: owner-only bundles of 1-64 grants
- evo-server --m10-max-grant-lifetime (dev only, 60..3600 s)
- presets client-network/client-package build QUIC; msquic links the CRT
fully static; msquic.dll and msquic/OpenSSL licences in the Windows package
- client-remote-session test against a loopback region
- ADR-0093/0091/0086 amendments, M10_CLIENT_SESSIONS guide, validation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RegionSessions owns the replication server: interest and acks per session, a
host-set default interest, and per-tick snapshots sized to each connection's
datagram limit. evo-server builds each interest's view from the region's own
habitat sites (default: the first site, radius 7). RegionSessionClient assembles,
acknowledges and presents snapshots and can change its interest; the probe prints
them. Loopback and real-process evidence. ADR-0093.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session protocol v2 adds Interest, SnapshotChunk and SnapshotAck (v1 is refused).
ReplicationServer filters by a server-capped client focus and a perception
predicate, sends per-tick entity deltas against the acknowledged baseline (kept
up to 40 ticks), nearest first within 16 KiB with a slower distant tier, as
self-contained datagram chunks. SnapshotAssembler applies complete snapshots to
one of two baselines and validates them as presentations. ADR-0093.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-session-probe resolves a host name to IPv4 and prints the address DNS answered,
and its closed line names why the QUIC connection ended (untrusted: the server
answered but failed the pinned check; idle: nothing answered), via the new
RegionSessionClient::transport_cause. Records the UDP route reachability check
(not yet the authenticated acceptance run).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-server gains an opt-in M10 listener (--m10-*, QUIC builds only): RegionSessions
with the PostgreSQL control and action ledgers on their own connection under the
host's lease, carried actions submitted to the host and its acknowledgements and
drops turned into outcomes. evo-session-probe is a dev-only headless client. Real
process tests cover intentions, durable action outcomes, SIGKILL before and after
an action's commit, duplicate retries and the probe. The simulated watermark now
reaches every session every tick, so clients always know the server tick.
ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each batch's actions are recorded in the same statement as its checkpoint: one
region_actions row per (world, account, action id) and the block in an append-only
table, compared on exact retries and resolution. The region host attaches actions to
the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome
batches unresolved. Outcomes come from the durable record via ActionLedger, so
crashes before or after a commit, unknown publications and cross-region moves all
resolve to one answer; a repeated id keeps its first record instead of stalling a
region. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session protocol v1 (not yet deployed, amended in place) gains realtime intentions on
datagrams with receipts and a cumulative simulated watermark, and account-scoped
actions with stable ids, receipts, final outcomes and queries. IntentAdmission
enforces session binding, rate, authority, epoch, vector, sequence, tick window,
duplicates and the unresolved bound; RegionSessions and the client carry it over
QUIC. Intentions and actions have no biological effect yet. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration v10 keeps, per world and account, the highest accepted grant serial and
the region session in control with a fencing generation and database-time expiry,
reachable only through claim/renew/release functions under the caller's region
lease. Serials survive restarts and apply across regions; a cross-region takeover
returns a wait so control never overlaps. RegionSessions takes durable control via
a ControlLedger before admitting a session, delays Welcome by the wait, renews, and
closes sessions whose control is fenced or cannot be confirmed in time.
PostgresControlLedger implements the ledger. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session protocol v1 session messages (Hello with grant, Welcome, Refresh,
RefreshResult, Close with reasons) on the QUIC reliable stream. RegionSessions
verifies grants with the issuer public keys and admits them through SessionControl:
a newer grant closes the account's old connection as superseded, expiry and epoch
advances close sessions, every close is announced and flushed. RegionSessionClient
logs in, refreshes in place and reports why a session ended. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Provider-independent 128-bit AccountIds; ECDSA P-256 session grants v1 signed by
issuer keys and verified by regions with public keys only, the environment byte
signed so dev grants never verify in stage or World0. Per-account serials make the
newest grant win: replay and older stolen grants are refused, a newer grant replaces
the account's session, and refresh renews in place. Attempt rates and capacities
are bounded. The evo_identity library provides the provider, account directory and
grant issuer interfaces with a dev-only test provider. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo_network_quic wraps msquic in a single-owner Endpoint: newest-wins datagram
channel and one length-framed reliable stream, both bounded, with explicit close
causes. The client pins exactly one CA and name with an owned Mbed TLS check that
is the sole authority (OpenSSL's verdict advisory), callbacks never call msquic
under the lock, and MinimumMtu starts at 1280 so 1,200-byte datagrams fit minimum
paths. An owned seeded UDP impairment relay drives loss, duplication, reordering,
corruption, MTU and outage tests, passing on MSVC, GCC and Clang ASan/UBSan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0089 records the maintainer-ratified 50 ms tick, 1 s flush with final
outcomes p95 <= 1.5 s, responsiveness targets and reference hardware, plus
derived scheduling, sequence, identity, baseline, queue/byte/work budgets and the
exact/tolerant determinism contract (schemas/network/m10-budgets-v1.md). Adds
SessionClient traffic counters, `evo-dev-session trace` and a reproducible trace
script; development-reference results are in M10_TIMING_TRACES. Records that
genome v1 and metabolism rates are per tick, so the quantum fixes biology rates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SDL-free action/HID-key bindings with conflict validation, a staging editor and
a versioned all-or-nothing controls file (ADR-0085). The adapter maps keys
through the bindings, releases and drains on rebinding, and captures the chosen
key ahead of the toolkit and world. The client adds a Controls panel, key hints,
startup loading with a bounded rejection notice, and --controls/--show-controls.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve pointer ownership per SDL event so same-batch panel clicks, wheel and
click-to-focus typing never reach the world; keep the Tab toggle independent of
idempotent external capture release; own launcher phase and remote session
together so Local fixture ends any remote attempt; wrap notices beside a
reserved close column within a bounded stack. Adds client-panels test and
host/session, same-batch and host-order Tab regressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Toolkit-independent launcher, selection, label and outcome-log models with
bounded inputs and fixed error text; Dear ImGui overlay in the native adapter
with toolkit-first event routing and capture; engineering panels for fixture vs
dev region, legend, inspector, session outcomes, notices and dialog. Additive
SessionClient::unreachable_attempts() bounds unreachable connects. Docs, CI
targets and validation evidence updated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- TLS: replace send() with a bounded queue/flush; an interrupted record is
retried with identical arguments and later frames wait behind it, so no
queued frame is lost or reordered; close() flushes first.
- Carriers: server-issued 'I','N' | 48-bit epoch | 64-bit ordinal ids
replace the aliasing low-32-bit packing; client sequences are kept
separately for replies; out-of-range epochs are refused, never truncated.
- Grants: a durable, fsynced, fail-closed consumed-grant ledger
(--session-grant-ledger, /objects/dev-session-grants.v1 in dev) keeps
single-use grants consumed across process and pod restarts.
- Liveness: protocol v1 (undeployed) gains Heartbeat and the timeout close
reason; both sides heartbeat when idle and treat 5 s of silence as dead.
- Client sends are bounded and unresolved tracking starts at send time.
- ci-client configures with an explicit C compiler.
Each finding has a regression test; ADR-0083, the schema, operations doc
and VALIDATION record the corrections.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run a non-blocking TLS 1.3 session server inside evo-server's loop:
environment views come from the region's own field/terrain stores with an
explicitly empty organism presentation; admitted intentions ride the next
synthetic input (content unchanged) and return a durable no-effect outcome
from the host's final acknowledgement, or not_committed when dropped.
Sessions are bound to their connection, epochs close stale sessions, and
connections have bounded output, input rate and deadlines.
SessionClient and evo-client --connect consume one single-use grant per
Hello (refusals before admission keep it), reset to a fresh baseline on
every session, keep stale views visible and report in-flight intentions as
unknown after loss. evo-dev-session generates dev credentials, grant
bundles and runs a restart probe. The dev deployment gains a drill-time,
loopback-only listener with an optional Secret, enforced by the policy
check, and tools/ops/session-drill.sh. Protocol v1 gains IntentOutcome and
the capacity close reason before any deployment. ADR-0083 records the
maintainer-confirmed decisions; M8-dev/stage remains not passed and no
cluster change was made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add protocol v1: versioned, bounded frames (Hello, Welcome, Intent,
IntentResult, environment/organism views, Close) carrying session, epoch
and commit watermark, with an explicit little-endian canonical codec, a
header-first FrameReader, seeded mutation tests and a libFuzzer target.
SessionAdmission enforces world/region scope, grant lifetime and skew,
single-use grants, one session per subject, epoch, replay, tick window
and per-session rate.
Dev identity is a 108-byte HMAC-SHA256 grant (PSA) that cannot be
created or verified outside the dev environment. Transport security is
TLS 1.3 only via pinned Mbed TLS 4.1.1 LTS with TLS 1.2, DTLS, tickets
and PSK compiled out, AEAD suites, ECDHE groups and a pinned dev CA.
Opt-in EVO_BUILD_NETWORK, new network CI suite. Sockets and region
integration follow in M9.7. ADR-0082 records the decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add OrganismPresentation cells derived via biology::derive_phenotype with
no controller, owner or level fields (pinned by test), so identical
biology renders identically for players and NPCs. OrganismLayer reuses
the ADR-0080 ordering/freshness semantics; death is a remnant state and
removal is absence. Rendering takes an explicit rectangle budget, falls
back to body-only detail and marks truncation instead of dropping data.
An opt-in rectangle-only hover inspector shows phenotype gauges; ImGui is
not adopted. ADR-0081 records the decision; evidence in VALIDATION.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add SDL-free owned EnvironmentPresentation copied from M6.3 views (no
store pointers or validation tokens), a single-slot EnvironmentLayer that
rejects foreign/regressed updates and reports none/current/stale against
observed authority watermarks, and a bounded camera projection with
explicit missing, not-requested, present-zero, undefined-moisture and
stale encodings (at most 2313 clipped rectangles, reused buffer).
The engineering app renders a local in-memory fixture authority with
configurable refresh cadence and layer selection. ADR-0080 records the
decision; validation evidence and limits are in VALIDATION.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first dev drill showed a full volume driving ~140 refused publications a
second. Commit retries now back off 100 ms doubling to a 2 s cap, reset on the
next confirmed commit. fault-drill.sh reports the empty-volume pod's exit code,
restarts and logs, and prints node-specific iptables rules for db-outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>