The fourth dev drill showed a synchronous PostgreSQL round trip per action on the
simulation thread: at admission (uniqueness) and again at commit (first-record rule).
About 12 lookups landed on each commit tick (commit ticks 58-70 ms, session p99 30-39 ms).
- ActionLedger::find_all; persistence::find_region_actions applies the schema v11
find_region_action to many pairs in one statement (no migration).
- IntentAdmission::admit/verify: actions only the ledger can refuse wait for one verify
per poll; receipts still go out in the same poll. committed() asks once per commit.
- Tests: deferred_actions (network_control), find_region_actions and find_all (Postgres).
- ADR-0097 fourth run, ADR-0092 amendment, VALIDATION; refresh the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. evo-load runs bots on worker threads, each a RegionSessionClient
with its own account and an in-process dev grant: one intention per tick
at its input lead with marker prediction, actions every 2 s, interest
moves every 10 s, and re-login after any close. Stepped ramps report
pooled and per-client distributions (response, receipt, outcome, login,
prediction error), snapshot rate, gaps and bandwidth, drops, failed
logins and refusals, generator CPU, a starvation check among connected
bots, a budget verdict and the region's evo_m10_* gauges, ending with the
sustainable bot count. A Postgres process smoke runs 4 then 8 bots
against a real evo-server.
Fixes found while ramping: the process fixture's short pumps never read
(a chatty child blocked on its stdout pipe), and the load governor
refused on a single tick of debt (now 2 refuse, 4 shed). Refreshes the
dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097 (M10.9 load and abuse qualification; only evolution-dev
reference-worker runs qualify). evo-server measures each simulated tick's
work (host step plus the M10 bridge, with the session layer's share kept
separately) and feeds a portable LoadGovernor: grant logins are refused
at a 2 s p95 of 15 ms or any tick debt, replication is shed in up to
three levels at 20 ms or debt >= 2 (radius cap 7/7/5/3, far updates
x1/x2/x2/x4, no fresh baselines, own markers unaffected), and both relax
after 5 s of calm. Handoff tickets are still admitted. /metrics gains
evo_m10_* gauges when the listener is enabled.
Adds the missing ADR index entries for ADR-0094 to ADR-0096. Refreshes
the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.
The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
abort deadline was in memory): schema v14 overdue_probe_transfers and a
session-layer sweep abort open outgoing transfers past the ticket
lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
took control back: admission of a committed transfer now requires the
record as that commit left it.
M10.8 is complete. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Session protocol v4 (v3 is deployed and frozen) adds HandoffOffer,
SessionRedirect, close reasons handed_off and moved, and ghost marker records.
Sessions use the durable probe record: logins open it where it was saved
(redirected when it lives elsewhere), and session ends save it. A marker
crossing into a neighbour's area is frozen, prepared, its control released
and a signed ticket offered; uncommitted transfers are aborted after the
ticket's lifetime. The destination shows arriving probes as ghosts (schema
v13 incoming_probe_transfers) and opens the marker where it crossed.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. A source region's handoff key signs a 160-byte ticket for one probe
transfer, account and destination; it rides the Hello grant slot and is
verified against neighbour keys bound to their regions. SessionControl
admits a ticket once per region, superseding like a grant, with the session
ending at the client's original grant expiry. RegionSessions makes the
destination's durable step first: PostgresHandoffLedger stages and commits
the transfer and moves the account's control, idempotently.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EXPERIMENT (ADR-0094). RegionSessions owns the probe markers: admitted
intentions schedule them, each poll advances them to the simulated tick, and
snapshots carry them. evo-server sets the motion area from the region's
habitat sites; the probe prints its own marker.
The engineering client predicts its marker with the adaptive input lead,
rebases on each snapshot, resends a lost or refused intention, smooths
corrections, draws markers and follows its own with camera and interest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RegionSessions owns the replication server: interest and acks per session, a
host-set default interest, and per-tick snapshots sized to each connection's
datagram limit. evo-server builds each interest's view from the region's own
habitat sites (default: the first site, radius 7). RegionSessionClient assembles,
acknowledges and presents snapshots and can change its interest; the probe prints
them. Loopback and real-process evidence. ADR-0093.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-server gains an opt-in M10 listener (--m10-*, QUIC builds only): RegionSessions
with the PostgreSQL control and action ledgers on their own connection under the
host's lease, carried actions submitted to the host and its acknowledgements and
drops turned into outcomes. evo-session-probe is a dev-only headless client. Real
process tests cover intentions, durable action outcomes, SIGKILL before and after
an action's commit, duplicate retries and the probe. The simulated watermark now
reaches every session every tick, so clients always know the server tick.
ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each batch's actions are recorded in the same statement as its checkpoint: one
region_actions row per (world, account, action id) and the block in an append-only
table, compared on exact retries and resolution. The region host attaches actions to
the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome
batches unresolved. Outcomes come from the durable record via ActionLedger, so
crashes before or after a commit, unknown publications and cross-region moves all
resolve to one answer; a repeated id keeps its first record instead of stalling a
region. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session protocol v1 (not yet deployed, amended in place) gains realtime intentions on
datagrams with receipts and a cumulative simulated watermark, and account-scoped
actions with stable ids, receipts, final outcomes and queries. IntentAdmission
enforces session binding, rate, authority, epoch, vector, sequence, tick window,
duplicates and the unresolved bound; RegionSessions and the client carry it over
QUIC. Intentions and actions have no biological effect yet. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration v10 keeps, per world and account, the highest accepted grant serial and
the region session in control with a fencing generation and database-time expiry,
reachable only through claim/renew/release functions under the caller's region
lease. Serials survive restarts and apply across regions; a cross-region takeover
returns a wait so control never overlaps. RegionSessions takes durable control via
a ControlLedger before admitting a session, delays Welcome by the wait, renews, and
closes sessions whose control is fenced or cannot be confirmed in time.
PostgresControlLedger implements the ledger. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Session protocol v1 session messages (Hello with grant, Welcome, Refresh,
RefreshResult, Close with reasons) on the QUIC reliable stream. RegionSessions
verifies grants with the issuer public keys and admits them through SessionControl:
a newer grant closes the account's old connection as superseded, expiry and epoch
advances close sessions, every close is announced and flushed. RegionSessionClient
logs in, refreshes in place and reports why a session ended. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first dev drill showed a full volume driving ~140 refused publications a
second. Commit retries now back off 100 ms doubling to a 2 s cap, reset on the
next confirmed commit. fault-drill.sh reports the empty-volume pod's exit code,
restarts and logs, and prints node-specific iptables rules for db-outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the
barrier takes a database-wide upgrade lock exclusively, while
acquisitions, lease checks, provisioning and the retired-build trigger
take it shared first, so a queued acquisition can no longer regain
authority for a just-retired build and concurrent barriers cannot
retire the last current build. Provisioning takes a per-world lock
before classifying a world as original or a clone of one source. The
operations listener makes at most four accept attempts per serve().
Race regressions prove each contender is blocked before commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0066: evo-server --ops-listen serves /livez (loop responding, never
authority), /readyz (200 only while the host holds the lease with a
verified recovered head) and /metrics (Prometheus text: 30 fixed
series for readiness, phase, lease epoch/remaining, simulated/committed
ticks, commit lag, checkpoint age, tick debt, queue use and bounds, and
counters; the only labels are a 6-value phase and 4-value drop reason).
The dependency-free OpsServer runs on the host loop with no thread:
GET on exact paths, 405/404/400/431 responses, 8 connections with excess
refused on accept, 4 KiB requests, 1 s deadlines, 16 KiB responses.
HostStatus/phase types moved to the portable evo::ops library with a
Windows-tested renderer. evo-server now refuses to start without its
required identity and connection settings.
Adds the server-runtime Docker target (evo-server + pinned libpq,
UID 10001, no credentials), a CI server-image job with smoke script and
optional publishing, and a Kubernetes example outside Argo.
Validated: Windows CTest 59/59; Linux Debug/Release/clang-tidy/ASan
CTest 59/59; PostgreSQL suites under Release, clang-tidy and ASan with
real process probes; two ops mutations caught. Docker build/smoke and
Gitea CI not executed locally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>