ADR-0097. The first evolution-dev load drill (image 6c96629e) did not
measure capacity: the governor refused six seconds after the restart with
no session (p95 over a window of only startup ticks), and the bots spent
their grants retrying every 2 s. Reference numbers with one session: tick
work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms.
- The governor acts on work only over a full 40-tick window; tick debt
still counts at once.
- evo-load backs off 10 s after a load refusal and, in bundle mode,
reuses the grant the region refused before using it.
- The drill waits for a calm load-report before the bots start, takes the
drill pod's whole log before restoring (the live follow died), and
confirms the restore against a pod other than the drill pod.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.
Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A window ending 5.2 h after start (b2f224d) put evo-region into a
crashloop (error m10-window-too-long); the server only sees it at
startup. The deploy policy now requires REGION_SESSION_WINDOW to name its
UTC activation and end within 4 h of it, with two selftest cases (one is
the b2f224d value). DEPLOYMENT.md documents the follow-on quarantine:
syncs recorded as Deployment-only run no hooks, so a promoted build is
never retained; the provision hook is re-run from the rendered manifest.
VALIDATION records the incident and the recovery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0097. evo-load runs bots on worker threads, each a RegionSessionClient
with its own account and an in-process dev grant: one intention per tick
at its input lead with marker prediction, actions every 2 s, interest
moves every 10 s, and re-login after any close. Stepped ramps report
pooled and per-client distributions (response, receipt, outcome, login,
prediction error), snapshot rate, gaps and bandwidth, drops, failed
logins and refusals, generator CPU, a starvation check among connected
bots, a budget verdict and the region's evo_m10_* gauges, ending with the
sustainable bot count. A Postgres process smoke runs 4 then 8 bots
against a real evo-server.
Fixes found while ramping: the process fixture's short pumps never read
(a chatty child blocked on its stdout pipe), and the load governor
refused on a single tick of debt (now 2 refuse, 4 shed). Refreshes the
dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.
The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
abort deadline was in memory): schema v14 overdue_probe_transfers and a
session-layer sweep abort open outgoing transfers past the ticket
lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
took control back: admission of a committed transfer now requires the
record as that commit left it.
M10.8 is complete. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Session protocol v4 (v3 is deployed and frozen) adds HandoffOffer,
SessionRedirect, close reasons handed_off and moved, and ghost marker records.
Sessions use the durable probe record: logins open it where it was saved
(redirected when it lives elsewhere), and session ends save it. A marker
crossing into a neighbour's area is frozen, prepared, its control released
and a signed ticket offered; uncommitted transfers are aborted after the
ticket's lifetime. The destination shows arriving probes as ghosts (schema
v13 incoming_probe_transfers) and opens the marker where it crossed.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable
probe records hold one owning region per account, so ownership is unique and
the test inventory is conserved by construction. A fenced transfer state
machine (prepared, staged, committed, aborted) under each caller's exact
region lease returns the recorded outcome on every retry; a commit requires
staging by the destination's current epoch; aborts are durable and only
before commit. handoff_account_control moves account control on a committed
transfer.
A PostgreSQL fault matrix covers restarts at each transition, stale epochs,
failed destinations, partitions and commit/abort races, checking one owner and
the conserved inventory after every scenario.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0095. The engineering client times presentation with the tick clock:
its own marker between ticks, other markers interpolated a few ticks behind
(grey when stale). Silence beyond 3 ticks and long-gap holds are stale. A lost
session leaves its last view frozen and labelled stale until the reconnect's
fresh baseline; nothing is predicted or sent meanwhile.
evo-motion-trace --remote measures remote-marker error between two clients
through the relay, with a 1.5 s outage (motion-trace-remote-smoke CTest).
Prediction errors are recorded only for ticks inputs are scheduled for.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-motion-trace drives the engineering client's own session code through
the owned impairment relay against a real-time 50 ms loopback region:
exact without loss, 94-99 % exact at 60-200 ms RTT with 2-5 % loss, every
profile converging after stopping, restart resync about 2-3 s
(docs/networking/M10_PREDICTION_TRACES.md). The CTest motion-trace-smoke
checks convergence, bounds and a majority of exact predictions.
The relay lets a new sender take over after the current client has been
silent for 1 s (reconnects); the client counts refusal receipts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-client's remote mode now runs over RegionSessionClient (QUIC, session
v2): snapshot-fed environment and organism views, the camera following the
server's view until the user moves it and then driving the interest, intention
records with a simulated state, and reconnects with the next grant of a bundle.
The launcher reads 160-byte M10 grants and defaults to the dev route.
- quic::resolve_ipv4 shared by the client and evo-session-probe
- evo-session-probe grants: owner-only bundles of 1-64 grants
- evo-server --m10-max-grant-lifetime (dev only, 60..3600 s)
- presets client-network/client-package build QUIC; msquic links the CRT
fully static; msquic.dll and msquic/OpenSSL licences in the Windows package
- client-remote-session test against a loopback region
- ADR-0093/0091/0086 amendments, M10_CLIENT_SESSIONS guide, validation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP
7400, so nothing reached it through the VIP. The route container port, Service and
NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check
requires exactly UDP 7400 for the route and counts any protocol on 7400 as session
exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and
internal DNS as separate reviewed steps, end-to-end probe as acceptance);
session protocol v1 recorded as frozen now that it is deployed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each batch's actions are recorded in the same statement as its checkpoint: one
region_actions row per (world, account, action id) and the block in an append-only
table, compared on exact retries and resolution. The region host attaches actions to
the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome
batches unresolved. Outcomes come from the durable record via ActionLedger, so
crashes before or after a commit, unknown publications and cross-region moves all
resolve to one answer; a repeated id keeps its first record instead of stalling a
region. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration v10 keeps, per world and account, the highest accepted grant serial and
the region session in control with a fencing generation and database-time expiry,
reachable only through claim/renew/release functions under the caller's region
lease. Serials survive restarts and apply across regions; a cross-region takeover
returns a wait so control never overlaps. RegionSessions takes durable control via
a ControlLedger before admitting a session, delays Welcome by the wait, renews, and
closes sessions whose control is fenced or cannot be confirmed in time.
PostgresControlLedger implements the ledger. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Formatting sorted <timeapi.h> before <windows.h>, which it requires. A separate include
block keeps the order. Rebuilt and tested with MSVC preset quic.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-quic-hol-measure sends the same 20 Hz state on the DATAGRAM channel and the
ordered reliable stream plus 16 KiB reliable messages through the impairment relay
across RTT x loss. Under 5-10 % loss the stream's p95 is 3-5.5x the datagrams' and
exceeds the 250 ms target at 100 ms RTT, confirming the ADR-0090 channel split.
The relay and tool request 1 ms timer resolution on Windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo_network_quic wraps msquic in a single-owner Endpoint: newest-wins datagram
channel and one length-framed reliable stream, both bounded, with explicit close
causes. The client pins exactly one CA and name with an owned Mbed TLS check that
is the sole authority (OpenSSL's verdict advisory), callbacks never call msquic
under the lock, and MinimumMtu starts at 1280 so 1,200-byte datagrams fit minimum
paths. An owned seeded UDP impairment relay drives loss, duplication, reordering,
corruption, MTU and outage tests, passing on MSVC, GCC and Clang ASan/UBSan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0090 records the maintainer's selection of msquic 2.6.2 with OpenSSL 3.5.9 LTS
built from pinned, checksum-verified sources on Windows and Linux (plus the
xdp-for-windows headers msquic needs on Windows), the trust, threading and
channel model, and the known gap that OpenSSL's verifier also trusts its default
paths. Adds the opt-in quic preset, explicit preparation, and a loopback smoke
test (datagram, stream, untrusted CA and wrong name refused) passing on MSVC and
GCC. windows-check picks up newly installed Perl/NASM and pins cl.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0089 records the maintainer-ratified 50 ms tick, 1 s flush with final
outcomes p95 <= 1.5 s, responsiveness targets and reference hardware, plus
derived scheduling, sequence, identity, baseline, queue/byte/work budgets and the
exact/tolerant determinism contract (schemas/network/m10-budgets-v1.md). Adds
SessionClient traffic counters, `evo-dev-session trace` and a reproducible trace
script; development-reference results are in M10_TIMING_TRACES. Records that
genome v1 and metabolism rates are per tick, so the quantum fixes biology rates.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rendered policies must keep default-deny in every environment, and no ingress rule
that can select a region pod may admit the session port (no or empty ports, the
session name, 7400 or a spanning range) except the validated route policy. Route
Service and policy validators are shared by source and rendered checks. Self-test
grows to 44 cases, including the reviewer's all-ingress reproduction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The policy check now renders every overlay with pinned, checksum-verified
kustomize v5.8.1 (ADR-0088) and validates effective namespaces, session/input/
listener environment and arguments, ports, Services, quotas, policies, Ingresses
and Secrets, so inline/JSON patches and namespace changes cannot bypass it.
Route enabling is gated on an ADR-0087 amendment with tested server windows, route
addresses must be RFC 1918 or explicitly approved, and the Local health-check node
port is documented with its enabling evidence. Self-test grows to 33 cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0087 records the maintainer's route (void.aeternalisgames.com:7400 via a
dedicated internal MetalLB VIP to the evolution-dev region pod, TLS end to end,
CIDR-limited) and operator-opened windows enforced by evo-server. The route is a
kustomize component included nowhere; the policy check encodes its shape and
refuses enabling until VIP/CIDR, the dev quota and --session-window exist, with a
20-case self-test in the CI deploy stage. The base listener stays loopback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Static-runtime client-package preset with /Brepro and a build identity printed by
evo-client --version (ADR-0086). tools/client packages an explicit allowlist with
licence texts from the verified sources, BUILD-INFO and SHA256SUMS into a
deterministic ZIP, refusing credentials, profile paths and non-system imports;
verification runs the extracted client without Visual Studio, and a self-test
proves each guard. Publication via Gitea Actions waits for a Windows runner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run a non-blocking TLS 1.3 session server inside evo-server's loop:
environment views come from the region's own field/terrain stores with an
explicitly empty organism presentation; admitted intentions ride the next
synthetic input (content unchanged) and return a durable no-effect outcome
from the host's final acknowledgement, or not_committed when dropped.
Sessions are bound to their connection, epochs close stale sessions, and
connections have bounded output, input rate and deadlines.
SessionClient and evo-client --connect consume one single-use grant per
Hello (refusals before admission keep it), reset to a fresh baseline on
every session, keep stale views visible and report in-flight intentions as
unknown after loss. evo-dev-session generates dev credentials, grant
bundles and runs a restart probe. The dev deployment gains a drill-time,
loopback-only listener with an optional Secret, enforced by the policy
check, and tools/ops/session-drill.sh. Protocol v1 gains IntentOutcome and
the capacity close reason before any deployment. ADR-0083 records the
maintainer-confirmed decisions; M8-dev/stage remains not passed and no
cluster change was made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add protocol v1: versioned, bounded frames (Hello, Welcome, Intent,
IntentResult, environment/organism views, Close) carrying session, epoch
and commit watermark, with an explicit little-endian canonical codec, a
header-first FrameReader, seeded mutation tests and a libFuzzer target.
SessionAdmission enforces world/region scope, grant lifetime and skew,
single-use grants, one session per subject, epoch, replay, tick window
and per-session rate.
Dev identity is a 108-byte HMAC-SHA256 grant (PSA) that cannot be
created or verified outside the dev environment. Transport security is
TLS 1.3 only via pinned Mbed TLS 4.1.1 LTS with TLS 1.2, DTLS, tickets
and PSK compiled out, AEAD suites, ECDHE groups and a pinned dev CA.
Opt-in EVO_BUILD_NETWORK, new network CI suite. Sockets and region
integration follow in M9.7. ADR-0082 records the decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P1: restore-world.sh claims its objects path (mkdir) and the database (a held
session with a restore try-lock and the exclusive upgrade lock), and removes only
what it created, so a losing restore never drops a winner's schema.
P2: barrier Ready-pod selection via shared jsonpath helper with fixtures; barrier
pod fsGroup; script-generated pods inspected by the policy check; World0 PV claim
references protected; ci-paths.sh uses the event's real base and runs on
uncertainty; fault-drill asserts RTO and needs positive quarantine evidence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked
(policy-enforced) until acknowledged outcomes survive host loss in an
independent failure domain. Adds in-place restore with an explicit rewind,
barrier-based fencing of a superseded database, and promotion of stage-pinned
digests only. dr-check.sh rehearses host loss in every PostgreSQL build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first dev drill showed a full volume driving ~140 refused publications a
second. Commit retries now back off 100 ms doubling to a 2 s cap, reset on the
next confirmed commit. fault-drill.sh reports the empty-volume pod's exit code,
restarts and logs, and prints node-specific iptables rules for db-outage.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kubectl jsonpath has no negation, so the deletionTimestamp filter was a
parse error on every call. Use a go-template that skips pods already
being deleted; verified against sample pods with Go's text/template,
which kubectl's go-template output uses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drill-only evo-server --synthetic-inputs development-v1 driver feeds
deterministic inputs from the committed head after every recovery, with
request IDs that match acknowledgements across owners (ADR-0073); it is
off in Git and the policy check keeps it off. fault-drill.sh loads dev
or stage and runs repeated forced kills (RTO, zero lost
acknowledgements), a full object volume (nothing acknowledged), an
empty volume (quarantine, never a fresh world) and an operator-driven
database outage (stale epoch never resumes), restoring the Git
configuration on exit. Process tests prove contiguous acknowledgement
across a SIGKILL and exact resumption after the committed head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Complete the maintainer's server image bump to edb39f by also setting
the region build identity, and add REGION_RETAINED so the host keeps
the builds that wrote its history on its recovery allowlist; without it
the upgraded host would quarantine its own head as unsupported. Record
the M8.4 drill (rollout 3.0 s, forced deletion 19.3 s to Ready, epochs
3 to 5) and the maintainer's M8.5 sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deploy evo-region per environment (ADR-0071): one replica with the
Recreate strategy (a rolling update would deadlock on the lease),
/readyz readiness, /livez liveness, no CPU limit and a drain inside the
grace period, with the image digest doubling as the build identity. A
new idempotent evo-world-clone provision subcommand creates genesis from
development_opening (byte-identical to the qualification fixture) as a
Sync hook after the migration. Dev and stage paths include their region;
a dev AppProject allows Deployment and Service; region-drill.sh times
rollout and forced-deletion recovery; the policy check enforces the
host's invariants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the maintainer's request everything but Secrets and the store node
label now goes through Argo CD. aevolution-platform syncs one bundle
(World0 admission policy, Retain storage class, each environment's
namespace, guardrails, service accounts and object claim) with
self-heal, no prune and Prune=false,Delete=false on every object; the
admission policy also protects the platform app and project. Store Jobs
become kustomize components used as PreSync (migrate) and PostSync
(check-store) hooks per environment, and the policy check refuses any
unresolved marker in a path an Application syncs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>