ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable
probe records hold one owning region per account, so ownership is unique and
the test inventory is conserved by construction. A fenced transfer state
machine (prepared, staged, committed, aborted) under each caller's exact
region lease returns the recorded outcome on every retry; a commit requires
staging by the destination's current epoch; aborts are durable and only
before commit. handoff_account_control moves account control on a committed
transfer.
A PostgreSQL fault matrix covers restarts at each transition, stale epochs,
failed destinations, partitions and commit/abort races, checking one owner and
the conserved inventory after every scenario.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evo-server gains an opt-in M10 listener (--m10-*, QUIC builds only): RegionSessions
with the PostgreSQL control and action ledgers on their own connection under the
host's lease, carried actions submitted to the host and its acknowledgements and
drops turned into outcomes. evo-session-probe is a dev-only headless client. Real
process tests cover intentions, durable action outcomes, SIGKILL before and after
an action's commit, duplicate retries and the probe. The simulated watermark now
reaches every session every tick, so clients always know the server tick.
ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each batch's actions are recorded in the same statement as its checkpoint: one
region_actions row per (world, account, action id) and the block in an append-only
table, compared on exact retries and resolution. The region host attaches actions to
the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome
batches unresolved. Outcomes come from the durable record via ActionLedger, so
crashes before or after a commit, unknown publications and cross-region moves all
resolve to one answer; a repeated id keeps its first record instead of stalling a
region. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration v10 keeps, per world and account, the highest accepted grant serial and
the region session in control with a fencing generation and database-time expiry,
reachable only through claim/renew/release functions under the caller's region
lease. Serials survive restarts and apply across regions; a cross-region takeover
returns a wait so control never overlaps. RegionSessions takes durable control via
a ControlLedger before admitting a session, delays Welcome by the wait, renews, and
closes sessions whose control is fenced or cannot be confirmed in time.
PostgresControlLedger implements the ledger. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run a non-blocking TLS 1.3 session server inside evo-server's loop:
environment views come from the region's own field/terrain stores with an
explicitly empty organism presentation; admitted intentions ride the next
synthetic input (content unchanged) and return a durable no-effect outcome
from the host's final acknowledgement, or not_committed when dropped.
Sessions are bound to their connection, epochs close stale sessions, and
connections have bounded output, input rate and deadlines.
SessionClient and evo-client --connect consume one single-use grant per
Hello (refusals before admission keep it), reset to a fresh baseline on
every session, keep stale views visible and report in-flight intentions as
unknown after loss. evo-dev-session generates dev credentials, grant
bundles and runs a restart probe. The dev deployment gains a drill-time,
loopback-only listener with an optional Secret, enforced by the policy
check, and tools/ops/session-drill.sh. Protocol v1 gains IntentOutcome and
the capacity close reason before any deployment. ADR-0083 records the
maintainer-confirmed decisions; M8-dev/stage remains not passed and no
cluster change was made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drill-only evo-server --synthetic-inputs development-v1 driver feeds
deterministic inputs from the committed head after every recovery, with
request IDs that match acknowledgements across owners (ADR-0073); it is
off in Git and the policy check keeps it off. fault-drill.sh loads dev
or stage and runs repeated forced kills (RTO, zero lost
acknowledgements), a full object volume (nothing acknowledged), an
empty volume (quarantine, never a fresh world) and an operator-driven
database outage (stale epoch never resumes), restoring the Git
configuration on exit. Process tests prove contiguous acknowledgement
across a SIGKILL and exact resumption after the committed head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The maintainer's login Secrets carry pg_service.conf ([evolution]),
pgpass and ca.crt. Add a strict read_pg_service (known keys only,
verify-full unless a local socket, bounded size) behind --pg-service in
evo-server and evo-world-clone, mount each login Secret whole in the
store Jobs instead of reading aevolution-db, and fix the migrate Job's
PGSERVICE to the [evolution] section. Record the store node choice,
per-environment databases and the backup plan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the filesystem object store on a node-local Retain-class PVC per
environment (ADR-0070), with the database on the external PostgreSQL VM.
Because local-path does not enforce sizes, the object store now refuses
new objects below a free-space reserve with a distinct full error, so a
full volume publishes and acknowledges nothing. evo-world-clone gains
check-store to qualify volume retention, reserve, database version/TLS
and the runtime privilege boundary from inside the cluster; it now ships
in the server image. Add the storage class, object claims, hand-applied
store Jobs, policy rules forbidding password env vars, and the runbook.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clone a verified committed head into an isolated world and database
(ADR-0067): world re-identification with a proven byte round trip, a
schema v7 clone genesis naming the source commit and digest with digested
inherited outcomes, a read-only source session, an owner-only atomic
target transaction and the evo-world-clone tool. audit_region_history
re-verifies every retained link. Real-process tests cover trajectory
equality after the cut, source isolation, killed clones and retention.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0066: evo-server --ops-listen serves /livez (loop responding, never
authority), /readyz (200 only while the host holds the lease with a
verified recovered head) and /metrics (Prometheus text: 30 fixed
series for readiness, phase, lease epoch/remaining, simulated/committed
ticks, commit lag, checkpoint age, tick debt, queue use and bounds, and
counters; the only labels are a 6-value phase and 4-value drop reason).
The dependency-free OpsServer runs on the host loop with no thread:
GET on exact paths, 405/404/400/431 responses, 8 connections with excess
refused on accept, 4 KiB requests, 1 s deadlines, 16 KiB responses.
HostStatus/phase types moved to the portable evo::ops library with a
Windows-tested renderer. evo-server now refuses to start without its
required identity and connection settings.
Adds the server-runtime Docker target (evo-server + pinned libpq,
UID 10001, no credentials), a CI server-image job with smoke script and
optional publishing, and a Kubernetes example outside Argo.
Validated: Windows CTest 59/59; Linux Debug/Release/clang-tidy/ASan
CTest 59/59; PostgreSQL suites under Release, clang-tidy and ASan with
real process probes; two ops mutations caught. Docker build/smoke and
Gitea CI not executed locally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0065: before any readiness, recovery now verifies the head's tail.
The trace object must exist with its manifest digest, its header must
bind profile, region, predecessor and range; the predecessor (schema v6
load_region_commit) must hash to the digest the head names; replaying
the trace from the predecessor's verified payload must reproduce every
report and the exact head bytes; and trace-derived decisions must equal
the head's durable decisions. Missing, corrupt or non-reproducing tails
quarantine (RecoveryError::tail); nothing older or empty is served.
Test-only crash seams (HostConfig::crash_hook, PublishHooks) and
evo-server --crash-at deliver real SIGKILL at before-upload,
before-transaction, after-commit and after-ack. The kill matrix,
a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as
the durability contract requires; successors verify before readiness.
Validated: PostgreSQL suites under GCC Release, clang-tidy and
ASan/UBSan; tail-skip and replay-equality mutations each caught. No
Docker/Gitea CI, production-lease kill test or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0064: RegionHost<State> (environmental and lineage profiles) is a
single-owner, single-threaded state machine advanced by step(now):
acquire the fenced lease and recover the verified head, run, resolve
ambiguous commits, drain and stop, or quarantine. Admission is
provisional and bounded (64 inputs, 64 KiB) with backpressure; FinalAck
and FinalOutcome are emitted only after a confirmed commit, and inputs
that will never be acknowledged are reported with a reason. Ticks follow
the caller's scheduling clock with bounded catch-up and measured tick
debt, never invented inputs. A death seals the batch and commits in the
same step, so no dependent tick uses an uncommitted outcome. Storage
outages pause and retry, unknown outcomes resolve by exact bytes,
fencing discards unconfirmed work and re-acquires; readiness fails
closed. Drain commits pending ticks and releases the lease.
evo-server hosts one region with a bounded stdin input feed, stdout
event lines, a readiness file and SIGTERM drain (exit 0/3/4).
Validated: Windows CTest 58/58; PostgreSQL suites under GCC Release,
clang-tidy and ASan/UBSan including real-process SIGTERM drain and
SIGKILL with successor recovery; four host mutations each caught. No
gateway, metrics, Docker/Gitea CI or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0061 publishes environmental-v1 full checkpoints. Manifest v1 is a
fixed 280-byte record (identity, chain, lease, versions, build, zero
RNG/history declarations, durability policy, payload length/SHA-256)
that PostgreSQL parses and digests itself. Schema v3 adds region_commits
with a digest CHECK and a composite head FK, owner-managed retained
builds, fenced publish/load functions and owner-only genesis; it drops
the bare v2 head kernel and fails closed while v2 heads exist.
Linux evo::region_store adds OpenSSL SHA-256, an immutable
content-addressed filesystem object store (temp+fsync+link, no
overwrite/delete, O_NOFOLLOW exact-length verified reads), and
publication (validate -> upload -> read back -> fenced transaction ->
receipt check) and recovery (re-digest, allowlist, verify, full restore).
Portable evo::region_checkpoint holds the codec and payload validation.
Validated: Windows CTest 56/56; WSL Release, clang-tidy and ASan/UBSan
private-PostgreSQL suites; four SQL and one host mutation each caught.
M7.4 stays open for 7.4b (lineage archive codec and history binding).
No Docker/Gitea CI, remote store, process kill or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0060 and schema v2 add database-time region leases and a fenced head
advance. Acquire/renew/release/advance lock the lease row, check holder,
epoch and expiry with clock_timestamp() after the lock wait, and the
advance compares the exact predecessor. Holders are DB-generated tokens;
epochs never wrap or repeat. The runtime group role gets EXECUTE on five
SECURITY DEFINER functions only - no table, provisioning or CREATE rights.
Migration history admits clean->v2, v1->v2 and a v2 no-op, and refuses
privileged runtime roles.
Host side: portable LeaseAuthority fences from send-time deadlines on
CLOCK_BOOTTIME, keeps the earlier deadline on unknown renewal and stays
fenced until a newer epoch. Selected timings: 20s lease, 5s renew, 2s
margin, 2s renew and 5s commit deadlines. Linux evo::postgres wraps the
functions with typed LeaseStoreError results.
Validated: Windows CTest 55/55; WSL Release, clang-tidy and ASan/UBSan
private-PostgreSQL suites (contenders, expiry, paused/queued stale
writers, unknown renewal, exhaustion, bypass denial, upgrade path).
Three SQL mutations were each caught. Docker/Gitea CI, remote TLS and
cluster runs were not executed. No director, host or deployment added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>