A window ending 5.2 h after start (b2f224d) put evo-region into a
crashloop (error m10-window-too-long); the server only sees it at
startup. The deploy policy now requires REGION_SESSION_WINDOW to name its
UTC activation and end within 4 h of it, with two selftest cases (one is
the b2f224d value). DEPLOYMENT.md documents the follow-on quarantine:
syncs recorded as Deployment-only run no hooks, so a promoted build is
never retained; the provision hook is re-run from the rendered manifest.
VALIDATION records the incident and the recovery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.
The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
abort deadline was in memory): schema v14 overdue_probe_transfers and a
session-layer sweep abort open outgoing transfers past the ticket
lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
took control back: admission of a committed transfer now requires the
record as that commit left it.
M10.8 is complete. Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096. Session protocol v4 (v3 is deployed and frozen) adds HandoffOffer,
SessionRedirect, close reasons handed_off and moved, and ghost marker records.
Sessions use the durable probe record: logins open it where it was saved
(redirected when it lives elsewhere), and session ends save it. A marker
crossing into a neighbour's area is frozen, prepared, its control released
and a signed ticket offered; uncommitted transfers are aborted after the
ticket's lifetime. The destination shows arriving probes as ghosts (schema
v13 incoming_probe_transfers) and opens the marker where it crossed.
Refreshes the dev session window.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0096, the persistence handoff ADR (EXPERIMENT probes, ADR-0094). Durable
probe records hold one owning region per account, so ownership is unique and
the test inventory is conserved by construction. A fenced transfer state
machine (prepared, staged, committed, aborted) under each caller's exact
region lease returns the recorded outcome on every retry; a commit requires
staging by the destination's current epoch; aborts are durable and only
before commit. handoff_account_control moves account control on a committed
transfer.
A PostgreSQL fault matrix covers restarts at each transition, stale epochs,
failed destinations, partitions and commit/abort races, checking one owner and
the conserved inventory after every scenario.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP
7400, so nothing reached it through the VIP. The route container port, Service and
NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check
requires exactly UDP 7400 for the route and counts any protocol on 7400 as session
exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and
internal DNS as separate reviewed steps, end-to-end probe as acceptance);
session protocol v1 recorded as frozen now that it is deployed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each batch's actions are recorded in the same statement as its checkpoint: one
region_actions row per (world, account, action id) and the block in an append-only
table, compared on exact retries and resolution. The region host attaches actions to
the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome
batches unresolved. Outcomes come from the durable record via ActionLedger, so
crashes before or after a commit, unknown publications and cross-region moves all
resolve to one answer; a repeated id keeps its first record instead of stalling a
region. ADR-0092.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Migration v10 keeps, per world and account, the highest accepted grant serial and
the region session in control with a fencing generation and database-time expiry,
reachable only through claim/renew/release functions under the caller's region
lease. Serials survive restarts and apply across regions; a cross-region takeover
returns a wait so control never overlaps. RegionSessions takes durable control via
a ControlLedger before admitting a session, delays Welcome by the wait, renews, and
closes sessions whose control is fenced or cannot be confirmed in time.
PostgresControlLedger implements the ledger. ADR-0091.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rendered policies must keep default-deny in every environment, and no ingress rule
that can select a region pod may admit the session port (no or empty ports, the
session name, 7400 or a spanning range) except the validated route policy. Route
Service and policy validators are shared by source and rendered checks. Self-test
grows to 44 cases, including the reviewer's all-ingress reproduction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The policy check now renders every overlay with pinned, checksum-verified
kustomize v5.8.1 (ADR-0088) and validates effective namespaces, session/input/
listener environment and arguments, ports, Services, quotas, policies, Ingresses
and Secrets, so inline/JSON patches and namespace changes cannot bypass it.
Route enabling is gated on an ADR-0087 amendment with tested server windows, route
addresses must be RFC 1918 or explicitly approved, and the Local health-check node
port is documented with its enabling evidence. Self-test grows to 33 cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0087 records the maintainer's route (void.aeternalisgames.com:7400 via a
dedicated internal MetalLB VIP to the evolution-dev region pod, TLS end to end,
CIDR-limited) and operator-opened windows enforced by evo-server. The route is a
kustomize component included nowhere; the policy check encodes its shape and
refuses enabling until VIP/CIDR, the dev quota and --session-window exist, with a
20-case self-test in the CI deploy stage. The base listener stays loopback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Run a non-blocking TLS 1.3 session server inside evo-server's loop:
environment views come from the region's own field/terrain stores with an
explicitly empty organism presentation; admitted intentions ride the next
synthetic input (content unchanged) and return a durable no-effect outcome
from the host's final acknowledgement, or not_committed when dropped.
Sessions are bound to their connection, epochs close stale sessions, and
connections have bounded output, input rate and deadlines.
SessionClient and evo-client --connect consume one single-use grant per
Hello (refusals before admission keep it), reset to a fresh baseline on
every session, keep stale views visible and report in-flight intentions as
unknown after loss. evo-dev-session generates dev credentials, grant
bundles and runs a restart probe. The dev deployment gains a drill-time,
loopback-only listener with an optional Secret, enforced by the policy
check, and tools/ops/session-drill.sh. Protocol v1 gains IntentOutcome and
the capacity close reason before any deployment. ADR-0083 records the
maintainer-confirmed decisions; M8-dev/stage remains not passed and no
cluster change was made.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add protocol v1: versioned, bounded frames (Hello, Welcome, Intent,
IntentResult, environment/organism views, Close) carrying session, epoch
and commit watermark, with an explicit little-endian canonical codec, a
header-first FrameReader, seeded mutation tests and a libFuzzer target.
SessionAdmission enforces world/region scope, grant lifetime and skew,
single-use grants, one session per subject, epoch, replay, tick window
and per-session rate.
Dev identity is a 108-byte HMAC-SHA256 grant (PSA) that cannot be
created or verified outside the dev environment. Transport security is
TLS 1.3 only via pinned Mbed TLS 4.1.1 LTS with TLS 1.2, DTLS, tickets
and PSK compiled out, AEAD suites, ECDHE groups and a pinned dev CA.
Opt-in EVO_BUILD_NETWORK, new network CI suite. Sockets and region
integration follow in M9.7. ADR-0082 records the decision.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
P1: restore-world.sh claims its objects path (mkdir) and the database (a held
session with a restore try-lock and the exclusive upgrade lock), and removes only
what it created, so a losing restore never drops a winner's schema.
P2: barrier Ready-pod selection via shared jsonpath helper with fixtures; barrier
pod fsGroup; script-generated pods inspected by the policy check; World0 PV claim
references protected; ci-paths.sh uses the event's real base and runs on
uncertainty; fault-drill asserts RTO and needs positive quarantine evidence.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ratifies dev/stage RPO/RTO from measurements and keeps World0 launch blocked
(policy-enforced) until acknowledged outcomes survive host loss in an
independent failure domain. Adds in-place restore with an explicit rewind,
barrier-based fencing of a superseded database, and promotion of stage-pinned
digests only. dr-check.sh rehearses host loss in every PostgreSQL build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A drill-only evo-server --synthetic-inputs development-v1 driver feeds
deterministic inputs from the committed head after every recovery, with
request IDs that match acknowledgements across owners (ADR-0073); it is
off in Git and the policy check keeps it off. fault-drill.sh loads dev
or stage and runs repeated forced kills (RTO, zero lost
acknowledgements), a full object volume (nothing acknowledged), an
empty volume (quarantine, never a fresh world) and an operator-driven
database outage (stale epoch never resumes), restoring the Git
configuration on exit. Process tests prove contiguous acknowledgement
across a SIGKILL and exact resumption after the committed head.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Complete the maintainer's server image bump to edb39f by also setting
the region build identity, and add REGION_RETAINED so the host keeps
the builds that wrote its history on its recovery allowlist; without it
the upgraded host would quarantine its own head as unsupported. Record
the M8.4 drill (rollout 3.0 s, forced deletion 19.3 s to Ready, epochs
3 to 5) and the maintainer's M8.5 sync.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deploy evo-region per environment (ADR-0071): one replica with the
Recreate strategy (a rolling update would deadlock on the lease),
/readyz readiness, /livez liveness, no CPU limit and a drain inside the
grace period, with the image digest doubling as the build identity. A
new idempotent evo-world-clone provision subcommand creates genesis from
development_opening (byte-identical to the qualification fixture) as a
Sync hook after the migration. Dev and stage paths include their region;
a dev AppProject allows Deployment and Service; region-drill.sh times
rollout and forced-deletion recovery; the policy check enforces the
host's invariants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
At the maintainer's request everything but Secrets and the store node
label now goes through Argo CD. aevolution-platform syncs one bundle
(World0 admission policy, Retain storage class, each environment's
namespace, guardrails, service accounts and object claim) with
self-heal, no prune and Prune=false,Delete=false on every object; the
admission policy also protects the platform app and project. Store Jobs
become kustomize components used as PreSync (migrate) and PostSync
(check-store) hooks per environment, and the policy check refuses any
unresolved marker in a path an Application syncs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep the filesystem object store on a node-local Retain-class PVC per
environment (ADR-0070), with the database on the external PostgreSQL VM.
Because local-path does not enforce sizes, the object store now refuses
new objects below a free-space reserve with a distinct full error, so a
full volume publishes and acknowledges nothing. evo-world-clone gains
check-store to qualify volume retention, reserve, database version/TLS
and the runtime privilege boundary from inside the cluster; it now ships
in the server image. Add the storage class, object claims, hand-applied
store Jobs, policy rules forbidding password env vars, and the runbook.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Per-environment platform layers (dev, stage, world0) with restricted pod
security, quotas, limit ranges, default-deny networking and token-less
service accounts, applied by an administrator rather than Argo. A native
ValidatingAdmissionPolicy denies deleting World0 data, identity and Argo
objects to every requester without the break-glass group. Stage and
World0 Argo projects are scoped to one namespace, World0 blocks
automated sync, and a cleanup tool refuses World0. An offline policy
check runs as a new CI suite; a live verification script uses only
can-i, reads and server dry runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the
barrier takes a database-wide upgrade lock exclusively, while
acquisitions, lease checks, provisioning and the retired-build trigger
take it shared first, so a queued acquisition can no longer regain
authority for a just-retired build and concurrent barriers cannot
retire the last current build. Provisioning takes a per-world lock
before classifying a world as original or a clone of one source. The
operations listener makes at most four accept attempts per serve().
Race regressions prove each contender is blocked before commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Clone a verified committed head into an isolated world and database
(ADR-0067): world re-identification with a proven byte round trip, a
schema v7 clone genesis naming the source commit and digest with digested
inherited outcomes, a read-only source session, an owner-only atomic
target transaction and the evo-world-clone tool. audit_region_history
re-verifies every retained link. Real-process tests cover trajectory
equality after the cut, source isolation, killed clones and retention.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0066: evo-server --ops-listen serves /livez (loop responding, never
authority), /readyz (200 only while the host holds the lease with a
verified recovered head) and /metrics (Prometheus text: 30 fixed
series for readiness, phase, lease epoch/remaining, simulated/committed
ticks, commit lag, checkpoint age, tick debt, queue use and bounds, and
counters; the only labels are a 6-value phase and 4-value drop reason).
The dependency-free OpsServer runs on the host loop with no thread:
GET on exact paths, 405/404/400/431 responses, 8 connections with excess
refused on accept, 4 KiB requests, 1 s deadlines, 16 KiB responses.
HostStatus/phase types moved to the portable evo::ops library with a
Windows-tested renderer. evo-server now refuses to start without its
required identity and connection settings.
Adds the server-runtime Docker target (evo-server + pinned libpq,
UID 10001, no credentials), a CI server-image job with smoke script and
optional publishing, and a Kubernetes example outside Argo.
Validated: Windows CTest 59/59; Linux Debug/Release/clang-tidy/ASan
CTest 59/59; PostgreSQL suites under Release, clang-tidy and ASan with
real process probes; two ops mutations caught. Docker build/smoke and
Gitea CI not executed locally.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0065: before any readiness, recovery now verifies the head's tail.
The trace object must exist with its manifest digest, its header must
bind profile, region, predecessor and range; the predecessor (schema v6
load_region_commit) must hash to the digest the head names; replaying
the trace from the predecessor's verified payload must reproduce every
report and the exact head bytes; and trace-derived decisions must equal
the head's durable decisions. Missing, corrupt or non-reproducing tails
quarantine (RecoveryError::tail); nothing older or empty is served.
Test-only crash seams (HostConfig::crash_hook, PublishHooks) and
evo-server --crash-at deliver real SIGKILL at before-upload,
before-transaction, after-commit and after-ack. The kill matrix,
a SIGSTOP/SIGCONT stale owner and damaged-head startups all behave as
the durability contract requires; successors verify before readiness.
Validated: PostgreSQL suites under GCC Release, clang-tidy and
ASan/UBSan; tail-skip and replay-equality mutations each caught. No
Docker/Gitea CI, production-lease kill test or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0064: RegionHost<State> (environmental and lineage profiles) is a
single-owner, single-threaded state machine advanced by step(now):
acquire the fenced lease and recover the verified head, run, resolve
ambiguous commits, drain and stop, or quarantine. Admission is
provisional and bounded (64 inputs, 64 KiB) with backpressure; FinalAck
and FinalOutcome are emitted only after a confirmed commit, and inputs
that will never be acknowledged are reported with a reason. Ticks follow
the caller's scheduling clock with bounded catch-up and measured tick
debt, never invented inputs. A death seals the batch and commits in the
same step, so no dependent tick uses an uncommitted outcome. Storage
outages pause and retry, unknown outcomes resolve by exact bytes,
fencing discards unconfirmed work and re-acquires; readiness fails
closed. Drain commits pending ticks and releases the lease.
evo-server hosts one region with a bounded stdin input feed, stdout
event lines, a readiness file and SIGTERM drain (exit 0/3/4).
Validated: Windows CTest 58/58; PostgreSQL suites under GCC Release,
clang-tidy and ASan/UBSan including real-process SIGTERM drain and
SIGKILL with successor recovery; four host mutations each caught. No
gateway, metrics, Docker/Gitea CI or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review P1 (object durability): a name installed by a put whose directory
fsync failed was later reused as success without syncing again, so a
publication could acknowledge an object that might vanish on power
loss. Reuse (existing and lost-link-race paths) now fsyncs the verified
file and the directory before succeeding, and namespace opens always
sync their parent. A test-only ObjectStoreFaults hook injects directory
sync failures; tests cover failed sync then retries, namespace creation
syncs, and publication refusing to proceed until durability holds.
Review P1 (pre-v5 history): v5 recovery requires one durable decision
per terminal actor, but v3/v4 published checkpoints carry none, so an
accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now
runs before 0005 in every upgrade path and refuses (EVL10) when any
published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is
unchanged. CI publishes a real v4 checkpoint and proves the upgrade is
refused with nothing changed; a committed death without a decision is
shown to recover as quarantined history.
Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2;
reverting each fix is caught by its test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0063: manifest v2 (360 bytes) binds a per-tick validation trace
object and a canonical decision block by SHA-256. Publication replays the
trace from the committed predecessor's verified payload and requires
every report and the candidate bytes, derives deaths deterministically,
uploads payload and trace with verified read-back, then sends manifest
and decisions to one fenced statement. RegionBatch seals at a terminal
transition or 64 ticks and refuses ticks until durable confirmation.
Schema v5 records the checkpoint, its decisions and outbox rows
atomically; a unique index keeps one terminal outcome per actor
(EVL08), triggers make commits/decisions immutable (EVL09), an exact
retry of a committed ID returns the original receipt, changed bytes
conflict, and read-only resolve_region_commit answers committed/absent/
conflict. Outbox pending/acknowledge give idempotent at-least-once
delivery. Recovery reconciles durable deaths with restored lifecycle
(and archive deaths for the lineage profile).
Validated: Windows CTest 58/58; Linux GCC Debug/Release, clang-tidy and
ASan/UBSan CTest 58/58 each; PostgreSQL suites Release, analysis, ASan
(lost response, committed-after-timeout, terminated orphan retry,
changed bytes, repeated/partial/reordered decisions, gap, outbox
redelivery, history quarantine); five mutations each caught. No
Docker/Gitea CI, process kill, broker or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ADR-0062 adds a canonical LineageArchive codec (parents before children,
organisms by id, FNV) that decodes by replaying admit/record_birth/
record_death and requires byte-identical re-encoding; it never overwrites
an existing archive. LineageRegion binds the environmental producer and
consumer accounts to archived organisms with identity high-waters,
enforces closure (lifecycle vs deaths, exact living set, tick bounds)
and records deaths in the tick that causes them.
Checkpoint profile 2 (environmental-lineage-v1, history profile 1)
publishes that container; schema v4 admits it with a 4,268,144-byte cap
and requires each commit to keep its predecessor's profile, RNG, history
and durability declarations. Linux adds provision/publish/recover for
the profile on the shared publication path.
Qualification found WSL2 stepping CLOCK_REALTIME back ~0.5 s every ~32 s,
which made two 1 s-lease tests flaky under ASan. Tests now wait on
database-time expiry; ADR-0060 records the slewed-clock requirement.
Validated: Windows CTest 57/57; Linux GCC Debug/Release, clang-tidy and
ASan/UBSan CTest 57/57 each; PostgreSQL suites 2x Release, 2x analysis,
5x ASan; SQL and C++ mutations each caught. No Docker/Gitea CI, remote
store, process kill or deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>