Commit Graph
15 Commits
Author SHA1 Message Date
dingoandClaude Opus 5.5 926537ef3a Add an operator script that retires a region's player lives
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 4s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 4m1s
Foundation / postgres-tools (push) Successful in 6s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 5s
DEV runs one region at a time and moves to a new one before test cells die. A life
left in the old region keeps its account's home there: the grant broker routes the
account to a region nobody serves (503 region_unavailable) and any other region
refuses its birth (existing life, one active life per world).

tools/ops/retire-region-lives.sql, run as the database owner, ends each published
life of the region at the region's latest commit (as end_cellular_lives does) and
cancels unpublished reservations. It refuses while a server holds the region's lease,
is a dry run unless confirm=yes, is idempotent, and leaves checkpoints and history
unchanged. Tested on a disposable migrated cluster.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 10:47:09 -04:00
dingoandClaude Opus 5.5 9f9a93d4c8 M10.9d: watermark coalescing, budgets v2, evo-load action ids
Foundation / checks (client) (push) Successful in 14s
Foundation / checks (checks) (push) Successful in 16s
Foundation / checks (deploy) (push) Successful in 15s
Foundation / checks (network) (push) Successful in 18s
Foundation / checks (postgres) (push) Successful in 1m34s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (tsan) (push) Successful in 5m44s
Foundation / server-image (push) Successful in 1m0s
Foundation / postgres-tools (push) Successful in 1m38s
Foundation / image (push) Successful in 2m34s
- IntentLimits::watermark_coalesce (evo-server: every 4, coalesce 4); ADR-0092 amendment.
- m10 load-report splits poll/replicate and commit-tick/other-tick maxima.
- Budgets v2: work p95 <= 25 ms over all ticks, commit ticks <= 50 ms (ADR-0097);
  moving the commit off the tick is a recorded follow-up.
- evo-load: time-based action ids, duplicate_actions, receipts before outcomes;
  drill ramps at 500 ms and judges by v2.
- Refresh the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 18:00:41 -04:00
dingoandClaude Opus 5.5 46b6358606 Share replication encoding per interest; fix dev retained builds
Foundation / checks (deploy) (push) Successful in 3m26s
Foundation / checks (ecs) (push) Successful in 1m54s
Foundation / checks (client) (push) Successful in 12m36s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m48s
Foundation / checks (checks) (push) Successful in 32m51s
Foundation / checks (postgres) (push) Successful in 46m20s
Foundation / postgres-tools (push) Successful in 1m35s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m14s
ADR-0097. The second evolution-dev load drill (image 7952185d) held 20
sessions with client budgets met (response p95 111 ms, outcome p95 about
1.2 s, 20 Hz, no starvation) but the server over its own: tick work p99
48-70 ms (25 ms budget), session layer 1.2-2.0 ms per session at p99
(0.5 ms budget), governor refusing and shedding to level 3 to little
effect. ReplicationServer::tick re-encoded every environment cell and
organism for each session. Sessions with the same requested interest
(no perception predicate) now share one encoding per tick; markers stay
per session. Locally 21 sessions: about 970 us -> 360 us per tick.
The drill now waits for two consecutive calm reports before the bots.

The dev overlay's REGION_RETAINED listed the deployed build itself
(61df6a1, f88bee2), which the deploy policy refuses (CI deploy failed on
those commits); it now lists only the other builds, keeping 6c96629e,
the build of the region head. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 15:03:45 -04:00
dingoandClaude Opus 5.5 d173f076ea Fix load governor startup refusal and drill issues from the first dev run
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 2m0s
Foundation / checks (client) (push) Successful in 12m24s
Foundation / checks (network) (push) Successful in 13m35s
Foundation / checks (tsan) (push) Successful in 7m58s
Foundation / checks (checks) (push) Successful in 32m32s
Foundation / checks (postgres) (push) Successful in 45m25s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m32s
Foundation / server-image (push) Successful in 4m15s
ADR-0097. The first evolution-dev load drill (image 6c96629e) did not
measure capacity: the governor refused six seconds after the restart with
no session (p95 over a window of only startup ticks), and the bots spent
their grants retrying every 2 s. Reference numbers with one session: tick
work p95 4.0 ms with 33 ms spikes, session layer p95 0.68 ms.

- The governor acts on work only over a full 40-tick window; tick debt
  still counts at once.
- evo-load backs off 10 s after a load refusal and, in bundle mode,
  reuses the grant the region refused before using it.
- The drill waits for a calm load-report before the bots start, takes the
  drill pod's whole log before restoring (the live follow died), and
  confirms the restore against a pod other than the drill pod.

Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 11:01:02 -04:00
dingoandClaude Opus 5.5 ada77fd761 Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 08:38:13 -04:00
dingoandClaude Opus 5.5 8e319322f4 Implement M9.7 dev region sessions with durable no-effect intentions
Run a non-blocking TLS 1.3 session server inside evo-server's loop:
environment views come from the region's own field/terrain stores with an
explicitly empty organism presentation; admitted intentions ride the next
synthetic input (content unchanged) and return a durable no-effect outcome
from the host's final acknowledgement, or not_committed when dropped.
Sessions are bound to their connection, epochs close stale sessions, and
connections have bounded output, input rate and deadlines.

SessionClient and evo-client --connect consume one single-use grant per
Hello (refusals before admission keep it), reset to a fresh baseline on
every session, keep stale views visible and report in-flight intentions as
unknown after loss. evo-dev-session generates dev credentials, grant
bundles and runs a restart probe. The dev deployment gains a drill-time,
loopback-only listener with an optional Secret, enforced by the policy
check, and tools/ops/session-drill.sh. Protocol v1 gains IntentOutcome and
the capacity close reason before any deployment. ADR-0083 records the
maintainer-confirmed decisions; M8-dev/stage remains not passed and no
cluster change was made.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-01 20:12:18 -04:00
dingoandClaude Opus 5.5 3a34ae39bc fix: M8 review findings (restore ownership, barrier tool, World0 claimRef, CI selection, drill verdicts)
P1: restore-world.sh claims its objects path (mkdir) and the database (a held
session with a restore try-lock and the exclusive upgrade lock), and removes only
what it created, so a losing restore never drops a winner's schema.
P2: barrier Ready-pod selection via shared jsonpath helper with fixtures; barrier
pod fsGroup; script-generated pods inspected by the policy check; World0 PV claim
references protected; ci-paths.sh uses the event's real base and runs on
uncertainty; fault-drill asserts RTO and needs positive quarantine evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 22:53:11 -04:00
dingoandClaude Opus 5.5 5b482ad6a3 feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)
Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 17:51:17 -04:00
dingoandClaude Opus 5.5 4a900b0bd3 fix(server): back off commit retries after store failures; sharpen fault drill
The first dev drill showed a full volume driving ~140 refused publications a
second. Commit retries now back off 100 ms doubling to a 2 s cap, reset on the
next confirmed commit. fault-drill.sh reports the empty-volume pod's exit code,
restarts and logs, and prints node-specific iptables rules for db-outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 15:12:58 -04:00
dingoandClaude Opus 5.5 6b5da397e8 fix(ops): list live region pods with a go-template in fault-drill
kubectl jsonpath has no negation, so the deletionTimestamp filter was a
parse error on every call. Use a go-template that skips pods already
being deleted; verified against sample pods with Go's text/template,
which kubectl's go-template output uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 14:37:17 -04:00
dingoandClaude Opus 5.5 3f08a1d221 Add M8.6 synthetic load driver and fault drill
A drill-only evo-server --synthetic-inputs development-v1 driver feeds
deterministic inputs from the committed head after every recovery, with
request IDs that match acknowledgements across owners (ADR-0073); it is
off in Git and the policy check keeps it off. fault-drill.sh loads dev
or stage and runs repeated forced kills (RTO, zero lost
acknowledgements), a full object volume (nothing acknowledged), an
empty volume (quarantine, never a fresh world) and an operator-driven
database outage (stale epoch never resumes), restoring the Git
configuration on exit. Process tests prove contiguous acknowledgement
across a SIGKILL and exact resumption after the committed head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 10:52:01 -04:00
Jason A 2c79d4748c fix(ops): select region pods without deletion timestamp 2026-09-30 00:11:48 -04:00
dingoandClaude Opus 5.5 8ef69fab1d Add M8.4 region host deployment, provisioning hook and drill
Deploy evo-region per environment (ADR-0071): one replica with the
Recreate strategy (a rolling update would deadlock on the lease),
/readyz readiness, /livez liveness, no CPU limit and a drain inside the
grace period, with the image digest doubling as the build identity. A
new idempotent evo-world-clone provision subcommand creates genesis from
development_opening (byte-identical to the qualification fixture) as a
Sync hook after the migration. Dev and stage paths include their region;
a dev AppProject allows Deployment and Service; region-drill.sh times
rollout and forced-deletion recovery; the policy check enforces the
host's invariants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 21:32:56 -04:00
dingoandClaude Opus 5.5 bd4128d55c Record M8.2 live verification, backlog and accepted default-project risk
The maintainer applied the platform layer, World0 admission policy and
projects; verify-isolation.sh passed every guardrail, RBAC-negative,
admission dry-run and cleanup check. Its three project failures were a
script defect (destinations and the empty cluster whitelist were printed
together as "[]"); check them separately. Ignore local result files, add
docs/BACKLOG.md with break-glass membership and the storage re-probe, and
record leaving Argo's default project unrestricted as an accepted risk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 13:03:34 -04:00
dingoandClaude Opus 5.5 df0c4f4efb Add M8.2 environment isolation and World0 protection
Per-environment platform layers (dev, stage, world0) with restricted pod
security, quotas, limit ranges, default-deny networking and token-less
service accounts, applied by an administrator rather than Argo. A native
ValidatingAdmissionPolicy denies deleting World0 data, identity and Argo
objects to every requester without the break-glass group. Stage and
World0 Argo projects are scoped to one namespace, World0 blocks
automated sync, and a cleanup tool refuses World0. An offline policy
check runs as a new CI suite; a live verification script uses only
can-i, reads and server dry runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 12:43:58 -04:00