feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)

Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 17:51:17 -04:00
co-authored by Claude Opus 5.5
parent 645d16b27a
commit 5b482ad6a3
19 changed files with 1028 additions and 12 deletions
+8
View File
@@ -829,3 +829,11 @@ durable failure domain exists, enforced by the deploy policy check; host-loss
restore in place with barrier fencing of the superseded database rehearsed
locally; promotion only of stage-pinned digests. **M8-dev/stage passed;
M8-World0 launch not passed.**
M8.9 implemented and locally qualified (2026-09-30), under ADR-0076: accelerated
clone runs of the current and candidate builds compared state-for-state and
outcome-for-outcome (`evo-world-clone compare`); failed-migration recovery and the
version barrier rehearsed with real hosts; `upgrade-barrier.sh` for dev/stage;
stores/compute node labels with policy-checked placement and `scheduling-check.sh`.
Rules changes stay refused until a converter ADR. In-cluster barrier and
additional-node evidence wait on the maintainer.
+18
View File
@@ -1141,3 +1141,21 @@ ADR-0075:
- `fence-database.sh` supersedes the old database through the upgrade barrier: a
still-running old host quarantines, and no host can acquire it again.
- `tools/ci/dr-check.sh` rehearses it in every PostgreSQL build.
## M8.9 upgrade and growth (2026-09-30)
ADR-0076:
- **Compare before upgrading.** `evo-server --synthetic-until TICK --accelerate N`
(drill-only, refused without the synthetic driver) runs clones to an exact tick
faster than real time; `evo-world-clone compare` (library:
`compare_region_histories`) audits both and requires the same world: same
versions, identical rebound head state and identical outcomes. Builds may differ.
- **Failed migration and barrier rehearsed** with real hosts
(`tools/ci/upgrade-check.sh`): a failing migration leaves schema v9 and the host
serving; the barrier fences the old build (exit 4), the new one resumes under a
newer epoch, and the history across both builds audits.
- **Dev/stage barrier:** `tools/ops/upgrade-barrier.sh <env> --retire HEX` retires an
old build only when it is in `REGION_RETAINED` and the new digest is already Ready.
- **Nodes:** `aevolution.io/stores` (one worker; every objects-volume pod selects it)
and `aevolution.io/compute` (preferred only; Alertmanager first).
`tools/ops/scheduling-check.sh <env>` verifies placement read-only.
+152
View File
@@ -0,0 +1,152 @@
# ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth
## Status
Accepted for M8.9, 2026-09-30. It builds on:
- ADR-0067 (clones);
- ADR-0068 (upgrade barrier);
- ADR-0073 (synthetic driver);
- ADR-0074/0075 (backup, restore, World0 gate).
## Context
M8.9 requires:
- clone, migrate, accelerate and compare before any world-wide rules change;
- a node label and affinity plan that works on one node and admits added compute
nodes.
The evidence asked for is:
- a dev/stage version-barrier rehearsal;
- recovery from a failed migration;
- scheduling onto an additional node where hardware exists.
The named risks are mixed physical laws, and assuming that added CPU makes storage
portable.
**No rules or format converter exists** (ADR-0068): a region never changes rules,
versions, quantum or profile. The only upgrade this foundation can perform is a
**build change with identical physical laws**, such as a bug fix, performance
work or a refactor. The question the rehearsal must answer is: does the candidate
build run the same world?
## Decision
**1. Clone, accelerate, compare.** Two drill-only `evo-server` flags, each refused
without `--synthetic-inputs`:
- `--synthetic-until TICK` stops the deterministic driver after that tick. Inputs
are never invented, so the region commits exactly up to TICK and idles.
- `--accelerate N` (2–1000) runs the scheduling clock N times faster than real
time. Lease deadlines stay real.
`evo-world-clone compare --world A --other-world B ...` calls the new library
function `compare_region_histories`:
- It fully audits both histories.
- It then requires equal head tick, frame, quantum, versions and history profile.
- It requires identical head states once B is rebound to A's world.
- It requires identical durable outcomes: tick, actor, outcome and organism,
including those inherited at clone genesis.
- Builds may differ. Exit 6 means "different world".
The procedure:
1. Clone the live world's cut twice into an isolated database.
2. Run the current build on one clone and the candidate on the other, with the same
inputs, accelerated, to the same tick.
3. `compare` must pass. Only then may the candidate proceed.
A rules change cannot pass this procedure. It stays refused until a converter ADR
defines its own comparison: ecological distributions and approval
(PERSISTENCE_ARCHITECTURE.md). Mixed physical laws are therefore impossible by
construction: a barrier admits only a build that `compare` proved runs the same
world.
**2. Migration failure.** A migration runs in one transaction under the migration
lock, with a 5 s lock timeout and a 10 s statement timeout (ADR-0068). A failure
leaves the previous schema exactly.
- While the migration holds its locks, a publication may wait. Its outcome is then
resolved (ADR-0063), or its lease lapses and the host re-acquires under a newer
epoch.
- In a cluster, the migration Job is a Sync hook at wave −1. When it fails, the
sync stops before wave 1, so the Deployment keeps the old image and nothing new
starts.
- Recovery means fixing the migration and syncing again. No database restore is
involved: a restore after new writes would itself be a rewind (ADR-0075).
**3. Barrier in dev/stage.** The ADR-0068 order has these steps:
1. Migrate.
2. Retain the new build. The provision hook does this.
3. Deploy the new build with the old build listed in `REGION_RETAINED`. The
Recreate strategy drains the old pod first.
4. Run `tools/ops/upgrade-barrier.sh <env> --retire OLD`.
The barrier tool refuses the retirement unless:
- the build is not the one Git deploys;
- it is listed in `REGION_RETAINED`;
- a Ready pod already runs the Git-pinned digest.
It then runs `upgrade_barrier` in a short Pod with the migrator login, using the
`migrate` component's egress, and measures the not-ready gap while the new host
re-acquires. A rollback to a retired image quarantines. That is intended.
**4. Nodes.**
| Label | Meaning | Rule (policy-checked) |
|---|---|---|
| `aevolution.io/stores=true` | Holds the node-local durable volumes. Exactly one worker (today rke2-worker-1). | Every pod that mounts `aevolution-objects` must select it. Region hosts, provisioning, store checks and backups stay there. |
| `aevolution.io/compute=true` | Admitted for stateless AEvolution work. | Only ever *preferred*, never required, so everything still schedules on one node or with no labelled node. Alertmanager is the first user. |
Adding a node means labelling it `aevolution.io/compute=true`, then running
`tools/ops/scheduling-check.sh <env>`. Only stateless pods move. A region moves to
another node only as a data operation: a restore or a clone onto that node's
storage (ADR-0074/0075). Rescheduling a region does not move it, because its
objects volume is pinned to the stores node by the local-path PV's node affinity.
The policy check rejects:
- a pod that mounts the objects volume without the stores selector;
- a required compute affinity;
- rehearsal flags in a deployed workload.
## Evidence
Every PostgreSQL build runs two tests.
`region_upgrade_tests` (real processes):
- A source world is run past its death to tick 30 and cloned four times.
- Build b1 and candidate build b2 each run to tick 150 at 20× acceleration: 120
ticks, 6 s of region time, in under 4 s of wall time.
- `compare` passes, with one outcome.
- A clone fed a temperature-shifted input stream from the same cut differs in
`state`.
- A clone stopped at tick 100 differs in `ticks`.
- Comparing a world with itself, or using the rehearsal flags without the driver,
is a usage error.
`tools/ci/upgrade-check.sh`:
- A migration that takes the migration lock, adds a column to `region_commits`,
holds its locks for 3 s and then fails runs while b1 serves. Afterwards the
schema is still v9, the column does not exist, b1 goes on acknowledging, and the
real migrate is a no-op.
- b2 is retained and started, and waits without becoming ready.
- `upgrade_barrier(b1)` fences one lease. b1 quarantines (exit 4). b2 resumes at a
newer epoch, at or past every b1 acknowledgement, and continues with the next
tick.
- A late b1 host quarantines at once.
- The history, now written by both builds, passes the full audit.
In the cluster, still to be run by the maintainer:
- `upgrade-barrier.sh` on dev, retiring a build listed in `REGION_RETAINED`;
- `scheduling-check.sh` on dev and stage, before and after labelling a second
worker `aevolution.io/compute=true`.
## Consequences
- Upgrades that keep the physical laws are qualified and operable in dev and stage.
Rules changes remain impossible until a converter ADR exists.
- Mutation tests show the compare check is not vacuous: removing the state
comparison, the stop tick or the acceleration each fails the tests.
- Region hosts stay on the stores node. Horizontal growth applies only to
stateless work until object storage is network-attached. The storage rework
(backlog) may revisit this.
## Alternatives rejected
- **Comparing live regions at arbitrary commits.** Commit boundaries depend on flush
timing. A shared stop tick makes the comparison exact.
- **Required node affinity for compute.** It would break the single-node plan and
strand pods while no node carries the label.
- **Moving a region by relabelling nodes.** The volume stays behind. That is
exactly the named risk.
- **Retiring the old build automatically on every sync.** Retirement is
irreversible. It stays a deliberate step, guarded by the checks above.
+1
View File
@@ -86,3 +86,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
- [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending.
- [ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion](0075-world0-acknowledgment-gate.md) - accepted for M8.8 ratified dev/stage RPO/RTO, World0 launch blocked (policy-enforced) until acknowledgements survive host loss in an independent failure domain, explicit-rewind in-place restore, barrier-based fencing of a superseded database, promotion only of stage-pinned digests.
- [ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth](0076-upgrade-and-growth.md) - accepted for M8.9 accelerated clone runs compared state-for-state before a build change, failed-migration recovery, guarded in-cluster barrier, stores/compute node labels with policy-checked placement.
+42
View File
@@ -2992,3 +2992,45 @@ Fixes (this segment):
- any in-cluster host-loss restore or fence;
- a second failure domain, which does not exist;
- any World0 deployment or promotion, which is blocked by design.
## M8.9 upgrade rehearsal and node plan - 2026-09-30
- `region_upgrade_tests` (real evo-server and evo-world-clone processes) passed in
the Release, analysis and ASan/UBSan PostgreSQL builds:
- Current build b1 and candidate b2 ran on clones of one cut, at 20×
acceleration, from tick 30 to tick 150: 120 ticks, 6 s of region time. Wall
times were 306/307 ms (Release), 322/323 ms (analysis) and 299/299 ms (ASan).
- `compare` passed: `compared tick=150 outcomes=1`.
- A temperature-shifted input stream gave `compare differs state` (exit 6).
- A clone stopped at tick 100 gave `compare differs ticks` (exit 6).
- Comparing a world with itself, and `--accelerate` or `--synthetic-until`
without the driver, were usage errors (exit 2).
- `tools/ci/upgrade-check.sh` passed in all three builds:
- A failing migration (migration lock, `ALTER TABLE region_commits`, locks held
3 s, then an error) left schema v9 and no new column. b1 went on
acknowledging, and the real migrate was a no-op.
- b2 waited while b1 held the lease. `upgrade_barrier(b1)` fenced one lease, and
b1 exited 4.
- b2 resumed at `epoch=2 committed=135`, at b1's last acknowledgement (135), and
continued with tick 136. The ASan run: 137.
- A late b1 host exited 4 without becoming ready.
- The history audit across both builds passed: 24 commits to tick 167.
- Mutations, each caught:
- `compare` without the state comparison: upgrade test line 201 (the diverged
world passed);
- `--synthetic-until` ignored: line 165;
- `--accelerate` as a no-op: line 184 (over 4 s).
- The WSL `checks` set: 59/59 in debug, release, analysis and ASan. clang-tidy
(analysis builds) and the format check were clean.
- Deploy policy: 67 objects, 0 failures. Negatives, each rejected: an
objects-volume pod without the stores selector; a required compute affinity;
`--accelerate` in the region Deployment. `upgrade-barrier.sh` and
`scheduling-check.sh` refuse World0 before kubectl.
- Found during validation: genesis seed 0xF1 collided with a checkpoint-store test
(fixture provisioning failed), and was replaced by the unused 0xD7. clang-tidy
flagged duplicated option branches, which were merged.
- Not executed:
- `upgrade-barrier.sh` or `scheduling-check.sh` against dev or stage;
- scheduling onto a second labelled node;
- the Windows MSVC build (no Windows target changed; the server and PostgreSQL
code are Linux-only).
+17
View File
@@ -232,3 +232,20 @@ rebuilt as isolated world ...` counts as restore evidence. The restore uses fres
roles on a throwaway cluster; nothing reaches the source database. A restored copy
keeps the source world identity and is never served; restoring in place after host
loss is M8.8.
## Upgrades and nodes (M8.9, ADR-0076)
**Build upgrade (same rules).**
1. Rehearse on clones first: clone the cut twice into an isolated database, run the
current and candidate builds with `--synthetic-inputs development-v1
--synthetic-until T --accelerate 20`, then `evo-world-clone compare` must exit 0.
2. Pin the new server digest in `deploy/kubernetes/region/<env>` (image and
`REGION_BUILD`), add the old digest to `REGION_RETAINED`, and sync. The migrate
hook (wave -1) stops the sync on failure, leaving the old host running.
3. When the new pod is Ready: `tools/ops/upgrade-barrier.sh <env> --retire OLD`.
Keep OLD in `REGION_RETAINED`: its history stays recoverable.
**Adding a node.** Label it `aevolution.io/compute=true` and run
`tools/ops/scheduling-check.sh <env>`. Only stateless pods (Alertmanager today) may
move there. Never move `aevolution.io/stores` to rebalance: the objects volume stays
on its node; a region moves only by restore or clone onto the new node's storage.