feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)

Drill-only --synthetic-until/--accelerate and evo-world-clone compare
(compare_region_histories) prove a candidate build runs the same world on
clones of one cut before any barrier. upgrade-check.sh rehearses a failed
migration under load and the version barrier with real hosts; upgrade-barrier.sh
retires an old build in dev/stage only once the new digest is Ready. Nodes:
aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is
preference-only (Alertmanager first), checked by policy and scheduling-check.sh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 17:51:17 -04:00
co-authored by Claude Opus 5.5
parent 645d16b27a
commit 5b482ad6a3
19 changed files with 1028 additions and 12 deletions
+42
View File
@@ -2992,3 +2992,45 @@ Fixes (this segment):
- any in-cluster host-loss restore or fence;
- a second failure domain, which does not exist;
- any World0 deployment or promotion, which is blocked by design.
## M8.9 upgrade rehearsal and node plan - 2026-09-30
- `region_upgrade_tests` (real evo-server and evo-world-clone processes) passed in
the Release, analysis and ASan/UBSan PostgreSQL builds:
- Current build b1 and candidate b2 ran on clones of one cut, at 20×
acceleration, from tick 30 to tick 150: 120 ticks, 6 s of region time. Wall
times were 306/307 ms (Release), 322/323 ms (analysis) and 299/299 ms (ASan).
- `compare` passed: `compared tick=150 outcomes=1`.
- A temperature-shifted input stream gave `compare differs state` (exit 6).
- A clone stopped at tick 100 gave `compare differs ticks` (exit 6).
- Comparing a world with itself, and `--accelerate` or `--synthetic-until`
without the driver, were usage errors (exit 2).
- `tools/ci/upgrade-check.sh` passed in all three builds:
- A failing migration (migration lock, `ALTER TABLE region_commits`, locks held
3 s, then an error) left schema v9 and no new column. b1 went on
acknowledging, and the real migrate was a no-op.
- b2 waited while b1 held the lease. `upgrade_barrier(b1)` fenced one lease, and
b1 exited 4.
- b2 resumed at `epoch=2 committed=135`, at b1's last acknowledgement (135), and
continued with tick 136. The ASan run: 137.
- A late b1 host exited 4 without becoming ready.
- The history audit across both builds passed: 24 commits to tick 167.
- Mutations, each caught:
- `compare` without the state comparison: upgrade test line 201 (the diverged
world passed);
- `--synthetic-until` ignored: line 165;
- `--accelerate` as a no-op: line 184 (over 4 s).
- The WSL `checks` set: 59/59 in debug, release, analysis and ASan. clang-tidy
(analysis builds) and the format check were clean.
- Deploy policy: 67 objects, 0 failures. Negatives, each rejected: an
objects-volume pod without the stores selector; a required compute affinity;
`--accelerate` in the region Deployment. `upgrade-barrier.sh` and
`scheduling-check.sh` refuse World0 before kubectl.
- Found during validation: genesis seed 0xF1 collided with a checkpoint-store test
(fixture provisioning failed), and was replaced by the unused 0xD7. clang-tidy
flagged duplicated option branches, which were merged.
- Not executed:
- `upgrade-barrier.sh` or `scheduling-check.sh` against dev or stage;
- scheduling onto a second labelled node;
- the Windows MSVC build (no Windows target changed; the server and PostgreSQL
code are Linux-only).