feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)
Drill-only --synthetic-until/--accelerate and evo-world-clone compare (compare_region_histories) prove a candidate build runs the same world on clones of one cut before any barrier. upgrade-check.sh rehearses a failed migration under load and the version barrier with real hosts; upgrade-barrier.sh retires an old build in dev/stage only once the new digest is Ready. Nodes: aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is preference-only (Alertmanager first), checked by policy and scheduling-check.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2992,3 +2992,45 @@ Fixes (this segment):
|
||||
- any in-cluster host-loss restore or fence;
|
||||
- a second failure domain, which does not exist;
|
||||
- any World0 deployment or promotion, which is blocked by design.
|
||||
|
||||
## M8.9 upgrade rehearsal and node plan - 2026-09-30
|
||||
|
||||
- `region_upgrade_tests` (real evo-server and evo-world-clone processes) passed in
|
||||
the Release, analysis and ASan/UBSan PostgreSQL builds:
|
||||
- Current build b1 and candidate b2 ran on clones of one cut, at 20×
|
||||
acceleration, from tick 30 to tick 150: 120 ticks, 6 s of region time. Wall
|
||||
times were 306/307 ms (Release), 322/323 ms (analysis) and 299/299 ms (ASan).
|
||||
- `compare` passed: `compared tick=150 outcomes=1`.
|
||||
- A temperature-shifted input stream gave `compare differs state` (exit 6).
|
||||
- A clone stopped at tick 100 gave `compare differs ticks` (exit 6).
|
||||
- Comparing a world with itself, and `--accelerate` or `--synthetic-until`
|
||||
without the driver, were usage errors (exit 2).
|
||||
- `tools/ci/upgrade-check.sh` passed in all three builds:
|
||||
- A failing migration (migration lock, `ALTER TABLE region_commits`, locks held
|
||||
3 s, then an error) left schema v9 and no new column. b1 went on
|
||||
acknowledging, and the real migrate was a no-op.
|
||||
- b2 waited while b1 held the lease. `upgrade_barrier(b1)` fenced one lease, and
|
||||
b1 exited 4.
|
||||
- b2 resumed at `epoch=2 committed=135`, at b1's last acknowledgement (135), and
|
||||
continued with tick 136. The ASan run: 137.
|
||||
- A late b1 host exited 4 without becoming ready.
|
||||
- The history audit across both builds passed: 24 commits to tick 167.
|
||||
- Mutations, each caught:
|
||||
- `compare` without the state comparison: upgrade test line 201 (the diverged
|
||||
world passed);
|
||||
- `--synthetic-until` ignored: line 165;
|
||||
- `--accelerate` as a no-op: line 184 (over 4 s).
|
||||
- The WSL `checks` set: 59/59 in debug, release, analysis and ASan. clang-tidy
|
||||
(analysis builds) and the format check were clean.
|
||||
- Deploy policy: 67 objects, 0 failures. Negatives, each rejected: an
|
||||
objects-volume pod without the stores selector; a required compute affinity;
|
||||
`--accelerate` in the region Deployment. `upgrade-barrier.sh` and
|
||||
`scheduling-check.sh` refuse World0 before kubectl.
|
||||
- Found during validation: genesis seed 0xF1 collided with a checkpoint-store test
|
||||
(fixture provisioning failed), and was replaced by the unused 0xD7. clang-tidy
|
||||
flagged duplicated option branches, which were merged.
|
||||
- Not executed:
|
||||
- `upgrade-barrier.sh` or `scheduling-check.sh` against dev or stage;
|
||||
- scheduling onto a second labelled node;
|
||||
- the Windows MSVC build (no Windows target changed; the server and PostgreSQL
|
||||
code are Linux-only).
|
||||
|
||||
Reference in New Issue
Block a user