feat(ops): M8.9 upgrade rehearsal on clones, barrier and node plan (ADR-0076)
Drill-only --synthetic-until/--accelerate and evo-world-clone compare (compare_region_histories) prove a candidate build runs the same world on clones of one cut before any barrier. upgrade-check.sh rehearses a failed migration under load and the version barrier with real hosts; upgrade-barrier.sh retires an old build in dev/stage only once the new digest is Ready. Nodes: aevolution.io/stores pins every objects-volume pod, aevolution.io/compute is preference-only (Alertmanager first), checked by policy and scheduling-check.sh. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -829,3 +829,11 @@ durable failure domain exists, enforced by the deploy policy check; host-loss
|
||||
restore in place with barrier fencing of the superseded database rehearsed
|
||||
locally; promotion only of stage-pinned digests. **M8-dev/stage passed;
|
||||
M8-World0 launch not passed.**
|
||||
|
||||
M8.9 implemented and locally qualified (2026-09-30), under ADR-0076: accelerated
|
||||
clone runs of the current and candidate builds compared state-for-state and
|
||||
outcome-for-outcome (`evo-world-clone compare`); failed-migration recovery and the
|
||||
version barrier rehearsed with real hosts; `upgrade-barrier.sh` for dev/stage;
|
||||
stores/compute node labels with policy-checked placement and `scheduling-check.sh`.
|
||||
Rules changes stay refused until a converter ADR. In-cluster barrier and
|
||||
additional-node evidence wait on the maintainer.
|
||||
|
||||
@@ -1141,3 +1141,21 @@ ADR-0075:
|
||||
- `fence-database.sh` supersedes the old database through the upgrade barrier: a
|
||||
still-running old host quarantines, and no host can acquire it again.
|
||||
- `tools/ci/dr-check.sh` rehearses it in every PostgreSQL build.
|
||||
|
||||
## M8.9 upgrade and growth (2026-09-30)
|
||||
|
||||
ADR-0076:
|
||||
- **Compare before upgrading.** `evo-server --synthetic-until TICK --accelerate N`
|
||||
(drill-only, refused without the synthetic driver) runs clones to an exact tick
|
||||
faster than real time; `evo-world-clone compare` (library:
|
||||
`compare_region_histories`) audits both and requires the same world: same
|
||||
versions, identical rebound head state and identical outcomes. Builds may differ.
|
||||
- **Failed migration and barrier rehearsed** with real hosts
|
||||
(`tools/ci/upgrade-check.sh`): a failing migration leaves schema v9 and the host
|
||||
serving; the barrier fences the old build (exit 4), the new one resumes under a
|
||||
newer epoch, and the history across both builds audits.
|
||||
- **Dev/stage barrier:** `tools/ops/upgrade-barrier.sh <env> --retire HEX` retires an
|
||||
old build only when it is in `REGION_RETAINED` and the new digest is already Ready.
|
||||
- **Nodes:** `aevolution.io/stores` (one worker; every objects-volume pod selects it)
|
||||
and `aevolution.io/compute` (preferred only; Alertmanager first).
|
||||
`tools/ops/scheduling-check.sh <env>` verifies placement read-only.
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
# ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth
|
||||
|
||||
## Status
|
||||
Accepted for M8.9, 2026-09-30. It builds on:
|
||||
- ADR-0067 (clones);
|
||||
- ADR-0068 (upgrade barrier);
|
||||
- ADR-0073 (synthetic driver);
|
||||
- ADR-0074/0075 (backup, restore, World0 gate).
|
||||
|
||||
## Context
|
||||
M8.9 requires:
|
||||
- clone, migrate, accelerate and compare before any world-wide rules change;
|
||||
- a node label and affinity plan that works on one node and admits added compute
|
||||
nodes.
|
||||
|
||||
The evidence asked for is:
|
||||
- a dev/stage version-barrier rehearsal;
|
||||
- recovery from a failed migration;
|
||||
- scheduling onto an additional node where hardware exists.
|
||||
|
||||
The named risks are mixed physical laws, and assuming that added CPU makes storage
|
||||
portable.
|
||||
|
||||
**No rules or format converter exists** (ADR-0068): a region never changes rules,
|
||||
versions, quantum or profile. The only upgrade this foundation can perform is a
|
||||
**build change with identical physical laws**, such as a bug fix, performance
|
||||
work or a refactor. The question the rehearsal must answer is: does the candidate
|
||||
build run the same world?
|
||||
|
||||
## Decision
|
||||
**1. Clone, accelerate, compare.** Two drill-only `evo-server` flags, each refused
|
||||
without `--synthetic-inputs`:
|
||||
- `--synthetic-until TICK` stops the deterministic driver after that tick. Inputs
|
||||
are never invented, so the region commits exactly up to TICK and idles.
|
||||
- `--accelerate N` (2–1000) runs the scheduling clock N times faster than real
|
||||
time. Lease deadlines stay real.
|
||||
|
||||
`evo-world-clone compare --world A --other-world B ...` calls the new library
|
||||
function `compare_region_histories`:
|
||||
- It fully audits both histories.
|
||||
- It then requires equal head tick, frame, quantum, versions and history profile.
|
||||
- It requires identical head states once B is rebound to A's world.
|
||||
- It requires identical durable outcomes: tick, actor, outcome and organism,
|
||||
including those inherited at clone genesis.
|
||||
- Builds may differ. Exit 6 means "different world".
|
||||
|
||||
The procedure:
|
||||
1. Clone the live world's cut twice into an isolated database.
|
||||
2. Run the current build on one clone and the candidate on the other, with the same
|
||||
inputs, accelerated, to the same tick.
|
||||
3. `compare` must pass. Only then may the candidate proceed.
|
||||
|
||||
A rules change cannot pass this procedure. It stays refused until a converter ADR
|
||||
defines its own comparison: ecological distributions and approval
|
||||
(PERSISTENCE_ARCHITECTURE.md). Mixed physical laws are therefore impossible by
|
||||
construction: a barrier admits only a build that `compare` proved runs the same
|
||||
world.
|
||||
|
||||
**2. Migration failure.** A migration runs in one transaction under the migration
|
||||
lock, with a 5 s lock timeout and a 10 s statement timeout (ADR-0068). A failure
|
||||
leaves the previous schema exactly.
|
||||
- While the migration holds its locks, a publication may wait. Its outcome is then
|
||||
resolved (ADR-0063), or its lease lapses and the host re-acquires under a newer
|
||||
epoch.
|
||||
- In a cluster, the migration Job is a Sync hook at wave −1. When it fails, the
|
||||
sync stops before wave 1, so the Deployment keeps the old image and nothing new
|
||||
starts.
|
||||
- Recovery means fixing the migration and syncing again. No database restore is
|
||||
involved: a restore after new writes would itself be a rewind (ADR-0075).
|
||||
|
||||
**3. Barrier in dev/stage.** The ADR-0068 order has these steps:
|
||||
1. Migrate.
|
||||
2. Retain the new build. The provision hook does this.
|
||||
3. Deploy the new build with the old build listed in `REGION_RETAINED`. The
|
||||
Recreate strategy drains the old pod first.
|
||||
4. Run `tools/ops/upgrade-barrier.sh <env> --retire OLD`.
|
||||
|
||||
The barrier tool refuses the retirement unless:
|
||||
- the build is not the one Git deploys;
|
||||
- it is listed in `REGION_RETAINED`;
|
||||
- a Ready pod already runs the Git-pinned digest.
|
||||
|
||||
It then runs `upgrade_barrier` in a short Pod with the migrator login, using the
|
||||
`migrate` component's egress, and measures the not-ready gap while the new host
|
||||
re-acquires. A rollback to a retired image quarantines. That is intended.
|
||||
|
||||
**4. Nodes.**
|
||||
|
||||
| Label | Meaning | Rule (policy-checked) |
|
||||
|---|---|---|
|
||||
| `aevolution.io/stores=true` | Holds the node-local durable volumes. Exactly one worker (today rke2-worker-1). | Every pod that mounts `aevolution-objects` must select it. Region hosts, provisioning, store checks and backups stay there. |
|
||||
| `aevolution.io/compute=true` | Admitted for stateless AEvolution work. | Only ever *preferred*, never required, so everything still schedules on one node or with no labelled node. Alertmanager is the first user. |
|
||||
|
||||
Adding a node means labelling it `aevolution.io/compute=true`, then running
|
||||
`tools/ops/scheduling-check.sh <env>`. Only stateless pods move. A region moves to
|
||||
another node only as a data operation: a restore or a clone onto that node's
|
||||
storage (ADR-0074/0075). Rescheduling a region does not move it, because its
|
||||
objects volume is pinned to the stores node by the local-path PV's node affinity.
|
||||
The policy check rejects:
|
||||
- a pod that mounts the objects volume without the stores selector;
|
||||
- a required compute affinity;
|
||||
- rehearsal flags in a deployed workload.
|
||||
|
||||
## Evidence
|
||||
Every PostgreSQL build runs two tests.
|
||||
|
||||
`region_upgrade_tests` (real processes):
|
||||
- A source world is run past its death to tick 30 and cloned four times.
|
||||
- Build b1 and candidate build b2 each run to tick 150 at 20× acceleration: 120
|
||||
ticks, 6 s of region time, in under 4 s of wall time.
|
||||
- `compare` passes, with one outcome.
|
||||
- A clone fed a temperature-shifted input stream from the same cut differs in
|
||||
`state`.
|
||||
- A clone stopped at tick 100 differs in `ticks`.
|
||||
- Comparing a world with itself, or using the rehearsal flags without the driver,
|
||||
is a usage error.
|
||||
|
||||
`tools/ci/upgrade-check.sh`:
|
||||
- A migration that takes the migration lock, adds a column to `region_commits`,
|
||||
holds its locks for 3 s and then fails runs while b1 serves. Afterwards the
|
||||
schema is still v9, the column does not exist, b1 goes on acknowledging, and the
|
||||
real migrate is a no-op.
|
||||
- b2 is retained and started, and waits without becoming ready.
|
||||
- `upgrade_barrier(b1)` fences one lease. b1 quarantines (exit 4). b2 resumes at a
|
||||
newer epoch, at or past every b1 acknowledgement, and continues with the next
|
||||
tick.
|
||||
- A late b1 host quarantines at once.
|
||||
- The history, now written by both builds, passes the full audit.
|
||||
|
||||
In the cluster, still to be run by the maintainer:
|
||||
- `upgrade-barrier.sh` on dev, retiring a build listed in `REGION_RETAINED`;
|
||||
- `scheduling-check.sh` on dev and stage, before and after labelling a second
|
||||
worker `aevolution.io/compute=true`.
|
||||
|
||||
## Consequences
|
||||
- Upgrades that keep the physical laws are qualified and operable in dev and stage.
|
||||
Rules changes remain impossible until a converter ADR exists.
|
||||
- Mutation tests show the compare check is not vacuous: removing the state
|
||||
comparison, the stop tick or the acceleration each fails the tests.
|
||||
- Region hosts stay on the stores node. Horizontal growth applies only to
|
||||
stateless work until object storage is network-attached. The storage rework
|
||||
(backlog) may revisit this.
|
||||
|
||||
## Alternatives rejected
|
||||
- **Comparing live regions at arbitrary commits.** Commit boundaries depend on flush
|
||||
timing. A shared stop tick makes the comparison exact.
|
||||
- **Required node affinity for compute.** It would break the single-node plan and
|
||||
strand pods while no node carries the label.
|
||||
- **Moving a region by relabelling nodes.** The volume stays behind. That is
|
||||
exactly the named risk.
|
||||
- **Retiring the old build automatically on every sync.** Retirement is
|
||||
irreversible. It stays a deliberate step, guarded by the checks above.
|
||||
@@ -86,3 +86,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
|
||||
- [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending.
|
||||
- [ADR-0075: World0 acknowledgment gate, host-loss recovery and promotion](0075-world0-acknowledgment-gate.md) - accepted for M8.8 ratified dev/stage RPO/RTO, World0 launch blocked (policy-enforced) until acknowledgements survive host loss in an independent failure domain, explicit-rewind in-place restore, barrier-based fencing of a superseded database, promotion only of stage-pinned digests.
|
||||
- [ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth](0076-upgrade-and-growth.md) - accepted for M8.9 accelerated clone runs compared state-for-state before a build change, failed-migration recovery, guarded in-cluster barrier, stores/compute node labels with policy-checked placement.
|
||||
|
||||
@@ -2992,3 +2992,45 @@ Fixes (this segment):
|
||||
- any in-cluster host-loss restore or fence;
|
||||
- a second failure domain, which does not exist;
|
||||
- any World0 deployment or promotion, which is blocked by design.
|
||||
|
||||
## M8.9 upgrade rehearsal and node plan - 2026-09-30
|
||||
|
||||
- `region_upgrade_tests` (real evo-server and evo-world-clone processes) passed in
|
||||
the Release, analysis and ASan/UBSan PostgreSQL builds:
|
||||
- Current build b1 and candidate b2 ran on clones of one cut, at 20×
|
||||
acceleration, from tick 30 to tick 150: 120 ticks, 6 s of region time. Wall
|
||||
times were 306/307 ms (Release), 322/323 ms (analysis) and 299/299 ms (ASan).
|
||||
- `compare` passed: `compared tick=150 outcomes=1`.
|
||||
- A temperature-shifted input stream gave `compare differs state` (exit 6).
|
||||
- A clone stopped at tick 100 gave `compare differs ticks` (exit 6).
|
||||
- Comparing a world with itself, and `--accelerate` or `--synthetic-until`
|
||||
without the driver, were usage errors (exit 2).
|
||||
- `tools/ci/upgrade-check.sh` passed in all three builds:
|
||||
- A failing migration (migration lock, `ALTER TABLE region_commits`, locks held
|
||||
3 s, then an error) left schema v9 and no new column. b1 went on
|
||||
acknowledging, and the real migrate was a no-op.
|
||||
- b2 waited while b1 held the lease. `upgrade_barrier(b1)` fenced one lease, and
|
||||
b1 exited 4.
|
||||
- b2 resumed at `epoch=2 committed=135`, at b1's last acknowledgement (135), and
|
||||
continued with tick 136. The ASan run: 137.
|
||||
- A late b1 host exited 4 without becoming ready.
|
||||
- The history audit across both builds passed: 24 commits to tick 167.
|
||||
- Mutations, each caught:
|
||||
- `compare` without the state comparison: upgrade test line 201 (the diverged
|
||||
world passed);
|
||||
- `--synthetic-until` ignored: line 165;
|
||||
- `--accelerate` as a no-op: line 184 (over 4 s).
|
||||
- The WSL `checks` set: 59/59 in debug, release, analysis and ASan. clang-tidy
|
||||
(analysis builds) and the format check were clean.
|
||||
- Deploy policy: 67 objects, 0 failures. Negatives, each rejected: an
|
||||
objects-volume pod without the stores selector; a required compute affinity;
|
||||
`--accelerate` in the region Deployment. `upgrade-barrier.sh` and
|
||||
`scheduling-check.sh` refuse World0 before kubectl.
|
||||
- Found during validation: genesis seed 0xF1 collided with a checkpoint-store test
|
||||
(fixture provisioning failed), and was replaced by the unused 0xD7. clang-tidy
|
||||
flagged duplicated option branches, which were merged.
|
||||
- Not executed:
|
||||
- `upgrade-barrier.sh` or `scheduling-check.sh` against dev or stage;
|
||||
- scheduling onto a second labelled node;
|
||||
- the Windows MSVC build (no Windows target changed; the server and PostgreSQL
|
||||
code are Linux-only).
|
||||
|
||||
@@ -232,3 +232,20 @@ rebuilt as isolated world ...` counts as restore evidence. The restore uses fres
|
||||
roles on a throwaway cluster; nothing reaches the source database. A restored copy
|
||||
keeps the source world identity and is never served; restoring in place after host
|
||||
loss is M8.8.
|
||||
|
||||
## Upgrades and nodes (M8.9, ADR-0076)
|
||||
|
||||
**Build upgrade (same rules).**
|
||||
1. Rehearse on clones first: clone the cut twice into an isolated database, run the
|
||||
current and candidate builds with `--synthetic-inputs development-v1
|
||||
--synthetic-until T --accelerate 20`, then `evo-world-clone compare` must exit 0.
|
||||
2. Pin the new server digest in `deploy/kubernetes/region/<env>` (image and
|
||||
`REGION_BUILD`), add the old digest to `REGION_RETAINED`, and sync. The migrate
|
||||
hook (wave -1) stops the sync on failure, leaving the old host running.
|
||||
3. When the new pod is Ready: `tools/ops/upgrade-barrier.sh <env> --retire OLD`.
|
||||
Keep OLD in `REGION_RETAINED`: its history stays recoverable.
|
||||
|
||||
**Adding a node.** Label it `aevolution.io/compute=true` and run
|
||||
`tools/ops/scheduling-check.sh <env>`. Only stateless pods (Alertmanager today) may
|
||||
move there. Never move `aevolution.io/stores` to rebalance: the objects volume stays
|
||||
on its node; a region moves only by restore or clone onto the new node's storage.
|
||||
|
||||
Reference in New Issue
Block a user