8.1 KiB
[Reviewed]
ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth
Status
Accepted for M8.9, 2026-09-30. It builds on:
- ADR-0067 (clones);
- ADR-0068 (upgrade barrier);
- ADR-0073 (synthetic driver);
- ADR-0074/0075 (backup, restore, World0 gate).
Context
M8.9 requires:
- clone, migrate, accelerate and compare before any world-wide rules change;
- a node label and affinity plan that works on one node and admits added compute nodes.
The evidence asked for is:
- a dev/stage version-barrier rehearsal;
- recovery from a failed migration;
- scheduling onto an additional node where hardware exists.
The named risks are mixed physical laws, and assuming that added CPU makes storage portable.
No rules or format converter exists (ADR-0068): a region never changes rules, versions, quantum or profile. The only upgrade this foundation can perform is a build change with identical physical laws, such as a bug fix, performance work or a refactor. The question the rehearsal must answer is: does the candidate build run the same world?
Decision
1. Clone, accelerate, compare. Two drill-only evo-server flags, each refused
without --synthetic-inputs:
--synthetic-until TICKstops the deterministic driver after that tick. Inputs are never invented, so the region commits exactly up to TICK and idles.--accelerate N(2–1000) runs the scheduling clock N times faster than real time. Lease deadlines stay real.
evo-world-clone compare --world A --other-world B ... calls the new library
function compare_region_histories:
- It fully audits both histories.
- It then requires equal head tick, frame, quantum, versions and history profile.
- It requires identical head states once B is rebound to A's world.
- It requires identical durable outcomes: tick, actor, outcome and organism, including those inherited at clone genesis.
- Builds may differ. Exit 6 means "different world".
The procedure:
- Clone the live world's cut twice into an isolated database.
- Run the current build on one clone and the candidate on the other, with the same inputs, accelerated, to the same tick.
comparemust pass. Only then may the candidate proceed.
A rules change cannot pass this procedure. It stays refused until a converter ADR
defines its own comparison: ecological distributions and approval
(PERSISTENCE_ARCHITECTURE.md). Mixed physical laws are therefore impossible by
construction: a barrier admits only a build that compare proved runs the same
world.
2. Migration failure. A migration runs in one transaction under the migration lock, with a 5 s lock timeout and a 10 s statement timeout (ADR-0068). A failure leaves the previous schema exactly.
- While the migration holds its locks, a publication may wait. Its outcome is then resolved (ADR-0063), or its lease lapses and the host re-acquires under a newer epoch.
- In a cluster, the migration Job is a Sync hook at wave −1. When it fails, the sync stops before wave 1, so the Deployment keeps the old image and nothing new starts.
- Recovery means fixing the migration and syncing again. No database restore is involved: a restore after new writes would itself be a rewind (ADR-0075).
3. Barrier in dev/stage. The ADR-0068 order has these steps:
- Migrate.
- Retain the new build. The provision hook does this.
- Deploy the new build with the old build listed in
REGION_RETAINED. The Recreate strategy drains the old pod first. - Run
tools/ops/upgrade-barrier.sh <env> --retire OLD.
The barrier tool refuses the retirement unless:
- the build is not the one Git deploys;
- it is listed in
REGION_RETAINED; - a Ready pod already runs the Git-pinned digest.
It then runs upgrade_barrier in a short Pod with the migrator login, using the
migrate component's egress, and measures the not-ready gap while the new host
re-acquires. A rollback to a retired image quarantines. That is intended.
4. Nodes.
| Label | Meaning | Rule (policy-checked) |
|---|---|---|
aevolution.io/stores=true |
Holds the node-local durable volumes. Exactly one worker (today rke2-worker-1). | Every pod that mounts aevolution-objects must select it. Region hosts, provisioning, store checks and backups stay there. |
aevolution.io/compute=true |
Admitted for stateless AEvolution work. | Only ever preferred, never required, so everything still schedules on one node or with no labelled node. Alertmanager is the first user. |
Adding a node means labelling it aevolution.io/compute=true, then running
tools/ops/scheduling-check.sh <env>. Only stateless pods move. A region moves to
another node only as a data operation: a restore or a clone onto that node's
storage (ADR-0074/0075). Rescheduling a region does not move it, because its
objects volume is pinned to the stores node by the local-path PV's node affinity.
The policy check rejects:
- a pod that mounts the objects volume without the stores selector;
- a required compute affinity;
- rehearsal flags in a deployed workload.
Evidence
Every PostgreSQL build runs two tests.
region_upgrade_tests (real processes):
- A source world is run past its death to tick 30 and cloned four times.
- Build b1 and candidate build b2 each run to tick 150 at 20× acceleration: 120 ticks, 6 s of region time, in under 4 s of wall time.
comparepasses, with one outcome.- A clone fed a temperature-shifted input stream from the same cut differs in
state. - A clone stopped at tick 100 differs in
ticks. - Comparing a world with itself, or using the rehearsal flags without the driver, is a usage error.
tools/ci/upgrade-check.sh:
- A migration that takes the migration lock, adds a column to
region_commits, holds its locks for 3 s and then fails runs while b1 serves. Afterwards the schema is still v9, the column does not exist, b1 goes on acknowledging, and the real migrate is a no-op. - b2 is retained and started, and waits without becoming ready.
upgrade_barrier(b1)fences one lease. b1 quarantines (exit 4). b2 resumes at a newer epoch, at or past every b1 acknowledgement, and continues with the next tick.- A late b1 host quarantines at once.
- The history, now written by both builds, passes the full audit.
In the cluster, still to be run by the maintainer:
upgrade-barrier.shon dev, retiring a build listed inREGION_RETAINED;scheduling-check.shon dev and stage, before and after labelling a second workeraevolution.io/compute=true.
Consequences
- Upgrades that keep the physical laws are qualified and operable in dev and stage. Rules changes remain impossible until a converter ADR exists.
- Mutation tests show the compare check is not vacuous: removing the state comparison, the stop tick or the acceleration each fails the tests.
- Region hosts stay on the stores node. Horizontal growth applies only to stateless work until object storage is network-attached. The storage rework (backlog) may revisit this.
Alternatives rejected
- Comparing live regions at arbitrary commits. Commit boundaries depend on flush timing. A shared stop tick makes the comparison exact.
- Required node affinity for compute. It would break the single-node plan and strand pods while no node carries the label.
- Moving a region by relabelling nodes. The volume stays behind. That is exactly the named risk.
- Retiring the old build automatically on every sync. Retirement is irreversible. It stays a deliberate step, guarded by the checks above.
Amendment: barrier tool fixes (M8 review, 2026-09-30)
Two defects in upgrade-barrier.sh would have blocked the in-cluster barrier:
- Its Ready-pod template printed the list root instead of the pod name. Selection
now uses a plain jsonpath (
tools/ops/lib/pods.sh) and requires exactly one Ready pod that is not being deleted. Fixtures cover one pod, not Ready, being deleted, no condition and two Ready pods. - Its pod lacked
fsGroup, so it could not read the 0440 Secret files. The pod now follows the same contract as the other database-tool pods.
Both script-generated pods (backup and barrier) render with --print-manifest,
and the deploy policy check inspects them.