Files
aevolution/docs/adr/0076-upgrade-and-growth.md
2026-10-08 12:08:53 -04:00

8.1 KiB
Raw Permalink Blame History

[Reviewed]

ADR-0076: Upgrade rehearsal on clones, version barrier in dev/stage, and node growth

Status

Accepted for M8.9, 2026-09-30. It builds on:

  • ADR-0067 (clones);
  • ADR-0068 (upgrade barrier);
  • ADR-0073 (synthetic driver);
  • ADR-0074/0075 (backup, restore, World0 gate).

Context

M8.9 requires:

  • clone, migrate, accelerate and compare before any world-wide rules change;
  • a node label and affinity plan that works on one node and admits added compute nodes.

The evidence asked for is:

  • a dev/stage version-barrier rehearsal;
  • recovery from a failed migration;
  • scheduling onto an additional node where hardware exists.

The named risks are mixed physical laws, and assuming that added CPU makes storage portable.

No rules or format converter exists (ADR-0068): a region never changes rules, versions, quantum or profile. The only upgrade this foundation can perform is a build change with identical physical laws, such as a bug fix, performance work or a refactor. The question the rehearsal must answer is: does the candidate build run the same world?

Decision

1. Clone, accelerate, compare. Two drill-only evo-server flags, each refused without --synthetic-inputs:

  • --synthetic-until TICK stops the deterministic driver after that tick. Inputs are never invented, so the region commits exactly up to TICK and idles.
  • --accelerate N (2–1000) runs the scheduling clock N times faster than real time. Lease deadlines stay real.

evo-world-clone compare --world A --other-world B ... calls the new library function compare_region_histories:

  • It fully audits both histories.
  • It then requires equal head tick, frame, quantum, versions and history profile.
  • It requires identical head states once B is rebound to A's world.
  • It requires identical durable outcomes: tick, actor, outcome and organism, including those inherited at clone genesis.
  • Builds may differ. Exit 6 means "different world".

The procedure:

  1. Clone the live world's cut twice into an isolated database.
  2. Run the current build on one clone and the candidate on the other, with the same inputs, accelerated, to the same tick.
  3. compare must pass. Only then may the candidate proceed.

A rules change cannot pass this procedure. It stays refused until a converter ADR defines its own comparison: ecological distributions and approval (PERSISTENCE_ARCHITECTURE.md). Mixed physical laws are therefore impossible by construction: a barrier admits only a build that compare proved runs the same world.

2. Migration failure. A migration runs in one transaction under the migration lock, with a 5 s lock timeout and a 10 s statement timeout (ADR-0068). A failure leaves the previous schema exactly.

  • While the migration holds its locks, a publication may wait. Its outcome is then resolved (ADR-0063), or its lease lapses and the host re-acquires under a newer epoch.
  • In a cluster, the migration Job is a Sync hook at wave −1. When it fails, the sync stops before wave 1, so the Deployment keeps the old image and nothing new starts.
  • Recovery means fixing the migration and syncing again. No database restore is involved: a restore after new writes would itself be a rewind (ADR-0075).

3. Barrier in dev/stage. The ADR-0068 order has these steps:

  1. Migrate.
  2. Retain the new build. The provision hook does this.
  3. Deploy the new build with the old build listed in REGION_RETAINED. The Recreate strategy drains the old pod first.
  4. Run tools/ops/upgrade-barrier.sh <env> --retire OLD.

The barrier tool refuses the retirement unless:

  • the build is not the one Git deploys;
  • it is listed in REGION_RETAINED;
  • a Ready pod already runs the Git-pinned digest.

It then runs upgrade_barrier in a short Pod with the migrator login, using the migrate component's egress, and measures the not-ready gap while the new host re-acquires. A rollback to a retired image quarantines. That is intended.

4. Nodes.

Label Meaning Rule (policy-checked)
aevolution.io/stores=true Holds the node-local durable volumes. Exactly one worker (today rke2-worker-1). Every pod that mounts aevolution-objects must select it. Region hosts, provisioning, store checks and backups stay there.
aevolution.io/compute=true Admitted for stateless AEvolution work. Only ever preferred, never required, so everything still schedules on one node or with no labelled node. Alertmanager is the first user.

Adding a node means labelling it aevolution.io/compute=true, then running tools/ops/scheduling-check.sh <env>. Only stateless pods move. A region moves to another node only as a data operation: a restore or a clone onto that node's storage (ADR-0074/0075). Rescheduling a region does not move it, because its objects volume is pinned to the stores node by the local-path PV's node affinity. The policy check rejects:

  • a pod that mounts the objects volume without the stores selector;
  • a required compute affinity;
  • rehearsal flags in a deployed workload.

Evidence

Every PostgreSQL build runs two tests.

region_upgrade_tests (real processes):

  • A source world is run past its death to tick 30 and cloned four times.
  • Build b1 and candidate build b2 each run to tick 150 at 20× acceleration: 120 ticks, 6 s of region time, in under 4 s of wall time.
  • compare passes, with one outcome.
  • A clone fed a temperature-shifted input stream from the same cut differs in state.
  • A clone stopped at tick 100 differs in ticks.
  • Comparing a world with itself, or using the rehearsal flags without the driver, is a usage error.

tools/ci/upgrade-check.sh:

  • A migration that takes the migration lock, adds a column to region_commits, holds its locks for 3 s and then fails runs while b1 serves. Afterwards the schema is still v9, the column does not exist, b1 goes on acknowledging, and the real migrate is a no-op.
  • b2 is retained and started, and waits without becoming ready.
  • upgrade_barrier(b1) fences one lease. b1 quarantines (exit 4). b2 resumes at a newer epoch, at or past every b1 acknowledgement, and continues with the next tick.
  • A late b1 host quarantines at once.
  • The history, now written by both builds, passes the full audit.

In the cluster, still to be run by the maintainer:

  • upgrade-barrier.sh on dev, retiring a build listed in REGION_RETAINED;
  • scheduling-check.sh on dev and stage, before and after labelling a second worker aevolution.io/compute=true.

Consequences

  • Upgrades that keep the physical laws are qualified and operable in dev and stage. Rules changes remain impossible until a converter ADR exists.
  • Mutation tests show the compare check is not vacuous: removing the state comparison, the stop tick or the acceleration each fails the tests.
  • Region hosts stay on the stores node. Horizontal growth applies only to stateless work until object storage is network-attached. The storage rework (backlog) may revisit this.

Alternatives rejected

  • Comparing live regions at arbitrary commits. Commit boundaries depend on flush timing. A shared stop tick makes the comparison exact.
  • Required node affinity for compute. It would break the single-node plan and strand pods while no node carries the label.
  • Moving a region by relabelling nodes. The volume stays behind. That is exactly the named risk.
  • Retiring the old build automatically on every sync. Retirement is irreversible. It stays a deliberate step, guarded by the checks above.

Amendment: barrier tool fixes (M8 review, 2026-09-30)

Two defects in upgrade-barrier.sh would have blocked the in-cluster barrier:

  • Its Ready-pod template printed the list root instead of the pod name. Selection now uses a plain jsonpath (tools/ops/lib/pods.sh) and requires exactly one Ready pod that is not being deleted. Fixtures cover one pod, not Ready, being deleted, no condition and two Ready pods.
  • Its pod lacked fsGroup, so it could not read the 0440 Secret files. The pod now follows the same contract as the other database-tool pods.

Both script-generated pods (backup and barrier) render with --print-manifest, and the deploy policy check inspects them.