Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the barrier takes a database-wide upgrade lock exclusively, while acquisitions, lease checks, provisioning and the retired-build trigger take it shared first, so a queued acquisition can no longer regain authority for a just-retired build and concurrent barriers cannot retire the last current build. Provisioning takes a per-world lock before classifying a world as original or a clone of one source. The operations listener makes at most four accept attempts per serve(). Race regressions prove each contender is blocked before commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.1 KiB
ADR-0068: Upgrade barrier, build-bound authority and durability qualification
Status
Accepted for M7.9b, 2026-09-29. It completes M7.9 with ADR-0067 (clones). It changes the ADR-0060 acquisition contract (schema v8). Rules and format conversions remain unimplemented: no converter exists, so every change is refused explicitly.
Context
M7.9 requires:
- explicit rule and format rejection or migration;
- evidence for a migration failure and for fencing old writers;
- measured recovery and batching latency.
PERSISTENCE_ARCHITECTURE's upgrade path is: barrier, clone, migrate, compare, approve, then a compatible deployment at a world barrier. That implies:
- A new build must be able to take over a region.
- Every writer of the old build must be stopped for good, including one that is paused, idle, or restarted from an old image.
- History written by the old build must stay recoverable.
Before v8, any retained build could publish, and acquisition did not name a build. A fenced old host could therefore re-acquire an idle region and renew it forever.
Decision
Build-bound authority (schema v8).
retained_builds.retired_atmarks a build as retired from publication. The row, and so the recoverability of its history, stays.- A
region_commitsinsert trigger refuses any new commit that names a retired build (EVL06). This covers publications, provisioned geneses and clone geneses. acquire_region_lease(world, region, duration, build)grants authority only to a retained, unretired build (EVL06 otherwise). The C++ API takes the build. The host quarantines onunsupported(exit 4): a retired image never serves or holds an idle lease.- The build-less 3-argument form remains for pre-v8 hosts during a rolling schema upgrade. It refuses once any build has been retired.
- Both forms wrap the internal
grant_region_lease, which the runtime cannot execute. The runtime audit now covers twelve functions.
Barrier. The owner-only upgrade_barrier(build) retires one build and clears
every current lease holder in the database, in one transaction:
- Existing holders are fenced at their next renewal or publication.
- They cannot re-acquire by either form.
- A commit already waiting on a lease row is judged after the barrier commits.
- Retiring the last current build is refused.
- Epochs are kept, so the next acquisition takes a fresh one.
Operator order:
- Migrate the schema. Old hosts keep their leases, because renew and publish are unchanged.
- Retain the new build.
- Deploy hosts with
--build new --retained old. - Run the barrier for the old build.
Explicit rejection. A region never changes profile, versions, quantum or
history profile (EVL06 at publication; unsupported at recovery). A clone
inherits them unchanged. No rules or format converter is registered. Adding one
requires a new ADR, a versioned conversion that is replayed and compared on a
clone, and approval. Retirement is irreversible through this interface. Undoing
it is an audited owner break-glass action, valid only while formats and rules are
identical.
Qualification fixture. The fixture is:
- the production lease policy (20 s lease, 5 s renewal, 2 s margin) and a 1 s flush;
- one lineage region paced at 50 ms per tick;
- a local socket and filesystem, on WSL2.
It measures:
- acknowledgement latency;
- the time from SIGKILL to the successor's readiness, against the proposed 60 s target (asserted for this fixture only);
- head verification, full audit and clone cost.
The results are in engineering/DURABILITY_QUALIFICATION.md.
Consequences
- Recovery after losing a worker is dominated by the lease: a successor waits out the dead holder's remaining lease, up to 20 s, and verification adds milliseconds. Shortening the lease trades that time against false fencing under database latency (ADR-0060).
- Migrations run in one transaction under an advisory lock and a 5 s lock timeout. A publication waiting on migration locks may become ambiguous; resolution handles it (ADR-0063). A failed migration leaves the previous schema exactly.
- Binaries built before M7.9b (3-argument acquisition) work against v8 until the first barrier. Binaries from M7.9b onwards require schema v8.
Alternatives rejected
- Retirement only, checked at publication. An idle old host could still hold a lease forever.
- Bumping epochs without retirement. Old hosts would simply re-acquire.
- Dropping the 3-argument acquisition in v8. Every old host would be stranded mid-upgrade before the new build was deployed.
- In-place rules migration of a live region. It would change physical laws without the clone-and-compare approval that the vision requires.
Amendment: serialized retirement (schema v9, 2026-09-29 review)
In v8, acquisition checked eligibility before locking the lease row, without serializing against the barrier. A request queued behind a barrier could therefore acquire for the build just retired, then keep renewing (reproduced by review). Schema v9 fixes this with one lock order: the upgrade lock (advisory, database-wide), then the world classification lock, then the lease row, then head and commit rows.
upgrade_barriertakes the upgrade lock exclusively before anything else. This also serializes barriers, so two operators cannot each retire what they believe is a non-last build.- Both acquisition forms,
check_region_lease(and with it publication, renewal and release), provisioning and the retired-build trigger take the upgrade lock shared first. Their eligibility checks therefore run after any concurrent barrier commits.
The lock order is the same everywhere, so publication cannot deadlock against the barrier. Regression tests hold a barrier open, prove the contender is blocked, commit, and check that a queued acquisition of the retired build is refused (both forms) and that the last current build cannot be retired by racing barriers.
A runtime login can still take PostgreSQL advisory locks directly, and so could delay barriers or acquisitions. It cannot gain authority that way. An owner can terminate such sessions. Removing that capability needs a pooler or role policy (M8).