Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the barrier takes a database-wide upgrade lock exclusively, while acquisitions, lease checks, provisioning and the retired-build trigger take it shared first, so a queued acquisition can no longer regain authority for a just-retired build and concurrent barriers cannot retire the last current build. Provisioning takes a per-world lock before classifying a world as original or a clone of one source. The operations listener makes at most four accept attempts per serve(). Race regressions prove each contender is blocked before commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
122 lines
6.1 KiB
Markdown
122 lines
6.1 KiB
Markdown
# ADR-0068: Upgrade barrier, build-bound authority and durability qualification
|
|
|
|
## Status
|
|
Accepted for M7.9b, 2026-09-29. It completes M7.9 with ADR-0067 (clones). It
|
|
changes the ADR-0060 acquisition contract (schema v8). Rules and format
|
|
conversions remain unimplemented: no converter exists, so every change is
|
|
refused explicitly.
|
|
|
|
## Context
|
|
M7.9 requires:
|
|
- explicit rule and format rejection or migration;
|
|
- evidence for a migration failure and for fencing old writers;
|
|
- measured recovery and batching latency.
|
|
|
|
PERSISTENCE_ARCHITECTURE's upgrade path is: barrier, clone, migrate, compare,
|
|
approve, then a compatible deployment at a world barrier. That implies:
|
|
- A new build must be able to take over a region.
|
|
- Every writer of the old build must be stopped for good, including one that is
|
|
paused, idle, or restarted from an old image.
|
|
- History written by the old build must stay recoverable.
|
|
|
|
Before v8, any retained build could publish, and acquisition did not name a
|
|
build. A fenced old host could therefore re-acquire an idle region and renew it
|
|
forever.
|
|
|
|
## Decision
|
|
**Build-bound authority (schema v8).**
|
|
- `retained_builds.retired_at` marks a build as retired from publication. The
|
|
row, and so the recoverability of its history, stays.
|
|
- A `region_commits` insert trigger refuses any new commit that names a retired
|
|
build (EVL06). This covers publications, provisioned geneses and clone
|
|
geneses.
|
|
- `acquire_region_lease(world, region, duration, build)` grants authority only to
|
|
a retained, unretired build (EVL06 otherwise). The C++ API takes the build. The
|
|
host quarantines on `unsupported` (exit 4): a retired image never serves or
|
|
holds an idle lease.
|
|
- The build-less 3-argument form remains for pre-v8 hosts during a rolling
|
|
schema upgrade. It refuses once any build has been retired.
|
|
- Both forms wrap the internal `grant_region_lease`, which the runtime cannot
|
|
execute. The runtime audit now covers twelve functions.
|
|
|
|
**Barrier.** The owner-only `upgrade_barrier(build)` retires one build and clears
|
|
every current lease holder in the database, in one transaction:
|
|
- Existing holders are fenced at their next renewal or publication.
|
|
- They cannot re-acquire by either form.
|
|
- A commit already waiting on a lease row is judged after the barrier commits.
|
|
- Retiring the last current build is refused.
|
|
- Epochs are kept, so the next acquisition takes a fresh one.
|
|
|
|
Operator order:
|
|
1. Migrate the schema. Old hosts keep their leases, because renew and publish are
|
|
unchanged.
|
|
2. Retain the new build.
|
|
3. Deploy hosts with `--build new --retained old`.
|
|
4. Run the barrier for the old build.
|
|
|
|
**Explicit rejection.** A region never changes profile, versions, quantum or
|
|
history profile (EVL06 at publication; `unsupported` at recovery). A clone
|
|
inherits them unchanged. No rules or format converter is registered. Adding one
|
|
requires a new ADR, a versioned conversion that is replayed and compared on a
|
|
clone, and approval. Retirement is irreversible through this interface. Undoing
|
|
it is an audited owner break-glass action, valid only while formats and rules are
|
|
identical.
|
|
|
|
**Qualification fixture.** The fixture is:
|
|
- the production lease policy (20 s lease, 5 s renewal, 2 s margin) and a 1 s flush;
|
|
- one lineage region paced at 50 ms per tick;
|
|
- a local socket and filesystem, on WSL2.
|
|
|
|
It measures:
|
|
- acknowledgement latency;
|
|
- the time from SIGKILL to the successor's readiness, against the proposed 60 s
|
|
target (asserted for this fixture only);
|
|
- head verification, full audit and clone cost.
|
|
|
|
The results are in engineering/DURABILITY_QUALIFICATION.md.
|
|
|
|
## Consequences
|
|
- Recovery after losing a worker is dominated by the lease: a successor waits out
|
|
the dead holder's remaining lease, up to 20 s, and verification adds
|
|
milliseconds. Shortening the lease trades that time against false fencing
|
|
under database latency (ADR-0060).
|
|
- Migrations run in one transaction under an advisory lock and a 5 s lock timeout.
|
|
A publication waiting on migration locks may become ambiguous; resolution
|
|
handles it (ADR-0063). A failed migration leaves the previous schema exactly.
|
|
- Binaries built before M7.9b (3-argument acquisition) work against v8 until the
|
|
first barrier. Binaries from M7.9b onwards require schema v8.
|
|
|
|
## Alternatives rejected
|
|
- **Retirement only, checked at publication.** An idle old host could still hold
|
|
a lease forever.
|
|
- **Bumping epochs without retirement.** Old hosts would simply re-acquire.
|
|
- **Dropping the 3-argument acquisition in v8.** Every old host would be stranded
|
|
mid-upgrade before the new build was deployed.
|
|
- **In-place rules migration of a live region.** It would change physical laws
|
|
without the clone-and-compare approval that the vision requires.
|
|
|
|
## Amendment: serialized retirement (schema v9, 2026-09-29 review)
|
|
In v8, acquisition checked eligibility before locking the lease row, without
|
|
serializing against the barrier. A request queued behind a barrier could
|
|
therefore acquire for the build just retired, then keep renewing (reproduced by
|
|
review). Schema v9 fixes this with one lock order: the upgrade lock (advisory,
|
|
database-wide), then the world classification lock, then the lease row, then
|
|
head and commit rows.
|
|
- `upgrade_barrier` takes the upgrade lock exclusively before anything else. This
|
|
also serializes barriers, so two operators cannot each retire what they believe
|
|
is a non-last build.
|
|
- Both acquisition forms, `check_region_lease` (and with it publication, renewal
|
|
and release), provisioning and the retired-build trigger take the upgrade lock
|
|
shared first. Their eligibility checks therefore run after any concurrent
|
|
barrier commits.
|
|
|
|
The lock order is the same everywhere, so publication cannot deadlock against the
|
|
barrier. Regression tests hold a barrier open, prove the contender is blocked,
|
|
commit, and check that a queued acquisition of the retired build is refused (both
|
|
forms) and that the last current build cannot be retired by racing barriers.
|
|
|
|
A runtime login can still take PostgreSQL advisory locks directly, and so could
|
|
delay barriers or acquisitions. It cannot gain authority that way. An owner can
|
|
terminate such sessions. Removing that capability needs a pooler or role policy
|
|
(M8).
|