Serialize build retirement and clone classification, bound ops accepts

Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the
barrier takes a database-wide upgrade lock exclusively, while
acquisitions, lease checks, provisioning and the retired-build trigger
take it shared first, so a queued acquisition can no longer regain
authority for a just-retired build and concurrent barriers cannot
retire the last current build. Provisioning takes a per-world lock
before classifying a world as original or a clone of one source. The
operations listener makes at most four accept attempts per serve().
Race regressions prove each contender is blocked before commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent f59cca2c3b
commit 168c90d5b2
14 changed files with 616 additions and 15 deletions
+5
View File
@@ -954,3 +954,8 @@ lease:
run. Host-loss durability is M8.
Next: M8.1, the host inventory and budgets.
Review fixes (2026-09-29): schema v9 serializes build retirement with acquisition,
leases and provisioning under one lock order. It also serializes clone-world
classification per world. The operations listener's accept work per loop pass is
now bounded. Regression tests and five mutations cover each fix.
+12
View File
@@ -99,3 +99,15 @@ zlib, CA certificates):
- A later gateway must honor readiness and the lease epoch before routing.
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
(M8).
## Amendment: bounded accept work (2026-09-29 review)
`serve()` used to accept, and close excess connections, until the backlog was
empty. A replenished stream of connections could therefore hold the host loop
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
attempts, refused ones included. Pending connections wait in the 16-entry listen
backlog, and deadlines are judged after the accept work.
Tests check the per-call budget, including at the connection cap, and that an
accepted probe is still answered under replenished pressure. A real server keeps
its lease and readiness while a client opens and closes connections continuously
through 60 ticks.
+15
View File
@@ -109,3 +109,18 @@ alone does not prove retention; the audit does.
defeats isolation.
- **Recording provenance only in a table.** The chain would not commit to its
origin, and cross-database clones could not verify it.
## Amendment: serialized world classification (schema v9, 2026-09-29 review)
The rule "original, or a clone of exactly one source" was enforced by unlocked
existence checks. Two concurrent clones of different sources into different
regions of one world could therefore both commit (reproduced by review). Schema
v9 fixes this:
- Both provisioning functions take a per-world advisory lock, keyed by a hash of
the world ID, after the shared upgrade lock (ADR-0068 amendment).
- They then check the classification. Because the checks run after the lock wait,
they see every committed provisioning of that world.
- Unrelated worlds provision in parallel. A hash collision only adds waiting.
Regression tests race different-source clones, and an original region against a
clone, into one world, and expect exactly one to succeed. Same-source clones into
different regions both succeed.
+25
View File
@@ -94,3 +94,28 @@ The results are in engineering/DURABILITY_QUALIFICATION.md.
mid-upgrade before the new build was deployed.
- **In-place rules migration of a live region.** It would change physical laws
without the clone-and-compare approval that the vision requires.
## Amendment: serialized retirement (schema v9, 2026-09-29 review)
In v8, acquisition checked eligibility before locking the lease row, without
serializing against the barrier. A request queued behind a barrier could
therefore acquire for the build just retired, then keep renewing (reproduced by
review). Schema v9 fixes this with one lock order: the upgrade lock (advisory,
database-wide), then the world classification lock, then the lease row, then
head and commit rows.
- `upgrade_barrier` takes the upgrade lock exclusively before anything else. This
also serializes barriers, so two operators cannot each retire what they believe
is a non-last build.
- Both acquisition forms, `check_region_lease` (and with it publication, renewal
and release), provisioning and the retired-build trigger take the upgrade lock
shared first. Their eligibility checks therefore run after any concurrent
barrier commits.
The lock order is the same everywhere, so publication cannot deadlock against the
barrier. Regression tests hold a barrier open, prove the contender is blocked,
commit, and check that a queued acquisition of the retired build is refused (both
forms) and that the last current build cannot be retired by racing barriers.
A runtime login can still take PostgreSQL advisory locks directly, and so could
delay barriers or acquisitions. It cannot gain authority that way. An owner can
terminate such sessions. Removing that capability needs a pooler or role policy
(M8).
+46
View File
@@ -2534,3 +2534,49 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
The migration test caught this one after the host-test correction.
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
host or storage loss (M8).
## M7.6–M7.9 review fixes (schema v9) - 2026-09-29
Review `.cache/reviews/M7_6-M7_9_REVIEW.md` found three issues. All three are
fixed:
- **[P1] Build retirement could race acquisition.** Schema v9 applies one
upgrade-lock order across the barrier, acquisitions, lease checks, provisioning
and the retired-build trigger.
- **[P2] Clone-world classification could race.** Provisioning now takes a
per-world lock and checks classification after it.
- **[P2] The operations listener accepted without bound.** Each `serve()` now
makes at most 4 accept attempts.
Evidence:
- WSL Linux `checks` set (Debug, Release, analysis, ASan/UBSan): CTest 59/59
each. The format check passed.
- PostgreSQL ASan/UBSan: full suite passed. Analysis and Release were rerun green
after fixing two clang-tidy findings in the new tests. Upgrade paths now end at
1..9.
- Race tests (database `evo_race`): each holds a transaction open, confirms the
contender is blocked on a lock via `pg_stat_activity`, then commits.
- A build-less acquisition queued behind a barrier is refused (EVL06).
- A build-aware acquisition of the build being retired is refused, and the
region stays unheld.
- The next build acquires.
- Racing barriers for the last two current builds: one returns 22023 and one
build stays current.
- Different-source clones into one world, and an original region racing a
clone: the second is refused (22023).
- Same-source clones into different regions both succeed.
- The final origins are exactly `2/1<10, 4/1<10, 4/2<10`.
- Accept budget:
- With 16 queued connections, successive `serve()` calls give 4 open, then 8
open with 0 refused, then 8 open with 4 refused.
- Refusals never exceed 4 per call under replenished connections.
- The first accepted probe got 200.
- A real server acknowledged 60 ticks under continuous connect/close churn
without losing readiness.
- Mutations, each reverting one fix (restored and rebuilt). Each was caught:
- an unlocked build-aware acquire;
- an unlocked legacy acquire;
- an unlocked barrier;
- no per-world lock;
- an accept budget of 1000.
- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s.
- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files.
+7
View File
@@ -164,6 +164,13 @@ To upgrade the build:
returns the number of leases fenced, and refuses to retire the last current
build.
Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The
barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions,
lease checks, provisioning and the retired-build trigger take it shared, and
provisioning also takes a per-world lock (71420003). A barrier therefore briefly
waits for in-flight commits and blocks new ones. Keep other advisory locks away
from these two keys.
Old images then quarantine (exit 4) instead of serving. Rules and format changes
are refused explicitly; there is no converter.