Complete M7.9b upgrade barrier and durability qualification

Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 07:31:04 -04:00
co-authored by Claude Opus 5.5
parent 703f67834f
commit f59cca2c3b
24 changed files with 893 additions and 52 deletions
+8
View File
@@ -593,3 +593,11 @@ M7.9a implemented and locally qualified (2026-09-29), under ADR-0067:
- a full-chain retention audit is added.
Next: M7.9b, covering the upgrade barrier and old-writer fencing, migration
failure while serving, and measured recovery and batching latency.
M7.9b implemented and locally qualified (2026-09-29), under ADR-0068:
- build-bound authority, and an upgrade barrier that retires a build and fences
old writers;
- a failed migration leaves the previous schema exactly, while writers continue;
- rule and format changes are refused explicitly;
- measured recovery (19.6 s against a 60 s target) and batching latency.
M7 is complete locally. Next: M8.1, the host inventory and budgets.
+26
View File
@@ -928,3 +928,29 @@ configurations; the PostgreSQL suites passed in three.
Next: M7.9b, covering the upgrade barrier and old-writer fencing, schema
migration failure while a host is serving, and measured recovery and batching
latency.
## M7.9b upgrade barrier and durability qualification (2026-09-29)
ADR-0068 and schema v8 bind authority to builds:
- Acquisition names the host's build, and only a retained build that has not been
retired may hold a lease.
- No commit may name a retired build.
- `upgrade_barrier` retires a build and fences every holder in one transaction.
- Hosts built before M7.9b keep working until the first barrier.
A lease-holding writer survived a failed v8 migration (the database stayed at v7
exactly) and then the real migration. After the barrier it could never regain
authority. The new build recovered the whole history, including links written by
the retired build.
Qualification (engineering/DURABILITY_QUALIFICATION.md), under the production
lease:
- SIGKILL to successor readiness: 19.5–19.6 s against the proposed 60 s target;
- acknowledgement latency with a 1 s flush: p50 549 ms, p99 1048 ms;
- a death was acknowledged in 44 ms;
- no acknowledged input was lost.
**M7 exit criteria are met locally.** Container, CI and cluster execution have not
run. Host-loss durability is M8.
Next: M8.1, the host inventory and budgets.
+96
View File
@@ -0,0 +1,96 @@
# ADR-0068: Upgrade barrier, build-bound authority and durability qualification
## Status
Accepted for M7.9b, 2026-09-29. It completes M7.9 with ADR-0067 (clones). It
changes the ADR-0060 acquisition contract (schema v8). Rules and format
conversions remain unimplemented: no converter exists, so every change is
refused explicitly.
## Context
M7.9 requires:
- explicit rule and format rejection or migration;
- evidence for a migration failure and for fencing old writers;
- measured recovery and batching latency.
PERSISTENCE_ARCHITECTURE's upgrade path is: barrier, clone, migrate, compare,
approve, then a compatible deployment at a world barrier. That implies:
- A new build must be able to take over a region.
- Every writer of the old build must be stopped for good, including one that is
paused, idle, or restarted from an old image.
- History written by the old build must stay recoverable.
Before v8, any retained build could publish, and acquisition did not name a
build. A fenced old host could therefore re-acquire an idle region and renew it
forever.
## Decision
**Build-bound authority (schema v8).**
- `retained_builds.retired_at` marks a build as retired from publication. The
row, and so the recoverability of its history, stays.
- A `region_commits` insert trigger refuses any new commit that names a retired
build (EVL06). This covers publications, provisioned geneses and clone
geneses.
- `acquire_region_lease(world, region, duration, build)` grants authority only to
a retained, unretired build (EVL06 otherwise). The C++ API takes the build. The
host quarantines on `unsupported` (exit 4): a retired image never serves or
holds an idle lease.
- The build-less 3-argument form remains for pre-v8 hosts during a rolling
schema upgrade. It refuses once any build has been retired.
- Both forms wrap the internal `grant_region_lease`, which the runtime cannot
execute. The runtime audit now covers twelve functions.
**Barrier.** The owner-only `upgrade_barrier(build)` retires one build and clears
every current lease holder in the database, in one transaction:
- Existing holders are fenced at their next renewal or publication.
- They cannot re-acquire by either form.
- A commit already waiting on a lease row is judged after the barrier commits.
- Retiring the last current build is refused.
- Epochs are kept, so the next acquisition takes a fresh one.
Operator order:
1. Migrate the schema. Old hosts keep their leases, because renew and publish are
unchanged.
2. Retain the new build.
3. Deploy hosts with `--build new --retained old`.
4. Run the barrier for the old build.
**Explicit rejection.** A region never changes profile, versions, quantum or
history profile (EVL06 at publication; `unsupported` at recovery). A clone
inherits them unchanged. No rules or format converter is registered. Adding one
requires a new ADR, a versioned conversion that is replayed and compared on a
clone, and approval. Retirement is irreversible through this interface. Undoing
it is an audited owner break-glass action, valid only while formats and rules are
identical.
**Qualification fixture.** The fixture is:
- the production lease policy (20 s lease, 5 s renewal, 2 s margin) and a 1 s flush;
- one lineage region paced at 50 ms per tick;
- a local socket and filesystem, on WSL2.
It measures:
- acknowledgement latency;
- the time from SIGKILL to the successor's readiness, against the proposed 60 s
target (asserted for this fixture only);
- head verification, full audit and clone cost.
The results are in engineering/DURABILITY_QUALIFICATION.md.
## Consequences
- Recovery after losing a worker is dominated by the lease: a successor waits out
the dead holder's remaining lease, up to 20 s, and verification adds
milliseconds. Shortening the lease trades that time against false fencing
under database latency (ADR-0060).
- Migrations run in one transaction under an advisory lock and a 5 s lock timeout.
A publication waiting on migration locks may become ambiguous; resolution
handles it (ADR-0063). A failed migration leaves the previous schema exactly.
- Binaries built before M7.9b (3-argument acquisition) work against v8 until the
first barrier. Binaries from M7.9b onwards require schema v8.
## Alternatives rejected
- **Retirement only, checked at publication.** An idle old host could still hold
a lease forever.
- **Bumping epochs without retirement.** Old hosts would simply re-acquire.
- **Dropping the 3-argument acquisition in v8.** Every old host would be stranded
mid-upgrade before the new build was deployed.
- **In-place rules migration of a live region.** It would change physical laws
without the clone-and-compare approval that the vision requires.
+1
View File
@@ -78,3 +78,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0065: Recovery before readiness](0065-recovery-before-readiness.md) - accepted for M7.7 head tail replay verification, schema v6 commit lookup and real kill-boundary tests.
- [ADR-0066: Operations endpoints](0066-operations-endpoints.md) - accepted for M7.8 liveness vs authority readiness, bounded metrics, dependency-free responder and the server image.
- [ADR-0067: Region clones and retained-history audit](0067-region-clones.md) - accepted for M7.9a re-identified clone genesis naming its source cut, isolated clone databases and full-chain retention audit.
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
+17
View File
@@ -235,3 +235,20 @@ database:
It names the exact link whose evidence is missing, even though head recovery
still succeeds. Migration, fencing of old writers, and measured recovery and
batching latency are still M7.9b.
M7.9b (ADR-0068) closes the clone/migration row.
- **Failed migration while serving.** A v8 migration that failed at its last
statement left v7 exactly, and the lease-holding writer kept publishing. The
real migration then committed under that same lease.
- **Old writers.** The upgrade barrier retires the old build and fences every
holder. After it:
- the old writer cannot publish, renew or re-acquire, by either acquisition
form;
- no link naming the retired build can commit, even under a current lease;
- a host of the old image quarantines;
- the new build recovered the whole history and continued.
- **Rejection.** Rule, format and profile changes are refused; no converter
exists.
- **Qualification.** Real SIGKILLs under the production lease were recovered in
19.5–19.6 s against the proposed 60 s target, with no acknowledged input lost
(see engineering/DURABILITY_QUALIFICATION.md).
@@ -0,0 +1,75 @@
# M7 durability qualification (M7.9b)
Measured on 2026-09-29 by `durability_qualification` in
`tests/region_migration_tests.cpp`, under `tools/ci/postgres-check.sh`. These are
local development measurements under the fixture below. They are not a
production service level, and they are not evidence about host loss (M8).
## Fixture
- **Machine:** WSL2 Ubuntu on the maintainer's Windows 11 workstation.
- **Database:** a private PostgreSQL 18.6 cluster, connected over a Unix socket.
- **Storage:** filesystem object store on the same volume (`single_volume_development`).
- **Workload:** one `environmental-lineage-v1` region with a 37,756-byte payload,
driven by real `evo-server` processes.
- **Host settings:** production lease policy (20 s lease, 5 s renewal, 2 s safety
margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
- **Inputs:** paced at the 50 ms tick, 300 in total over three owners. The
consumer dies at tick 20.
- **Kills:** after each 100 inputs, 10 more were sent and the owner was
SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two
measured recoveries.
## Results
| Measure | Release (GCC14 -O3) | clang-tidy build | ASan/UBSan |
|---|---|---|---|
| Ack latency p50 / p99 / max (ms), n=299 | 549 / 1048 / 1048 | 549 / 1045 / 1049 | 530 / 1028 / 1028 |
| Death tick ack latency (ms) | 44 | 48 | 23 |
| SIGKILL to successor ready (ms), runs 1 and 2 | 19599 / 19548 | 19605 / 19557 | 19627 / 19561 |
| Head verification p50 / max (µs), n=10 | 1626 / 11949 | 5826 / 16184 | 14419 / 25352 |
| Full-history audit, 16 links, 300 ticks (ms) | 20 | 82 | 204 |
| Clone into the second database (ms) | 12 | 19 | 36 |
The lease-store microbenchmarks in the same run (200 samples, Release) had these
p50/p99 values:
| Operation | p50 (µs) | p99 (µs) |
|---|---|---|
| Acquire | 624 | 844 |
| Renew | 600 | 744 |
| Publish manifest | 750 | 965 |
| Release | 603 | 717 |
Batch publication, including payload and trace upload with read-back, took
6–8 ms.
## Interpretation
- **Recovery is dominated by the lease.** Kill to ready was 19.5–19.6 s in every
build. The dead holder's lease must expire in database time, and it had been
renewed less than 5 s earlier. Verification (tens of milliseconds even under
ASan) and process start are small.
- **The target holds.** The proposed 60 s recovery target holds for this fixture
with about 40 s of margin. The test asserts < 60 s for this fixture only.
- **Nothing acknowledged was lost.** Every acknowledged input was at or below the
committed head at each kill. The one death was recorded exactly once across
three owners, and each successor resumed exactly at the committed head with a
new epoch.
- **Acknowledgement latency is the batching cost.** It is set by the flush
interval: an input waits for its batch to commit, which averages about half a
flush interval, plus a commit of under 10 ms. A death seals its batch and
commits at once, so it is acknowledged fastest.
- **Verification and audit costs scale.** They grow with payload size and chain
length. At this size the full history is re-verified in about 20 ms (Release).
Larger regions and longer histories need their own measurement before any
target is claimed.
## Not measured
The following were not measured:
- host or storage loss;
- network partitions between host and database;
- a separate database host;
- concurrent regions;
- large payloads;
- long histories;
- Kubernetes rescheduling time.
These belong to M8, together with off-host backup and restore.
+49
View File
@@ -2485,3 +2485,52 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
stale-binary run was discarded and the mutation was rerun.
- Not executed: clones across separate PostgreSQL clusters or hosts, Docker, the
Gitea CI, and cluster runs.
## M7.9b upgrade barrier and durability qualification - 2026-09-29
- Every M7.9b change is in the Linux PostgreSQL build or in docs. Windows was not
rerun; its 59/59 from M7.9a still covers the portable code, which is unchanged.
- WSL Linux `checks` set (GCC14 Debug and Release, clang19 analysis, clang19
ASan/UBSan) passed CTest 59/59 each. The format check passed.
- The PostgreSQL suites passed in GCC Release, clang-tidy analysis and ASan/UBSan.
After the mutation run, one host test was corrected: `unrestorable_head_quarantines`
now retains its build, so the quarantine it checks again comes from recovery
rather than acquisition. Release and analysis were rebuilt and rerun green after
that change. The ASan build compiled with it, but its full run predates it.
- Harness: upgrade paths now end at 1..8. The populated v6 database upgrades to v8.
Version and checksum drift checks include v8.
- Migration and barrier test (database `evo_online`, created at v7 from the
checked-in files):
- A pre-v8 writer with a build-less lease published ticks 1..20 (with the
death).
- An `apply.sql` run with an invalid v8 checksum failed. It left history 1..7,
no `retired_at` column and no four-argument acquire. The writer renewed and
published ticks 21..40.
- The real migration then reached v8 while the writer held its lease, and the
writer published ticks 41..60.
- Barrier refusals: the runtime cannot call it (42501), and retiring the last
current build is refused (22023).
- After the barrier (1 lease fenced), the old writer's publication and renewal
were fenced. Build-less acquisition returned EVL06, and acquisition with b1
returned unsupported.
- A b2 lease publishing a link that names b1 was refused as unsupported.
- A b2 `evo-server` became ready at epoch 3 from committed tick 60 and
acknowledged through tick 90. Its final head is built by b2.
- The audit passes with both builds retained, and reports unsupported at
sequence 0 with only b2.
- A b1 `evo-server` quarantined (exit 4) without becoming ready, and the head
was unchanged.
- Qualification: see engineering/DURABILITY_QUALIFICATION.md. SIGKILL to ready
took 19.5–19.6 s under the production lease in all three builds (the 60 s
target is asserted). Acknowledgement latency p50 was about 549 ms and p99
about 1048 ms. No acknowledged input was lost, and one durable death was
recorded across five owners.
- Mutations (each restored and rebuilt). Each was caught:
- no retired-build trigger;
- no build check in four-argument acquisition;
- no barrier check in build-less acquisition;
- a barrier that does not fence holders;
- a host that retries an unsupported acquisition instead of quarantining.
The migration test caught this one after the host-test correction.
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
host or storage loss (M8).
+23 -1
View File
@@ -68,7 +68,8 @@ every role transitively in it, and rolls back on any violation:
`pg_write_all_data`);
- any table privilege in `evolution`, including via PUBLIC;
- CREATE on the schema, the database or `public`;
- EXECUTE outside the eleven fenced functions (schema v6).
- EXECUTE outside the twelve fenced functions (schema v8 adds the build-aware
`acquire_region_lease`).
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
@@ -145,6 +146,27 @@ receives clones. `evo-world-clone audit` verifies a region's whole retained
history; it is read-only and needs only runtime read access. Never point a clone
target at a production database or World0.
Schema v8 (ADR-0068) binds authority to builds:
- Hosts from M7.9b onwards acquire with their build identity. Only a retained
build that has not been retired may hold a lease.
- No new commit may name a retired build (EVL06). History written by a retired
build stays recoverable, so keep it retained.
- Hosts built before M7.9b use build-less acquisition, which keeps working until
the first barrier.
To upgrade the build:
1. Migrate the schema. Running hosts keep their leases.
2. Insert the new build into `retained_builds`.
3. Deploy hosts with `--build <new> --retained <old>`.
4. On an owner connection, run
`SELECT evolution.upgrade_barrier('\x<old build>');`
It retires the old build and fences every current holder in the database. It
returns the number of leases fenced, and refuses to retire the last current
build.
Old images then quarantine (exit 4) instead of serving. Rules and format changes
are refused explicitly; there is no converter.
There is no operator CLI for provisioning original regions yet, and no real world
has been provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
have no payload.