Complete M7.9b upgrade barrier and durability qualification

Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 07:31:04 -04:00
co-authored by Claude Opus 5.5
parent 703f67834f
commit f59cca2c3b
24 changed files with 893 additions and 52 deletions
@@ -0,0 +1,75 @@
# M7 durability qualification (M7.9b)
Measured on 2026-09-29 by `durability_qualification` in
`tests/region_migration_tests.cpp`, under `tools/ci/postgres-check.sh`. These are
local development measurements under the fixture below. They are not a
production service level, and they are not evidence about host loss (M8).
## Fixture
- **Machine:** WSL2 Ubuntu on the maintainer's Windows 11 workstation.
- **Database:** a private PostgreSQL 18.6 cluster, connected over a Unix socket.
- **Storage:** filesystem object store on the same volume (`single_volume_development`).
- **Workload:** one `environmental-lineage-v1` region with a 37,756-byte payload,
driven by real `evo-server` processes.
- **Host settings:** production lease policy (20 s lease, 5 s renewal, 2 s safety
margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
- **Inputs:** paced at the 50 ms tick, 300 in total over three owners. The
consumer dies at tick 20.
- **Kills:** after each 100 inputs, 10 more were sent and the owner was
SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two
measured recoveries.
## Results
| Measure | Release (GCC14 -O3) | clang-tidy build | ASan/UBSan |
|---|---|---|---|
| Ack latency p50 / p99 / max (ms), n=299 | 549 / 1048 / 1048 | 549 / 1045 / 1049 | 530 / 1028 / 1028 |
| Death tick ack latency (ms) | 44 | 48 | 23 |
| SIGKILL to successor ready (ms), runs 1 and 2 | 19599 / 19548 | 19605 / 19557 | 19627 / 19561 |
| Head verification p50 / max (µs), n=10 | 1626 / 11949 | 5826 / 16184 | 14419 / 25352 |
| Full-history audit, 16 links, 300 ticks (ms) | 20 | 82 | 204 |
| Clone into the second database (ms) | 12 | 19 | 36 |
The lease-store microbenchmarks in the same run (200 samples, Release) had these
p50/p99 values:
| Operation | p50 (µs) | p99 (µs) |
|---|---|---|
| Acquire | 624 | 844 |
| Renew | 600 | 744 |
| Publish manifest | 750 | 965 |
| Release | 603 | 717 |
Batch publication, including payload and trace upload with read-back, took
6–8 ms.
## Interpretation
- **Recovery is dominated by the lease.** Kill to ready was 19.5–19.6 s in every
build. The dead holder's lease must expire in database time, and it had been
renewed less than 5 s earlier. Verification (tens of milliseconds even under
ASan) and process start are small.
- **The target holds.** The proposed 60 s recovery target holds for this fixture
with about 40 s of margin. The test asserts < 60 s for this fixture only.
- **Nothing acknowledged was lost.** Every acknowledged input was at or below the
committed head at each kill. The one death was recorded exactly once across
three owners, and each successor resumed exactly at the committed head with a
new epoch.
- **Acknowledgement latency is the batching cost.** It is set by the flush
interval: an input waits for its batch to commit, which averages about half a
flush interval, plus a commit of under 10 ms. A death seals its batch and
commits at once, so it is acknowledged fastest.
- **Verification and audit costs scale.** They grow with payload size and chain
length. At this size the full history is re-verified in about 20 ms (Release).
Larger regions and longer histories need their own measurement before any
target is claimed.
## Not measured
The following were not measured:
- host or storage loss;
- network partitions between host and database;
- a separate database host;
- concurrent regions;
- large payloads;
- long histories;
- Kubernetes rescheduling time.
These belong to M8, together with off-host backup and restore.
+49
View File
@@ -2485,3 +2485,52 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
stale-binary run was discarded and the mutation was rerun.
- Not executed: clones across separate PostgreSQL clusters or hosts, Docker, the
Gitea CI, and cluster runs.
## M7.9b upgrade barrier and durability qualification - 2026-09-29
- Every M7.9b change is in the Linux PostgreSQL build or in docs. Windows was not
rerun; its 59/59 from M7.9a still covers the portable code, which is unchanged.
- WSL Linux `checks` set (GCC14 Debug and Release, clang19 analysis, clang19
ASan/UBSan) passed CTest 59/59 each. The format check passed.
- The PostgreSQL suites passed in GCC Release, clang-tidy analysis and ASan/UBSan.
After the mutation run, one host test was corrected: `unrestorable_head_quarantines`
now retains its build, so the quarantine it checks again comes from recovery
rather than acquisition. Release and analysis were rebuilt and rerun green after
that change. The ASan build compiled with it, but its full run predates it.
- Harness: upgrade paths now end at 1..8. The populated v6 database upgrades to v8.
Version and checksum drift checks include v8.
- Migration and barrier test (database `evo_online`, created at v7 from the
checked-in files):
- A pre-v8 writer with a build-less lease published ticks 1..20 (with the
death).
- An `apply.sql` run with an invalid v8 checksum failed. It left history 1..7,
no `retired_at` column and no four-argument acquire. The writer renewed and
published ticks 21..40.
- The real migration then reached v8 while the writer held its lease, and the
writer published ticks 41..60.
- Barrier refusals: the runtime cannot call it (42501), and retiring the last
current build is refused (22023).
- After the barrier (1 lease fenced), the old writer's publication and renewal
were fenced. Build-less acquisition returned EVL06, and acquisition with b1
returned unsupported.
- A b2 lease publishing a link that names b1 was refused as unsupported.
- A b2 `evo-server` became ready at epoch 3 from committed tick 60 and
acknowledged through tick 90. Its final head is built by b2.
- The audit passes with both builds retained, and reports unsupported at
sequence 0 with only b2.
- A b1 `evo-server` quarantined (exit 4) without becoming ready, and the head
was unchanged.
- Qualification: see engineering/DURABILITY_QUALIFICATION.md. SIGKILL to ready
took 19.5–19.6 s under the production lease in all three builds (the 60 s
target is asserted). Acknowledgement latency p50 was about 549 ms and p99
about 1048 ms. No acknowledged input was lost, and one durable death was
recorded across five owners.
- Mutations (each restored and rebuilt). Each was caught:
- no retired-build trigger;
- no build check in four-argument acquisition;
- no barrier check in build-less acquisition;
- a barrier that does not fence holders;
- a host that retries an unsupported acquisition instead of quarantining.
The migration test caught this one after the host-test correction.
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
host or storage loss (M8).