Complete M7.9b upgrade barrier and durability qualification
Schema v8 binds authority to builds (ADR-0068): acquisition names the host build and only retained, unretired builds may hold a lease; no commit may name a retired build; upgrade_barrier retires a build and fences every holder in one transaction, while build-less pre-v8 hosts keep working until the first barrier. Tests cover a writer surviving a failed and then a real online migration, old-writer fencing on every path, whole-history recovery by the new build, and the M7 qualification fixture (19.6 s SIGKILL-to-ready under the production lease, no lost acknowledgements). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# M7 durability qualification (M7.9b)
|
||||
|
||||
Measured on 2026-09-29 by `durability_qualification` in
|
||||
`tests/region_migration_tests.cpp`, under `tools/ci/postgres-check.sh`. These are
|
||||
local development measurements under the fixture below. They are not a
|
||||
production service level, and they are not evidence about host loss (M8).
|
||||
|
||||
## Fixture
|
||||
- **Machine:** WSL2 Ubuntu on the maintainer's Windows 11 workstation.
|
||||
- **Database:** a private PostgreSQL 18.6 cluster, connected over a Unix socket.
|
||||
- **Storage:** filesystem object store on the same volume (`single_volume_development`).
|
||||
- **Workload:** one `environmental-lineage-v1` region with a 37,756-byte payload,
|
||||
driven by real `evo-server` processes.
|
||||
- **Host settings:** production lease policy (20 s lease, 5 s renewal, 2 s safety
|
||||
margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
|
||||
- **Inputs:** paced at the 50 ms tick, 300 in total over three owners. The
|
||||
consumer dies at tick 20.
|
||||
- **Kills:** after each 100 inputs, 10 more were sent and the owner was
|
||||
SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two
|
||||
measured recoveries.
|
||||
|
||||
## Results
|
||||
|
||||
| Measure | Release (GCC14 -O3) | clang-tidy build | ASan/UBSan |
|
||||
|---|---|---|---|
|
||||
| Ack latency p50 / p99 / max (ms), n=299 | 549 / 1048 / 1048 | 549 / 1045 / 1049 | 530 / 1028 / 1028 |
|
||||
| Death tick ack latency (ms) | 44 | 48 | 23 |
|
||||
| SIGKILL to successor ready (ms), runs 1 and 2 | 19599 / 19548 | 19605 / 19557 | 19627 / 19561 |
|
||||
| Head verification p50 / max (µs), n=10 | 1626 / 11949 | 5826 / 16184 | 14419 / 25352 |
|
||||
| Full-history audit, 16 links, 300 ticks (ms) | 20 | 82 | 204 |
|
||||
| Clone into the second database (ms) | 12 | 19 | 36 |
|
||||
|
||||
The lease-store microbenchmarks in the same run (200 samples, Release) had these
|
||||
p50/p99 values:
|
||||
|
||||
| Operation | p50 (µs) | p99 (µs) |
|
||||
|---|---|---|
|
||||
| Acquire | 624 | 844 |
|
||||
| Renew | 600 | 744 |
|
||||
| Publish manifest | 750 | 965 |
|
||||
| Release | 603 | 717 |
|
||||
|
||||
Batch publication, including payload and trace upload with read-back, took
|
||||
6–8 ms.
|
||||
|
||||
## Interpretation
|
||||
- **Recovery is dominated by the lease.** Kill to ready was 19.5–19.6 s in every
|
||||
build. The dead holder's lease must expire in database time, and it had been
|
||||
renewed less than 5 s earlier. Verification (tens of milliseconds even under
|
||||
ASan) and process start are small.
|
||||
- **The target holds.** The proposed 60 s recovery target holds for this fixture
|
||||
with about 40 s of margin. The test asserts < 60 s for this fixture only.
|
||||
- **Nothing acknowledged was lost.** Every acknowledged input was at or below the
|
||||
committed head at each kill. The one death was recorded exactly once across
|
||||
three owners, and each successor resumed exactly at the committed head with a
|
||||
new epoch.
|
||||
- **Acknowledgement latency is the batching cost.** It is set by the flush
|
||||
interval: an input waits for its batch to commit, which averages about half a
|
||||
flush interval, plus a commit of under 10 ms. A death seals its batch and
|
||||
commits at once, so it is acknowledged fastest.
|
||||
- **Verification and audit costs scale.** They grow with payload size and chain
|
||||
length. At this size the full history is re-verified in about 20 ms (Release).
|
||||
Larger regions and longer histories need their own measurement before any
|
||||
target is claimed.
|
||||
|
||||
## Not measured
|
||||
The following were not measured:
|
||||
- host or storage loss;
|
||||
- network partitions between host and database;
|
||||
- a separate database host;
|
||||
- concurrent regions;
|
||||
- large payloads;
|
||||
- long histories;
|
||||
- Kubernetes rescheduling time.
|
||||
These belong to M8, together with off-host backup and restore.
|
||||
@@ -2485,3 +2485,52 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
stale-binary run was discarded and the mutation was rerun.
|
||||
- Not executed: clones across separate PostgreSQL clusters or hosts, Docker, the
|
||||
Gitea CI, and cluster runs.
|
||||
|
||||
## M7.9b upgrade barrier and durability qualification - 2026-09-29
|
||||
|
||||
- Every M7.9b change is in the Linux PostgreSQL build or in docs. Windows was not
|
||||
rerun; its 59/59 from M7.9a still covers the portable code, which is unchanged.
|
||||
- WSL Linux `checks` set (GCC14 Debug and Release, clang19 analysis, clang19
|
||||
ASan/UBSan) passed CTest 59/59 each. The format check passed.
|
||||
- The PostgreSQL suites passed in GCC Release, clang-tidy analysis and ASan/UBSan.
|
||||
After the mutation run, one host test was corrected: `unrestorable_head_quarantines`
|
||||
now retains its build, so the quarantine it checks again comes from recovery
|
||||
rather than acquisition. Release and analysis were rebuilt and rerun green after
|
||||
that change. The ASan build compiled with it, but its full run predates it.
|
||||
- Harness: upgrade paths now end at 1..8. The populated v6 database upgrades to v8.
|
||||
Version and checksum drift checks include v8.
|
||||
- Migration and barrier test (database `evo_online`, created at v7 from the
|
||||
checked-in files):
|
||||
- A pre-v8 writer with a build-less lease published ticks 1..20 (with the
|
||||
death).
|
||||
- An `apply.sql` run with an invalid v8 checksum failed. It left history 1..7,
|
||||
no `retired_at` column and no four-argument acquire. The writer renewed and
|
||||
published ticks 21..40.
|
||||
- The real migration then reached v8 while the writer held its lease, and the
|
||||
writer published ticks 41..60.
|
||||
- Barrier refusals: the runtime cannot call it (42501), and retiring the last
|
||||
current build is refused (22023).
|
||||
- After the barrier (1 lease fenced), the old writer's publication and renewal
|
||||
were fenced. Build-less acquisition returned EVL06, and acquisition with b1
|
||||
returned unsupported.
|
||||
- A b2 lease publishing a link that names b1 was refused as unsupported.
|
||||
- A b2 `evo-server` became ready at epoch 3 from committed tick 60 and
|
||||
acknowledged through tick 90. Its final head is built by b2.
|
||||
- The audit passes with both builds retained, and reports unsupported at
|
||||
sequence 0 with only b2.
|
||||
- A b1 `evo-server` quarantined (exit 4) without becoming ready, and the head
|
||||
was unchanged.
|
||||
- Qualification: see engineering/DURABILITY_QUALIFICATION.md. SIGKILL to ready
|
||||
took 19.5–19.6 s under the production lease in all three builds (the 60 s
|
||||
target is asserted). Acknowledgement latency p50 was about 549 ms and p99
|
||||
about 1048 ms. No acknowledged input was lost, and one durable death was
|
||||
recorded across five owners.
|
||||
- Mutations (each restored and rebuilt). Each was caught:
|
||||
- no retired-build trigger;
|
||||
- no build check in four-argument acquisition;
|
||||
- no barrier check in build-less acquisition;
|
||||
- a barrier that does not fence holders;
|
||||
- a host that retries an unsupported acquisition instead of quarantining.
|
||||
The migration test caught this one after the host-test correction.
|
||||
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
|
||||
host or storage loss (M8).
|
||||
|
||||
Reference in New Issue
Block a user