Files
aevolution/docs/engineering/DURABILITY_QUALIFICATION.md
dingoandClaude Opus 5.5 f59cca2c3b Complete M7.9b upgrade barrier and durability qualification
Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 07:31:04 -04:00

76 lines
3.5 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# M7 durability qualification (M7.9b)
Measured on 2026-09-29 by `durability_qualification` in
`tests/region_migration_tests.cpp`, under `tools/ci/postgres-check.sh`. These are
local development measurements under the fixture below. They are not a
production service level, and they are not evidence about host loss (M8).
## Fixture
- **Machine:** WSL2 Ubuntu on the maintainer's Windows 11 workstation.
- **Database:** a private PostgreSQL 18.6 cluster, connected over a Unix socket.
- **Storage:** filesystem object store on the same volume (`single_volume_development`).
- **Workload:** one `environmental-lineage-v1` region with a 37,756-byte payload,
driven by real `evo-server` processes.
- **Host settings:** production lease policy (20 s lease, 5 s renewal, 2 s safety
margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
- **Inputs:** paced at the 50 ms tick, 300 in total over three owners. The
consumer dies at tick 20.
- **Kills:** after each 100 inputs, 10 more were sent and the owner was
SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two
measured recoveries.
## Results
| Measure | Release (GCC14 -O3) | clang-tidy build | ASan/UBSan |
|---|---|---|---|
| Ack latency p50 / p99 / max (ms), n=299 | 549 / 1048 / 1048 | 549 / 1045 / 1049 | 530 / 1028 / 1028 |
| Death tick ack latency (ms) | 44 | 48 | 23 |
| SIGKILL to successor ready (ms), runs 1 and 2 | 19599 / 19548 | 19605 / 19557 | 19627 / 19561 |
| Head verification p50 / max (µs), n=10 | 1626 / 11949 | 5826 / 16184 | 14419 / 25352 |
| Full-history audit, 16 links, 300 ticks (ms) | 20 | 82 | 204 |
| Clone into the second database (ms) | 12 | 19 | 36 |
The lease-store microbenchmarks in the same run (200 samples, Release) had these
p50/p99 values:
| Operation | p50 (µs) | p99 (µs) |
|---|---|---|
| Acquire | 624 | 844 |
| Renew | 600 | 744 |
| Publish manifest | 750 | 965 |
| Release | 603 | 717 |
Batch publication, including payload and trace upload with read-back, took
6–8 ms.
## Interpretation
- **Recovery is dominated by the lease.** Kill to ready was 19.5–19.6 s in every
build. The dead holder's lease must expire in database time, and it had been
renewed less than 5 s earlier. Verification (tens of milliseconds even under
ASan) and process start are small.
- **The target holds.** The proposed 60 s recovery target holds for this fixture
with about 40 s of margin. The test asserts < 60 s for this fixture only.
- **Nothing acknowledged was lost.** Every acknowledged input was at or below the
committed head at each kill. The one death was recorded exactly once across
three owners, and each successor resumed exactly at the committed head with a
new epoch.
- **Acknowledgement latency is the batching cost.** It is set by the flush
interval: an input waits for its batch to commit, which averages about half a
flush interval, plus a commit of under 10 ms. A death seals its batch and
commits at once, so it is acknowledged fastest.
- **Verification and audit costs scale.** They grow with payload size and chain
length. At this size the full history is re-verified in about 20 ms (Release).
Larger regions and longer histories need their own measurement before any
target is claimed.
## Not measured
The following were not measured:
- host or storage loss;
- network partitions between host and database;
- a separate database host;
- concurrent regions;
- large payloads;
- long histories;
- Kubernetes rescheduling time.
These belong to M8, together with off-host backup and restore.