Files
aevolution/docs/engineering/DURABILITY_QUALIFICATION.md
dingoandClaude Opus 5.5 f59cca2c3b Complete M7.9b upgrade barrier and durability qualification
Schema v8 binds authority to builds (ADR-0068): acquisition names the
host build and only retained, unretired builds may hold a lease; no
commit may name a retired build; upgrade_barrier retires a build and
fences every holder in one transaction, while build-less pre-v8 hosts
keep working until the first barrier. Tests cover a writer surviving a
failed and then a real online migration, old-writer fencing on every
path, whole-history recovery by the new build, and the M7 qualification
fixture (19.6 s SIGKILL-to-ready under the production lease, no lost
acknowledgements).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 07:31:04 -04:00

3.5 KiB
Raw Permalink Blame History

M7 durability qualification (M7.9b)

Measured on 2026-09-29 by durability_qualification in tests/region_migration_tests.cpp, under tools/ci/postgres-check.sh. These are local development measurements under the fixture below. They are not a production service level, and they are not evidence about host loss (M8).

Fixture

  • Machine: WSL2 Ubuntu on the maintainer's Windows 11 workstation.
  • Database: a private PostgreSQL 18.6 cluster, connected over a Unix socket.
  • Storage: filesystem object store on the same volume (single_volume_development).
  • Workload: one environmental-lineage-v1 region with a 37,756-byte payload, driven by real evo-server processes.
  • Host settings: production lease policy (20 s lease, 5 s renewal, 2 s safety margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
  • Inputs: paced at the 50 ms tick, 300 in total over three owners. The consumer dies at tick 20.
  • Kills: after each 100 inputs, 10 more were sent and the owner was SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two measured recoveries.

Results

Measure Release (GCC14 -O3) clang-tidy build ASan/UBSan
Ack latency p50 / p99 / max (ms), n=299 549 / 1048 / 1048 549 / 1045 / 1049 530 / 1028 / 1028
Death tick ack latency (ms) 44 48 23
SIGKILL to successor ready (ms), runs 1 and 2 19599 / 19548 19605 / 19557 19627 / 19561
Head verification p50 / max (µs), n=10 1626 / 11949 5826 / 16184 14419 / 25352
Full-history audit, 16 links, 300 ticks (ms) 20 82 204
Clone into the second database (ms) 12 19 36

The lease-store microbenchmarks in the same run (200 samples, Release) had these p50/p99 values:

Operation p50 (µs) p99 (µs)
Acquire 624 844
Renew 600 744
Publish manifest 750 965
Release 603 717

Batch publication, including payload and trace upload with read-back, took 6–8 ms.

Interpretation

  • Recovery is dominated by the lease. Kill to ready was 19.5–19.6 s in every build. The dead holder's lease must expire in database time, and it had been renewed less than 5 s earlier. Verification (tens of milliseconds even under ASan) and process start are small.
  • The target holds. The proposed 60 s recovery target holds for this fixture with about 40 s of margin. The test asserts < 60 s for this fixture only.
  • Nothing acknowledged was lost. Every acknowledged input was at or below the committed head at each kill. The one death was recorded exactly once across three owners, and each successor resumed exactly at the committed head with a new epoch.
  • Acknowledgement latency is the batching cost. It is set by the flush interval: an input waits for its batch to commit, which averages about half a flush interval, plus a commit of under 10 ms. A death seals its batch and commits at once, so it is acknowledged fastest.
  • Verification and audit costs scale. They grow with payload size and chain length. At this size the full history is re-verified in about 20 ms (Release). Larger regions and longer histories need their own measurement before any target is claimed.

Not measured

The following were not measured:

  • host or storage loss;
  • network partitions between host and database;
  • a separate database host;
  • concurrent regions;
  • large payloads;
  • long histories;
  • Kubernetes rescheduling time. These belong to M8, together with off-host backup and restore.