Complete M7.9b upgrade barrier and durability qualification
Schema v8 binds authority to builds (ADR-0068): acquisition names the host build and only retained, unretired builds may hold a lease; no commit may name a retired build; upgrade_barrier retires a build and fences every holder in one transaction, while build-less pre-v8 hosts keep working until the first barrier. Tests cover a writer surviving a failed and then a real online migration, old-writer fencing on every path, whole-history recovery by the new build, and the M7 qualification fixture (19.6 s SIGKILL-to-ready under the production lease, no lost acknowledgements). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -593,3 +593,11 @@ M7.9a implemented and locally qualified (2026-09-29), under ADR-0067:
|
||||
- a full-chain retention audit is added.
|
||||
Next: M7.9b, covering the upgrade barrier and old-writer fencing, migration
|
||||
failure while serving, and measured recovery and batching latency.
|
||||
|
||||
M7.9b implemented and locally qualified (2026-09-29), under ADR-0068:
|
||||
- build-bound authority, and an upgrade barrier that retires a build and fences
|
||||
old writers;
|
||||
- a failed migration leaves the previous schema exactly, while writers continue;
|
||||
- rule and format changes are refused explicitly;
|
||||
- measured recovery (19.6 s against a 60 s target) and batching latency.
|
||||
M7 is complete locally. Next: M8.1, the host inventory and budgets.
|
||||
|
||||
@@ -928,3 +928,29 @@ configurations; the PostgreSQL suites passed in three.
|
||||
Next: M7.9b, covering the upgrade barrier and old-writer fencing, schema
|
||||
migration failure while a host is serving, and measured recovery and batching
|
||||
latency.
|
||||
|
||||
## M7.9b upgrade barrier and durability qualification (2026-09-29)
|
||||
|
||||
ADR-0068 and schema v8 bind authority to builds:
|
||||
- Acquisition names the host's build, and only a retained build that has not been
|
||||
retired may hold a lease.
|
||||
- No commit may name a retired build.
|
||||
- `upgrade_barrier` retires a build and fences every holder in one transaction.
|
||||
- Hosts built before M7.9b keep working until the first barrier.
|
||||
|
||||
A lease-holding writer survived a failed v8 migration (the database stayed at v7
|
||||
exactly) and then the real migration. After the barrier it could never regain
|
||||
authority. The new build recovered the whole history, including links written by
|
||||
the retired build.
|
||||
|
||||
Qualification (engineering/DURABILITY_QUALIFICATION.md), under the production
|
||||
lease:
|
||||
- SIGKILL to successor readiness: 19.5–19.6 s against the proposed 60 s target;
|
||||
- acknowledgement latency with a 1 s flush: p50 549 ms, p99 1048 ms;
|
||||
- a death was acknowledged in 44 ms;
|
||||
- no acknowledged input was lost.
|
||||
|
||||
**M7 exit criteria are met locally.** Container, CI and cluster execution have not
|
||||
run. Host-loss durability is M8.
|
||||
|
||||
Next: M8.1, the host inventory and budgets.
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# ADR-0068: Upgrade barrier, build-bound authority and durability qualification
|
||||
|
||||
## Status
|
||||
Accepted for M7.9b, 2026-09-29. It completes M7.9 with ADR-0067 (clones). It
|
||||
changes the ADR-0060 acquisition contract (schema v8). Rules and format
|
||||
conversions remain unimplemented: no converter exists, so every change is
|
||||
refused explicitly.
|
||||
|
||||
## Context
|
||||
M7.9 requires:
|
||||
- explicit rule and format rejection or migration;
|
||||
- evidence for a migration failure and for fencing old writers;
|
||||
- measured recovery and batching latency.
|
||||
|
||||
PERSISTENCE_ARCHITECTURE's upgrade path is: barrier, clone, migrate, compare,
|
||||
approve, then a compatible deployment at a world barrier. That implies:
|
||||
- A new build must be able to take over a region.
|
||||
- Every writer of the old build must be stopped for good, including one that is
|
||||
paused, idle, or restarted from an old image.
|
||||
- History written by the old build must stay recoverable.
|
||||
|
||||
Before v8, any retained build could publish, and acquisition did not name a
|
||||
build. A fenced old host could therefore re-acquire an idle region and renew it
|
||||
forever.
|
||||
|
||||
## Decision
|
||||
**Build-bound authority (schema v8).**
|
||||
- `retained_builds.retired_at` marks a build as retired from publication. The
|
||||
row, and so the recoverability of its history, stays.
|
||||
- A `region_commits` insert trigger refuses any new commit that names a retired
|
||||
build (EVL06). This covers publications, provisioned geneses and clone
|
||||
geneses.
|
||||
- `acquire_region_lease(world, region, duration, build)` grants authority only to
|
||||
a retained, unretired build (EVL06 otherwise). The C++ API takes the build. The
|
||||
host quarantines on `unsupported` (exit 4): a retired image never serves or
|
||||
holds an idle lease.
|
||||
- The build-less 3-argument form remains for pre-v8 hosts during a rolling
|
||||
schema upgrade. It refuses once any build has been retired.
|
||||
- Both forms wrap the internal `grant_region_lease`, which the runtime cannot
|
||||
execute. The runtime audit now covers twelve functions.
|
||||
|
||||
**Barrier.** The owner-only `upgrade_barrier(build)` retires one build and clears
|
||||
every current lease holder in the database, in one transaction:
|
||||
- Existing holders are fenced at their next renewal or publication.
|
||||
- They cannot re-acquire by either form.
|
||||
- A commit already waiting on a lease row is judged after the barrier commits.
|
||||
- Retiring the last current build is refused.
|
||||
- Epochs are kept, so the next acquisition takes a fresh one.
|
||||
|
||||
Operator order:
|
||||
1. Migrate the schema. Old hosts keep their leases, because renew and publish are
|
||||
unchanged.
|
||||
2. Retain the new build.
|
||||
3. Deploy hosts with `--build new --retained old`.
|
||||
4. Run the barrier for the old build.
|
||||
|
||||
**Explicit rejection.** A region never changes profile, versions, quantum or
|
||||
history profile (EVL06 at publication; `unsupported` at recovery). A clone
|
||||
inherits them unchanged. No rules or format converter is registered. Adding one
|
||||
requires a new ADR, a versioned conversion that is replayed and compared on a
|
||||
clone, and approval. Retirement is irreversible through this interface. Undoing
|
||||
it is an audited owner break-glass action, valid only while formats and rules are
|
||||
identical.
|
||||
|
||||
**Qualification fixture.** The fixture is:
|
||||
- the production lease policy (20 s lease, 5 s renewal, 2 s margin) and a 1 s flush;
|
||||
- one lineage region paced at 50 ms per tick;
|
||||
- a local socket and filesystem, on WSL2.
|
||||
|
||||
It measures:
|
||||
- acknowledgement latency;
|
||||
- the time from SIGKILL to the successor's readiness, against the proposed 60 s
|
||||
target (asserted for this fixture only);
|
||||
- head verification, full audit and clone cost.
|
||||
|
||||
The results are in engineering/DURABILITY_QUALIFICATION.md.
|
||||
|
||||
## Consequences
|
||||
- Recovery after losing a worker is dominated by the lease: a successor waits out
|
||||
the dead holder's remaining lease, up to 20 s, and verification adds
|
||||
milliseconds. Shortening the lease trades that time against false fencing
|
||||
under database latency (ADR-0060).
|
||||
- Migrations run in one transaction under an advisory lock and a 5 s lock timeout.
|
||||
A publication waiting on migration locks may become ambiguous; resolution
|
||||
handles it (ADR-0063). A failed migration leaves the previous schema exactly.
|
||||
- Binaries built before M7.9b (3-argument acquisition) work against v8 until the
|
||||
first barrier. Binaries from M7.9b onwards require schema v8.
|
||||
|
||||
## Alternatives rejected
|
||||
- **Retirement only, checked at publication.** An idle old host could still hold
|
||||
a lease forever.
|
||||
- **Bumping epochs without retirement.** Old hosts would simply re-acquire.
|
||||
- **Dropping the 3-argument acquisition in v8.** Every old host would be stranded
|
||||
mid-upgrade before the new build was deployed.
|
||||
- **In-place rules migration of a live region.** It would change physical laws
|
||||
without the clone-and-compare approval that the vision requires.
|
||||
@@ -78,3 +78,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0065: Recovery before readiness](0065-recovery-before-readiness.md) - accepted for M7.7 head tail replay verification, schema v6 commit lookup and real kill-boundary tests.
|
||||
- [ADR-0066: Operations endpoints](0066-operations-endpoints.md) - accepted for M7.8 liveness vs authority readiness, bounded metrics, dependency-free responder and the server image.
|
||||
- [ADR-0067: Region clones and retained-history audit](0067-region-clones.md) - accepted for M7.9a re-identified clone genesis naming its source cut, isolated clone databases and full-chain retention audit.
|
||||
- [ADR-0068: Upgrade barrier and durability qualification](0068-upgrade-barrier.md) - accepted for M7.9b build-bound authority, retirement barrier fencing old writers, explicit refusal of rule/format changes and the measured recovery fixture.
|
||||
|
||||
@@ -235,3 +235,20 @@ database:
|
||||
It names the exact link whose evidence is missing, even though head recovery
|
||||
still succeeds. Migration, fencing of old writers, and measured recovery and
|
||||
batching latency are still M7.9b.
|
||||
|
||||
M7.9b (ADR-0068) closes the clone/migration row.
|
||||
- **Failed migration while serving.** A v8 migration that failed at its last
|
||||
statement left v7 exactly, and the lease-holding writer kept publishing. The
|
||||
real migration then committed under that same lease.
|
||||
- **Old writers.** The upgrade barrier retires the old build and fences every
|
||||
holder. After it:
|
||||
- the old writer cannot publish, renew or re-acquire, by either acquisition
|
||||
form;
|
||||
- no link naming the retired build can commit, even under a current lease;
|
||||
- a host of the old image quarantines;
|
||||
- the new build recovered the whole history and continued.
|
||||
- **Rejection.** Rule, format and profile changes are refused; no converter
|
||||
exists.
|
||||
- **Qualification.** Real SIGKILLs under the production lease were recovered in
|
||||
19.5–19.6 s against the proposed 60 s target, with no acknowledged input lost
|
||||
(see engineering/DURABILITY_QUALIFICATION.md).
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# M7 durability qualification (M7.9b)
|
||||
|
||||
Measured on 2026-09-29 by `durability_qualification` in
|
||||
`tests/region_migration_tests.cpp`, under `tools/ci/postgres-check.sh`. These are
|
||||
local development measurements under the fixture below. They are not a
|
||||
production service level, and they are not evidence about host loss (M8).
|
||||
|
||||
## Fixture
|
||||
- **Machine:** WSL2 Ubuntu on the maintainer's Windows 11 workstation.
|
||||
- **Database:** a private PostgreSQL 18.6 cluster, connected over a Unix socket.
|
||||
- **Storage:** filesystem object store on the same volume (`single_volume_development`).
|
||||
- **Workload:** one `environmental-lineage-v1` region with a 37,756-byte payload,
|
||||
driven by real `evo-server` processes.
|
||||
- **Host settings:** production lease policy (20 s lease, 5 s renewal, 2 s safety
|
||||
margin, 2 s renewal and 5 s commit deadlines) and the default 1 s flush interval.
|
||||
- **Inputs:** paced at the 50 ms tick, 300 in total over three owners. The
|
||||
consumer dies at tick 20.
|
||||
- **Kills:** after each 100 inputs, 10 more were sent and the owner was
|
||||
SIGKILLed 200 ms later, mid-batch. The successor started immediately, for two
|
||||
measured recoveries.
|
||||
|
||||
## Results
|
||||
|
||||
| Measure | Release (GCC14 -O3) | clang-tidy build | ASan/UBSan |
|
||||
|---|---|---|---|
|
||||
| Ack latency p50 / p99 / max (ms), n=299 | 549 / 1048 / 1048 | 549 / 1045 / 1049 | 530 / 1028 / 1028 |
|
||||
| Death tick ack latency (ms) | 44 | 48 | 23 |
|
||||
| SIGKILL to successor ready (ms), runs 1 and 2 | 19599 / 19548 | 19605 / 19557 | 19627 / 19561 |
|
||||
| Head verification p50 / max (µs), n=10 | 1626 / 11949 | 5826 / 16184 | 14419 / 25352 |
|
||||
| Full-history audit, 16 links, 300 ticks (ms) | 20 | 82 | 204 |
|
||||
| Clone into the second database (ms) | 12 | 19 | 36 |
|
||||
|
||||
The lease-store microbenchmarks in the same run (200 samples, Release) had these
|
||||
p50/p99 values:
|
||||
|
||||
| Operation | p50 (µs) | p99 (µs) |
|
||||
|---|---|---|
|
||||
| Acquire | 624 | 844 |
|
||||
| Renew | 600 | 744 |
|
||||
| Publish manifest | 750 | 965 |
|
||||
| Release | 603 | 717 |
|
||||
|
||||
Batch publication, including payload and trace upload with read-back, took
|
||||
6–8 ms.
|
||||
|
||||
## Interpretation
|
||||
- **Recovery is dominated by the lease.** Kill to ready was 19.5–19.6 s in every
|
||||
build. The dead holder's lease must expire in database time, and it had been
|
||||
renewed less than 5 s earlier. Verification (tens of milliseconds even under
|
||||
ASan) and process start are small.
|
||||
- **The target holds.** The proposed 60 s recovery target holds for this fixture
|
||||
with about 40 s of margin. The test asserts < 60 s for this fixture only.
|
||||
- **Nothing acknowledged was lost.** Every acknowledged input was at or below the
|
||||
committed head at each kill. The one death was recorded exactly once across
|
||||
three owners, and each successor resumed exactly at the committed head with a
|
||||
new epoch.
|
||||
- **Acknowledgement latency is the batching cost.** It is set by the flush
|
||||
interval: an input waits for its batch to commit, which averages about half a
|
||||
flush interval, plus a commit of under 10 ms. A death seals its batch and
|
||||
commits at once, so it is acknowledged fastest.
|
||||
- **Verification and audit costs scale.** They grow with payload size and chain
|
||||
length. At this size the full history is re-verified in about 20 ms (Release).
|
||||
Larger regions and longer histories need their own measurement before any
|
||||
target is claimed.
|
||||
|
||||
## Not measured
|
||||
The following were not measured:
|
||||
- host or storage loss;
|
||||
- network partitions between host and database;
|
||||
- a separate database host;
|
||||
- concurrent regions;
|
||||
- large payloads;
|
||||
- long histories;
|
||||
- Kubernetes rescheduling time.
|
||||
These belong to M8, together with off-host backup and restore.
|
||||
@@ -2485,3 +2485,52 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
stale-binary run was discarded and the mutation was rerun.
|
||||
- Not executed: clones across separate PostgreSQL clusters or hosts, Docker, the
|
||||
Gitea CI, and cluster runs.
|
||||
|
||||
## M7.9b upgrade barrier and durability qualification - 2026-09-29
|
||||
|
||||
- Every M7.9b change is in the Linux PostgreSQL build or in docs. Windows was not
|
||||
rerun; its 59/59 from M7.9a still covers the portable code, which is unchanged.
|
||||
- WSL Linux `checks` set (GCC14 Debug and Release, clang19 analysis, clang19
|
||||
ASan/UBSan) passed CTest 59/59 each. The format check passed.
|
||||
- The PostgreSQL suites passed in GCC Release, clang-tidy analysis and ASan/UBSan.
|
||||
After the mutation run, one host test was corrected: `unrestorable_head_quarantines`
|
||||
now retains its build, so the quarantine it checks again comes from recovery
|
||||
rather than acquisition. Release and analysis were rebuilt and rerun green after
|
||||
that change. The ASan build compiled with it, but its full run predates it.
|
||||
- Harness: upgrade paths now end at 1..8. The populated v6 database upgrades to v8.
|
||||
Version and checksum drift checks include v8.
|
||||
- Migration and barrier test (database `evo_online`, created at v7 from the
|
||||
checked-in files):
|
||||
- A pre-v8 writer with a build-less lease published ticks 1..20 (with the
|
||||
death).
|
||||
- An `apply.sql` run with an invalid v8 checksum failed. It left history 1..7,
|
||||
no `retired_at` column and no four-argument acquire. The writer renewed and
|
||||
published ticks 21..40.
|
||||
- The real migration then reached v8 while the writer held its lease, and the
|
||||
writer published ticks 41..60.
|
||||
- Barrier refusals: the runtime cannot call it (42501), and retiring the last
|
||||
current build is refused (22023).
|
||||
- After the barrier (1 lease fenced), the old writer's publication and renewal
|
||||
were fenced. Build-less acquisition returned EVL06, and acquisition with b1
|
||||
returned unsupported.
|
||||
- A b2 lease publishing a link that names b1 was refused as unsupported.
|
||||
- A b2 `evo-server` became ready at epoch 3 from committed tick 60 and
|
||||
acknowledged through tick 90. Its final head is built by b2.
|
||||
- The audit passes with both builds retained, and reports unsupported at
|
||||
sequence 0 with only b2.
|
||||
- A b1 `evo-server` quarantined (exit 4) without becoming ready, and the head
|
||||
was unchanged.
|
||||
- Qualification: see engineering/DURABILITY_QUALIFICATION.md. SIGKILL to ready
|
||||
took 19.5–19.6 s under the production lease in all three builds (the 60 s
|
||||
target is asserted). Acknowledgement latency p50 was about 549 ms and p99
|
||||
about 1048 ms. No acknowledged input was lost, and one durable death was
|
||||
recorded across five owners.
|
||||
- Mutations (each restored and rebuilt). Each was caught:
|
||||
- no retired-build trigger;
|
||||
- no build check in four-argument acquisition;
|
||||
- no barrier check in build-less acquisition;
|
||||
- a barrier that does not fence holders;
|
||||
- a host that retries an unsupported acquisition instead of quarantining.
|
||||
The migration test caught this one after the host-test correction.
|
||||
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
|
||||
host or storage loss (M8).
|
||||
|
||||
@@ -68,7 +68,8 @@ every role transitively in it, and rolls back on any violation:
|
||||
`pg_write_all_data`);
|
||||
- any table privilege in `evolution`, including via PUBLIC;
|
||||
- CREATE on the schema, the database or `public`;
|
||||
- EXECUTE outside the eleven fenced functions (schema v6).
|
||||
- EXECUTE outside the twelve fenced functions (schema v8 adds the build-aware
|
||||
`acquire_region_lease`).
|
||||
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
||||
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
||||
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
||||
@@ -145,6 +146,27 @@ receives clones. `evo-world-clone audit` verifies a region's whole retained
|
||||
history; it is read-only and needs only runtime read access. Never point a clone
|
||||
target at a production database or World0.
|
||||
|
||||
Schema v8 (ADR-0068) binds authority to builds:
|
||||
- Hosts from M7.9b onwards acquire with their build identity. Only a retained
|
||||
build that has not been retired may hold a lease.
|
||||
- No new commit may name a retired build (EVL06). History written by a retired
|
||||
build stays recoverable, so keep it retained.
|
||||
- Hosts built before M7.9b use build-less acquisition, which keeps working until
|
||||
the first barrier.
|
||||
|
||||
To upgrade the build:
|
||||
1. Migrate the schema. Running hosts keep their leases.
|
||||
2. Insert the new build into `retained_builds`.
|
||||
3. Deploy hosts with `--build <new> --retained <old>`.
|
||||
4. On an owner connection, run
|
||||
`SELECT evolution.upgrade_barrier('\x<old build>');`
|
||||
It retires the old build and fences every current holder in the database. It
|
||||
returns the number of leases fenced, and refuses to retire the last current
|
||||
build.
|
||||
|
||||
Old images then quarantine (exit 4) instead of serving. Rules and format changes
|
||||
are refused explicitly; there is no converter.
|
||||
|
||||
There is no operator CLI for provisioning original regions yet, and no real world
|
||||
has been provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
have no payload.
|
||||
|
||||
Reference in New Issue
Block a user