Fix object reuse durability and refuse unsafe pre-v5 upgrades

Review P1 (object durability): a name installed by a put whose directory
fsync failed was later reused as success without syncing again, so a
publication could acknowledge an object that might vanish on power
loss. Reuse (existing and lost-link-race paths) now fsyncs the verified
file and the directory before succeeding, and namespace opens always
sync their parent. A test-only ObjectStoreFaults hook injects directory
sync failures; tests cover failed sync then retries, namespace creation
syncs, and publication refusing to proceed until durability holds.

Review P1 (pre-v5 history): v5 recovery requires one durable decision
per terminal actor, but v3/v4 published checkpoints carry none, so an
accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now
runs before 0005 in every upgrade path and refuses (EVL10) when any
published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is
unchanged. CI publishes a real v4 checkpoint and proves the upgrade is
refused with nothing changed; a committed death without a decision is
shown to recover as quarantined history.

Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2;
reverting each fix is caught by its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 19:20:35 -04:00
co-authored by Claude Opus 5.5
parent dd7e8028b0
commit ab4bf5355f
10 changed files with 259 additions and 32 deletions
+6 -2
View File
@@ -54,8 +54,12 @@ computes commit digests, and the host checks that the receipt matches.
are named by the hex SHA-256 of their content.
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
link never replaces an existing name), and then fsyncs the directory.
- Existing identical content is reused. Differing content already under a key is
reported as corrupt and never overwritten.
- Existing identical content is reused only after its file and directory are
fsynced again, whether it came from an earlier attempt whose directory sync failed
or from a concurrent installer. A visible name alone is not durability (review
fix). Namespace directories sync their parent on every open, not only on
creation. Differing content already under a key is reported as corrupt and never
overwritten.
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
and verify the digest.
- There is no delete or GC operation: unreferenced uploads are orphans.
+7
View File
@@ -63,6 +63,13 @@ parameter; these profiles can produce at most 2. The decision identity is
outbox rows only a first delivery timestamp (EVL09).
- The runtime role's audited function set grows to ten.
**Upgrades** from v3 or v4 run `migrations/pre_v5_guard.sql` inside the same
transaction, before 0005. The guard refuses (EVL10), changing nothing, if any
published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold
terminal organisms without decisions, and v5 recovery would quarantine it.
Genesis-only regions pass, because tick 0 has no terminal organisms. A verified
legacy-history conversion is future work; there is none now.
**Recovery** loads decisions and requires lifecycle agreement:
- one death per terminal actor, none for an active actor;
- no death after the head tick, and organisms match the profile;
+31
View File
@@ -2265,3 +2265,34 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
0.75-0.86 ms.
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
consumer, an injected failure between commit insert and statement end, and TSan.
### M7.5 review fixes - 2026-09-28
- P1 object durability:
- `put()` now re-fsyncs the file and the directory before reusing an existing
object, on both the found-existing and the lost link-race paths.
- Namespace opens always sync the parent directory.
- A test-only `ObjectStoreFaults` hook injects directory-sync failures and
counts successful syncs.
- Tests: a namespace-creation sync failure, then a reopen still failing, then
success with exactly 2 syncs. An install sync failure leaves the object
visible but unacknowledged, three retries while syncs fail all return io with
no extra syncs, then success with +1 sync; every reuse adds a sync.
- Publication with 1, 1 and 2 injected failures returns `upload`/io and sends
no manifest (head stays 0), then publishes once syncs succeed.
- Reverting the reuse sync or the always-sync namespace behavior makes these
tests fail.
- P1 pre-v5 history:
- `pre_v5_guard.sql` (EVL10) runs before 0005 in every upgrade path. 0005 and
its checksum are unchanged.
- CI builds a real v4 database: v1 genesis, a lease, and a sequence-1
publication through the v4 function. The v5 upgrade is refused with the
expected message; history stays 1..4, both commits remain, and no decision
table is created. Replacing the guard with a no-op makes that check fail.
- A committed death published without any decision (the state an accepted
legacy upgrade would produce) recovers as `history`, quarantined and never
served.
- Reran GCC14 Release (3 runs), clang-tidy analysis (1) and ASan/UBSan (2)
PostgreSQL suites, all passing, and the repository format check. The changes
are Linux-only persistence code, SQL and tests; the portable targets and
Windows are unchanged.
+3 -1
View File
@@ -120,7 +120,9 @@ Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
a first delivery time. Owners can disable triggers, so treat that as an audited
break-glass action. EVL08 means an outcome was already final. After an
ambiguous publication, resolve the exact prepared bytes before doing anything
else.
else. Upgrading a v3/v4 database that has published checkpoints (not just
genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions,
or wait for an explicit legacy-history conversion.
There is no operator CLI for provisioning yet, and no real world has been
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads