Fix M5.7 compaction and failure recovery

This commit is contained in:
dingo
2026-09-26 19:40:16 -04:00
parent 56f533c738
commit a2a12fdc69
6 changed files with 187 additions and 29 deletions
+18 -10
View File
@@ -27,20 +27,27 @@ state silently. No automatic orphan deletion in this gate.
Snapshot checkpoint validates bytes by restoring them under budget before publication.
A journal commit must match the current checkpoint's exact bytes and high-water, and
pass detached replay. Only one journal segment is active per checkpoint; checkpoint
after replay to compact it. Recovery validates the selected manifest and referenced
payloads using M5.6, then replays the optional segment before returning a candidate.
Existing valid manifest stays the source until new publication completes.
pass detached replay. Only one journal segment is active per checkpoint. While one is
published, compaction accepts only the byte-identical canonical snapshot of recovery's
replayed result; command high-water alone cannot prove continuation or prevent a fork.
Recovery validates the selected manifest and referenced payloads using M5.6, then
replays the optional segment before returning a candidate. Existing valid manifest
stays the source until new publication completes. A poisoned handle refuses recovery,
reports no current generation, and requires reopen to resolve an ambiguous publication
error.
Implement the minimal native OS boundary directly: POSIX lock/write/fsync/link/rename/
directory fsync; Windows exclusive CreateFile and write-through replacement. No third
party library. Root directory is trusted and locally administered. Files are not
network-safe/authenticated; FNV detects accidental corruption only.
Provide one-shot failure injection at payload sync, manifest sync, and manifest publish
for deterministic crash-boundary tests. After injected interruption, discard the handle
and reopen. A failure before manifest replacement yields old state; after replacement
yields new state. Allocation/OS errors do not clear dirty state.
Provide one-shot failure injection at payload sync, manifest sync, manifest publish,
partial payload/manifest write (out of space), and post-replacement directory sync for
deterministic crash-boundary tests. Partial-write failures leave only an ignored temp
file and allow retry; injected interruptions poison the handle. After an interruption
or ambiguous sync failure, discard the handle and reopen. A failure before manifest
replacement yields old state; after replacement yields new state. Allocation/OS errors
do not clear dirty state.
## Alternatives Considered
In-place overwrite can destroy the old checkpoint on short write. A single manifest can
@@ -60,6 +67,7 @@ differ. Same-host files do not survive host loss. Fail-closed corruption may req
operator recovery. No production DR/RPO claim.
## Consequences
Test first checkpoint, journal, compaction, restart continuation, competing lock, corrupt
payload/manifest, every injected boundary, missing sequence and full-disk/OS errors.
Test first checkpoint, journal, exact replay compaction, divergent-state rejection,
restart continuation, competing lock, corrupt payload/manifest, every injected boundary,
partial-write recovery, missing sequence and full-disk/OS errors.
Record Linux checks separately. M5.8 performs the whole create/edit/compact/restart proof.
+27 -8
View File
@@ -28,14 +28,33 @@ Executed on Windows x64 / MSVC 19.44.35207:
- `tools/windows-check.ps1 -Preset release -Action format`: passed.
- `git diff --check`: passed before commit.
New coverage validates checkpoint/journal/recovery, compaction over a published
journal, exclusive writer lock, all three injected publication boundaries, corrupt
manifest and missing published payload failure. Before-manifest failures recover
the previous generation; after-manifest replacement recovers the new generation.
Actual power-cut/full-disk behavior, POSIX syscall path, Linux CI/libFuzzer, and
filesystem-controller flush guarantees remain unverified locally. There is no
automatic orphan cleanup, production storage adapter, Kubernetes deployment or
World0 change.
New coverage validates checkpoint/journal/recovery, exact compaction over a published
journal, divergent-state rejection, exclusive writer lock, injected publication
boundaries, partial-write/out-of-space outcomes, corrupt manifest and missing payload.
Before-manifest failures recover the previous state; after replacement, reopen
recovers the new state. Actual power-cut/disk-full behavior, POSIX syscall path,
Linux CI/libFuzzer, and filesystem-controller flush guarantees remain unverified
locally. There is no automatic orphan cleanup, production storage adapter,
Kubernetes deployment or World0 change.
### M5.7 review corrections
The follow-up review found that command high-water alone allowed a divergent
checkpoint to replace an active journal. Checkpoint now compares its canonical bytes
with the fully recovered journal result before compaction. Poisoned handles reject
recovery; reopen resolves ambiguous post-replacement directory-sync failures.
Interruption tests now use a different edited payload and compare recovered canonical
state and debris across the commit point. Deterministic partial-write/out-of-space
injection covers payload and manifest temporaries, including retry, and a directory
sync failure hook verifies poison/reopen behavior. The injection models a partial
write outcome; it is not a physical disk-full or power-cut test.
Revalidated after these corrections on Windows x64 / MSVC 19.44.35207:
- Release configure/build and 47/47 CTests passed.
- clang-tidy analysis build and 47/47 CTests passed.
- ASan build and 47/47 CTests passed.
- clang-format and `git diff --check` passed.
## Tools and host
+4 -1
View File
@@ -82,6 +82,9 @@ and [the manifest schema](../../schemas/local-world-manifest.md). Two alternatin
manifest slots select immutable M5.6 snapshot/journal payloads. Recovery validates
the selected snapshot and optional journal into a detached store. The manifest
replacement is the local commit point; this provides no multi-host durability,
production lease, authentication or disaster recovery guarantee.
production lease, authentication or disaster recovery guarantee. Compaction while a
journal is active accepts only its byte-identical replay result, and poisoned handles
must reopen before recovery. Partial-write and post-replacement sync failures have
deterministic test injection.
M5.6 does not write files, fsync, delete journals, clear dirty state or recover a live
server process. M7 supplies the production durability/ownership path.