Fix M5.7 compaction and failure recovery

This commit is contained in:
dingo
2026-09-26 19:40:16 -04:00
parent 56f533c738
commit a2a12fdc69
6 changed files with 187 additions and 29 deletions
+18 -10
View File
@@ -27,20 +27,27 @@ state silently. No automatic orphan deletion in this gate.
Snapshot checkpoint validates bytes by restoring them under budget before publication.
A journal commit must match the current checkpoint's exact bytes and high-water, and
pass detached replay. Only one journal segment is active per checkpoint; checkpoint
after replay to compact it. Recovery validates the selected manifest and referenced
payloads using M5.6, then replays the optional segment before returning a candidate.
Existing valid manifest stays the source until new publication completes.
pass detached replay. Only one journal segment is active per checkpoint. While one is
published, compaction accepts only the byte-identical canonical snapshot of recovery's
replayed result; command high-water alone cannot prove continuation or prevent a fork.
Recovery validates the selected manifest and referenced payloads using M5.6, then
replays the optional segment before returning a candidate. Existing valid manifest
stays the source until new publication completes. A poisoned handle refuses recovery,
reports no current generation, and requires reopen to resolve an ambiguous publication
error.
Implement the minimal native OS boundary directly: POSIX lock/write/fsync/link/rename/
directory fsync; Windows exclusive CreateFile and write-through replacement. No third
party library. Root directory is trusted and locally administered. Files are not
network-safe/authenticated; FNV detects accidental corruption only.
Provide one-shot failure injection at payload sync, manifest sync, and manifest publish
for deterministic crash-boundary tests. After injected interruption, discard the handle
and reopen. A failure before manifest replacement yields old state; after replacement
yields new state. Allocation/OS errors do not clear dirty state.
Provide one-shot failure injection at payload sync, manifest sync, manifest publish,
partial payload/manifest write (out of space), and post-replacement directory sync for
deterministic crash-boundary tests. Partial-write failures leave only an ignored temp
file and allow retry; injected interruptions poison the handle. After an interruption
or ambiguous sync failure, discard the handle and reopen. A failure before manifest
replacement yields old state; after replacement yields new state. Allocation/OS errors
do not clear dirty state.
## Alternatives Considered
In-place overwrite can destroy the old checkpoint on short write. A single manifest can
@@ -60,6 +67,7 @@ differ. Same-host files do not survive host loss. Fail-closed corruption may req
operator recovery. No production DR/RPO claim.
## Consequences
Test first checkpoint, journal, compaction, restart continuation, competing lock, corrupt
payload/manifest, every injected boundary, missing sequence and full-disk/OS errors.
Test first checkpoint, journal, exact replay compaction, divergent-state rejection,
restart continuation, competing lock, corrupt payload/manifest, every injected boundary,
partial-write recovery, missing sequence and full-disk/OS errors.
Record Linux checks separately. M5.8 performs the whole create/edit/compact/restart proof.
+27 -8
View File
@@ -28,14 +28,33 @@ Executed on Windows x64 / MSVC 19.44.35207:
- `tools/windows-check.ps1 -Preset release -Action format`: passed.
- `git diff --check`: passed before commit.
New coverage validates checkpoint/journal/recovery, compaction over a published
journal, exclusive writer lock, all three injected publication boundaries, corrupt
manifest and missing published payload failure. Before-manifest failures recover
the previous generation; after-manifest replacement recovers the new generation.
Actual power-cut/full-disk behavior, POSIX syscall path, Linux CI/libFuzzer, and
filesystem-controller flush guarantees remain unverified locally. There is no
automatic orphan cleanup, production storage adapter, Kubernetes deployment or
World0 change.
New coverage validates checkpoint/journal/recovery, exact compaction over a published
journal, divergent-state rejection, exclusive writer lock, injected publication
boundaries, partial-write/out-of-space outcomes, corrupt manifest and missing payload.
Before-manifest failures recover the previous state; after replacement, reopen
recovers the new state. Actual power-cut/disk-full behavior, POSIX syscall path,
Linux CI/libFuzzer, and filesystem-controller flush guarantees remain unverified
locally. There is no automatic orphan cleanup, production storage adapter,
Kubernetes deployment or World0 change.
### M5.7 review corrections
The follow-up review found that command high-water alone allowed a divergent
checkpoint to replace an active journal. Checkpoint now compares its canonical bytes
with the fully recovered journal result before compaction. Poisoned handles reject
recovery; reopen resolves ambiguous post-replacement directory-sync failures.
Interruption tests now use a different edited payload and compare recovered canonical
state and debris across the commit point. Deterministic partial-write/out-of-space
injection covers payload and manifest temporaries, including retry, and a directory
sync failure hook verifies poison/reopen behavior. The injection models a partial
write outcome; it is not a physical disk-full or power-cut test.
Revalidated after these corrections on Windows x64 / MSVC 19.44.35207:
- Release configure/build and 47/47 CTests passed.
- clang-tidy analysis build and 47/47 CTests passed.
- ASan build and 47/47 CTests passed.
- clang-format and `git diff --check` passed.
## Tools and host
+4 -1
View File
@@ -82,6 +82,9 @@ and [the manifest schema](../../schemas/local-world-manifest.md). Two alternatin
manifest slots select immutable M5.6 snapshot/journal payloads. Recovery validates
the selected snapshot and optional journal into a detached store. The manifest
replacement is the local commit point; this provides no multi-host durability,
production lease, authentication or disaster recovery guarantee.
production lease, authentication or disaster recovery guarantee. Compaction while a
journal is active accepts only its byte-identical replay result, and poisoned handles
must reopen before recovery. Partial-write and post-replacement sync failures have
deterministic test injection.
M5.6 does not write files, fsync, delete journals, clear dirty state or recover a live
server process. M7 supplies the production durability/ownership path.
+14 -2
View File
@@ -31,6 +31,12 @@ enum class PublicationInterruption : std::uint8_t {
after_manifest_sync,
after_manifest_publish
};
enum class PublicationFailure : std::uint8_t {
none,
payload_out_of_space,
manifest_out_of_space,
manifest_directory_sync
};
struct FileCommit {
std::uint64_t generation{}, covered_command{}, snapshot_checksum{};
bool has_journal{};
@@ -40,17 +46,21 @@ class LocalWorldFiles {
public:
[[nodiscard]] static std::expected<std::unique_ptr<LocalWorldFiles>, LocalFileError>
open(const std::filesystem::path& trusted_root, WorldIdentity identity, std::size_t byte_budget,
PublicationInterruption interrupt_once = PublicationInterruption::none);
PublicationInterruption interrupt_once = PublicationInterruption::none,
PublicationFailure fail_once = PublicationFailure::none);
LocalWorldFiles(const LocalWorldFiles&) = delete;
LocalWorldFiles& operator=(const LocalWorldFiles&) = delete;
LocalWorldFiles(LocalWorldFiles&&) = delete;
LocalWorldFiles& operator=(LocalWorldFiles&&) = delete;
~LocalWorldFiles();
// An ambiguous publication failure poisons this handle; reopen before recovery.
[[nodiscard]] std::expected<std::unique_ptr<ChunkStore>, LocalFileError> recover() const;
// While a journal is committed, compaction requires its byte-identical replay result.
[[nodiscard]] std::expected<FileCommit, LocalFileError> checkpoint(const ChunkStore& store);
[[nodiscard]] std::expected<FileCommit, LocalFileError>
publish_journal(std::span<const std::uint8_t> checkpoint,
std::span<const std::uint8_t> journal);
// Empty on an uninitialized or poisoned handle.
[[nodiscard]] std::optional<FileCommit> current() const noexcept;
private:
@@ -61,7 +71,8 @@ class LocalWorldFiles {
bool has_journal{};
};
LocalWorldFiles(std::filesystem::path directory, WorldIdentity identity, std::size_t budget,
PublicationInterruption interruption, std::unique_ptr<Lock> lock);
PublicationInterruption interruption, PublicationFailure failure,
std::unique_ptr<Lock> lock);
[[nodiscard]] std::expected<FileCommit, LocalFileError>
publish(std::span<const std::uint8_t> payload, bool journal, std::uint64_t covered,
std::uint64_t payload_checksum);
@@ -69,6 +80,7 @@ class LocalWorldFiles {
WorldIdentity identity_;
std::size_t budget_;
PublicationInterruption interruption_;
PublicationFailure failure_;
std::unique_ptr<Lock> lock_;
std::optional<Manifest> manifest_;
std::size_t active_slot_{};
+29 -6
View File
@@ -211,14 +211,14 @@ struct LocalWorldFiles::Lock {
LocalWorldFiles::LocalWorldFiles(std::filesystem::path directory, WorldIdentity identity,
std::size_t budget, PublicationInterruption interruption,
std::unique_ptr<Lock> lock)
PublicationFailure failure, std::unique_ptr<Lock> lock)
: directory_{std::move(directory)}, identity_{identity}, budget_{budget},
interruption_{interruption}, lock_{std::move(lock)} {}
interruption_{interruption}, failure_{failure}, lock_{std::move(lock)} {}
LocalWorldFiles::~LocalWorldFiles() = default;
std::expected<std::unique_ptr<LocalWorldFiles>, LocalFileError>
LocalWorldFiles::open(const std::filesystem::path& root, WorldIdentity identity, std::size_t budget,
PublicationInterruption interruption) {
PublicationInterruption interruption, PublicationFailure failure) {
if (!identity.world || !identity.frame || !budget)
return std::unexpected(LocalFileError::invalid_identity);
const auto directory = root / ("world-" + hex(identity.world) + "-" + hex(identity.frame));
@@ -238,7 +238,7 @@ LocalWorldFiles::open(const std::filesystem::path& root, WorldIdentity identity,
return std::unexpected(LocalFileError::busy);
#endif
auto result = std::unique_ptr<LocalWorldFiles>(
new LocalWorldFiles(directory, identity, budget, interruption, std::move(lock)));
new LocalWorldFiles(directory, identity, budget, interruption, failure, std::move(lock)));
for (std::size_t slot = 0; slot < 2; ++slot) {
auto bytes = read_file(directory / ("manifest." + std::to_string(slot)), manifest_size);
if (!bytes) {
@@ -260,13 +260,15 @@ LocalWorldFiles::open(const std::filesystem::path& root, WorldIdentity identity,
}
std::optional<FileCommit> LocalWorldFiles::current() const noexcept {
if (!manifest_)
if (poisoned_ || !manifest_)
return std::nullopt;
return FileCommit{manifest_->generation, manifest_->covered_command,
manifest_->snapshot_checksum, manifest_->has_journal};
}
std::expected<std::unique_ptr<ChunkStore>, LocalFileError> LocalWorldFiles::recover() const {
if (poisoned_)
return std::unexpected(LocalFileError::poisoned);
if (!manifest_)
return std::unexpected(LocalFileError::empty);
const auto snapshot_path =
@@ -314,6 +316,12 @@ LocalWorldFiles::publish(std::span<const std::uint8_t> payload, bool journal, st
const auto temp = directory_ / (std::string{prefix} + ".payload.tmp");
std::filesystem::remove(temp, ec);
ec.clear();
if (failure_ == PublicationFailure::payload_out_of_space) {
failure_ = PublicationFailure::none;
const auto partial_size = std::max<std::size_t>(1, payload.size() / 2);
(void)write_synced(temp, payload.first(partial_size));
return std::unexpected(LocalFileError::io);
}
if (!write_synced(temp, payload))
return std::unexpected(LocalFileError::io);
if (interruption_ == PublicationInterruption::after_payload_sync) {
@@ -349,6 +357,12 @@ LocalWorldFiles::publish(std::span<const std::uint8_t> payload, bool journal, st
const auto encoded = encode_manifest(identity_, candidate);
std::filesystem::remove(temp_manifest, ec);
ec.clear();
if (failure_ == PublicationFailure::manifest_out_of_space) {
failure_ = PublicationFailure::none;
const auto partial_size = std::max<std::size_t>(1, encoded.size() / 2);
(void)write_synced(temp_manifest, std::span{encoded}.first(partial_size));
return std::unexpected(LocalFileError::io);
}
if (!write_synced(temp_manifest, encoded))
return std::unexpected(LocalFileError::io);
if (interruption_ == PublicationInterruption::after_manifest_sync) {
@@ -358,7 +372,8 @@ LocalWorldFiles::publish(std::span<const std::uint8_t> payload, bool journal, st
}
if (!replace_file(temp_manifest, dest_manifest))
return std::unexpected(LocalFileError::io);
if (!sync_directory(directory_)) {
if (failure_ == PublicationFailure::manifest_directory_sync || !sync_directory(directory_)) {
failure_ = PublicationFailure::none;
poisoned_ = true;
return std::unexpected(LocalFileError::io);
}
@@ -389,6 +404,14 @@ std::expected<FileCommit, LocalFileError> LocalWorldFiles::checkpoint(const Chun
: 0;
if (covered < required)
return std::unexpected(LocalFileError::stale_checkpoint);
if (manifest_ && manifest_->has_journal) {
auto committed = recover();
if (!committed)
return std::unexpected(committed.error());
const auto committed_bytes = (*committed)->snapshot(identity_);
if (!committed_bytes || !std::ranges::equal(*bytes, *committed_bytes))
return std::unexpected(LocalFileError::stale_checkpoint);
}
return publish(*bytes, false, covered, hash(*bytes));
}
+95 -2
View File
@@ -37,6 +37,18 @@ std::unique_ptr<ChunkStore> make_store() {
CHECK((*result)->load({0, 0, -2}));
return std::move(*result);
}
std::unique_ptr<ChunkStore> edited_store(std::uint32_t amount) {
auto result = make_store();
const auto view = result->view({0, 0, -2});
CHECK(view);
const std::array<CellCut, 1> cuts{{{{0, 0, -2}, {0, 0, 0}, view->revision(), amount}}};
ExcavationRequest request{1, 1, cuts, 0};
const auto digest = excavation_digest(request);
CHECK(digest);
request.digest = *digest;
CHECK(result->excavate(request));
return result;
}
std::filesystem::path world_dir(const std::filesystem::path& root) {
return root / "world-0000000000000047-000000000000005b";
}
@@ -65,6 +77,10 @@ void checkpoint_and_journal_round_trip() {
CHECK(!(*files)->publish_journal(*checkpoint, *journal));
const auto stale_checkpoint = (*files)->checkpoint(*original);
CHECK(!stale_checkpoint && stale_checkpoint.error() == LocalFileError::stale_checkpoint);
auto divergent = edited_store(3);
const auto divergent_checkpoint = (*files)->checkpoint(*divergent);
CHECK(!divergent_checkpoint &&
divergent_checkpoint.error() == LocalFileError::stale_checkpoint);
auto recovered = (*files)->recover();
CHECK(recovered);
auto expected = replay_edit_journal(*checkpoint, *journal, identity, budget);
@@ -87,18 +103,27 @@ void interruptions_preserve_commit_point() {
{PublicationInterruption::after_payload_sync, PublicationInterruption::after_manifest_sync,
PublicationInterruption::after_manifest_publish}) {
TempRoot root;
WorldBytes old_snapshot;
{
auto files = LocalWorldFiles::open(root.path, identity, budget);
CHECK(files);
auto initial = make_store();
auto snapshot = initial->snapshot(identity);
CHECK(snapshot);
old_snapshot = *snapshot;
CHECK((*files)->checkpoint(*initial));
}
auto interrupted = LocalWorldFiles::open(root.path, identity, budget, point);
CHECK(interrupted);
auto changed = make_store();
auto changed = edited_store(2);
auto new_snapshot = changed->snapshot(identity);
CHECK(new_snapshot);
const auto result = (*interrupted)->checkpoint(*changed);
CHECK(!result && result.error() == LocalFileError::interrupted);
CHECK(!(*interrupted)->checkpoint(*changed));
const auto poisoned_recovery = (*interrupted)->recover();
CHECK(!poisoned_recovery && poisoned_recovery.error() == LocalFileError::poisoned);
CHECK(!(*interrupted)->current());
(*interrupted).reset();
auto recovered = LocalWorldFiles::open(root.path, identity, budget);
CHECK(recovered);
@@ -106,9 +131,75 @@ void interruptions_preserve_commit_point() {
CHECK(commit.has_value());
const auto generation = commit.value_or(FileCommit{}).generation;
CHECK(generation == (point == PublicationInterruption::after_manifest_publish ? 2 : 1));
CHECK((*recovered)->recover());
auto recovered_state = (*recovered)->recover();
CHECK(recovered_state);
const auto& expected =
point == PublicationInterruption::after_manifest_publish ? *new_snapshot : old_snapshot;
CHECK((**recovered_state).snapshot(identity) == expected);
const auto expected_debris =
point == PublicationInterruption::after_manifest_publish ? 2U : 0U;
const auto debris = (**recovered_state).debris({1});
CHECK(debris && *debris == VolumeUnits{expected_debris});
}
}
void partial_write_failures_preserve_and_retry() {
for (const auto failure :
{PublicationFailure::payload_out_of_space, PublicationFailure::manifest_out_of_space}) {
TempRoot root;
WorldBytes old_snapshot;
{
auto files = LocalWorldFiles::open(root.path, identity, budget);
CHECK(files);
auto initial = make_store();
auto snapshot = initial->snapshot(identity);
CHECK(snapshot);
old_snapshot = *snapshot;
CHECK((*files)->checkpoint(*initial));
}
auto files = LocalWorldFiles::open(root.path, identity, budget,
PublicationInterruption::none, failure);
CHECK(files);
auto changed = edited_store(2);
auto new_snapshot = changed->snapshot(identity);
CHECK(new_snapshot);
const auto failed = (*files)->checkpoint(*changed);
CHECK(!failed && failed.error() == LocalFileError::io);
auto before_retry = (*files)->recover();
CHECK(before_retry);
CHECK((**before_retry).snapshot(identity) == old_snapshot);
auto retried = (*files)->checkpoint(*changed);
CHECK(retried);
auto after_retry = (*files)->recover();
CHECK(after_retry);
CHECK((**after_retry).snapshot(identity) == *new_snapshot);
}
}
void manifest_directory_sync_failure_requires_reopen() {
TempRoot root;
{
auto files = LocalWorldFiles::open(root.path, identity, budget);
CHECK(files);
auto initial = make_store();
CHECK((*files)->checkpoint(*initial));
}
auto files = LocalWorldFiles::open(root.path, identity, budget, PublicationInterruption::none,
PublicationFailure::manifest_directory_sync);
CHECK(files);
auto changed = edited_store(2);
auto expected = changed->snapshot(identity);
CHECK(expected);
const auto failed = (*files)->checkpoint(*changed);
CHECK(!failed && failed.error() == LocalFileError::io);
const auto poisoned = (*files)->recover();
CHECK(!poisoned && poisoned.error() == LocalFileError::poisoned);
CHECK(!(*files)->current());
(*files).reset();
auto reopened = LocalWorldFiles::open(root.path, identity, budget);
CHECK(reopened);
auto recovered = (*reopened)->recover();
CHECK(recovered);
CHECK((**recovered).snapshot(identity) == *expected);
}
void lock_and_corruption_fail_closed() {
TempRoot root;
auto first = LocalWorldFiles::open(root.path, identity, budget);
@@ -164,6 +255,8 @@ void missing_published_payload_fails_closed() {
int main() try {
checkpoint_and_journal_round_trip();
interruptions_preserve_commit_point();
partial_write_failures_preserve_and_retry();
manifest_directory_sync_failure_requires_reopen();
lock_and_corruption_fail_closed();
missing_published_payload_fails_closed();
} catch (const std::exception&) {