diff --git a/docs/STATUS.md b/docs/STATUS.md index 3ed90a5..06af653 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1,5 +1,22 @@ # Project status and next steps +## M12.4l local migration storage failures - 2026-10-18 + +The private matrix now refuses preparation, staging or paired publication after +capsule/result directory-sync failures. Real read-permission denial after commit +pauses current-payload recovery for retry or quarantines unavailable counterpart +replay evidence. Both paths preserve the exact database heads and receipt; full +recovery restores exact biology, with newer authority after quarantine. + +GCC private population/migration checks pass in 17 s, ASan/UBSan storage/TLS +checks in 64 s, and Clang analysis/format checks pass. +Complete private persistence/process/backup/DR and migration/permission +qualification, including the combined fault matrix, passes in 546 s. +[Evidence and limits](engineering/VALIDATION.md) distinguish local sync/read +failures from physical media, storage-host and orchestrator drills. Linked ecology, +maturation and later M12 integration/soak gates remain open. This segment changes +tests/documentation only. + ## M12.4k independent migration TLS path faults - 2026-10-18 Test-only relays now exercise real loopback TCP/end-to-end TLS request/receipt diff --git a/docs/architecture/PERSISTENCE_ARCHITECTURE.md b/docs/architecture/PERSISTENCE_ARCHITECTURE.md index 17dd8a5..191c194 100644 --- a/docs/architecture/PERSISTENCE_ARCHITECTURE.md +++ b/docs/architecture/PERSISTENCE_ARCHITECTURE.md @@ -35,6 +35,12 @@ owner's actual lease expiry, replacement and healed stale fencing in both representations. Independent database observation checks the outcomes. See [transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host network/kernel, object-store/orchestrator failures and linked ecology remain separate. +M12.4l extends local qualification with capsule/result directory-sync failures and +actual object read-permission denial after commit. Current-payload I/O failure +pauses complete recovery for retry; unavailable retained counterpart evidence +quarantines. Replacement fully verifies the original pair under newer authority. See +[storage scope](../engineering/M12_MIGRATION_STORAGE_FAULTS.md). Remote storage-host, +physical media and orchestrator failures remain separate evidence. M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md) and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not diff --git a/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md b/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md index 8a6fd0a..409a03b 100644 --- a/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md +++ b/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md @@ -208,3 +208,17 @@ application-boundary byte loss over real local TCP/TLS, not multi-host/kernel networking, database/object-store loss or orchestration. Linked ecology and later M12 gates remain open. No SQL, production code, biological rule/format or deployment changes; the added relays and credential tools belong to tests/CI only. + +## M12.4l local storage fault qualification + +The private migration matrix now fails source/destination capsule and result +directory syncs before publication and denies actual reads after paired commit. +Current-result I/O failures pause verified recovery for retry under the same +epochs. Unavailable counterpart replay evidence quarantines; restoring access +leaves that quarantine sticky. Explicit lease retirement and replacement recover +the exact original pair under newer authority. Independent database observations +prove unchanged predecessor heads before publication and unchanged committed heads +after it; exact retries never transfer stock twice. Both representations and +outbound/return directions run these cases without changing immutable bytes. +See [the storage scope](M12_MIGRATION_STORAGE_FAULTS.md). Physical media, remote +storage-host and orchestrator failures remain separate evidence. diff --git a/docs/engineering/M12_MIGRATION_STORAGE_FAULTS.md b/docs/engineering/M12_MIGRATION_STORAGE_FAULTS.md new file mode 100644 index 0000000..a4ae575 --- /dev/null +++ b/docs/engineering/M12_MIGRATION_STORAGE_FAULTS.md @@ -0,0 +1,59 @@ +# M12.4l migration storage fault qualification + +This extends the ADR-0147–0149 private PostgreSQL migration matrix. It changes +tests and documentation only, with the existing ADR-0061 filesystem adapter and +its test-only directory-sync seam. No SQL, production API, biological rule, +format, dependency, deployment or live world changes. + +## Matrix and command + +```sh +bash tools/ci/postgres-check.sh --population-only "$PWD/build/m97-postgres/evo-postgres-tests" +``` + +The full, sanitizer and TLS qualification paths also run these cases as part of +the population suite. Both raw/profile-4 and cohort/profile-5 cases use real +player-origin wildlife and outbound/return transfers. + +- Failure of the source capsule's directory sync refuses preparation without a + pending source pointer. A subsequent ordinary preparation verifies and syncs + the retained object before using it. +- Failure of the destination capsule's directory sync refuses staging. The + source remains frozen and no destination pending pointer appears. +- Failure of either result object's directory sync refuses paired publication. + Independent database observations prove both heads unchanged and both pending + roles retained. A visible object alone never grants ownership; later retries + verify and sync identical bytes again. +- After a real paired commit, the unprivileged fixture user loses read permission + on both current result objects (outbound) or both local counterpart predecessor + capsules (return). Direct adapter reads must report actual I/O errors. Paused + owners refuse advancement while independent database observation proves the + committed heads remain unchanged. Current-payload I/O failure is retriable and + increments the store-failure counter; unavailable paired replay evidence is a + tail-verification failure and quarantines. Pointer settlement cannot resume + stale source stock or activate an unverified destination. +- A scope guard restores original permissions without changing immutable bytes. + Temporary current-payload I/O failures reconcile the complete pair under the + same epochs. Tail-verification quarantine remains sticky: the test coordinator + explicitly retires the captured current database leases, then replacement hosts + obtain strictly newer epochs and recover the exact committed payloads. Exact + paired retries return the original receipt without a second transfer. + +The read-permission fixture requires an unprivileged Linux user, as does the +disposable PostgreSQL cluster. It fails if permissions do not actually deny +reads; it does not silently skip under elevated filesystem privileges. + +## Limits + +Directory-sync failures use the existing adapter test seam; they are not physical +disk failure, a power-loss durability experiment or a filesystem remount. Read +denial is a real filesystem permission failure on selected retained objects, +without deleting, corrupting or rewriting their contents. It does not establish +remote object-storage availability, multi-host failover, capacity/performance or +orchestrator recovery. Explicit lease retirement in the test is a coordinator +action under current captured authority, not an automatic quarantine policy. + +Multi-host/kernel-network faults, storage-host failure, orchestrator drills, +linked ecology, maturation, world census/cuts, target-host costs and actual +elapsed-day ecology remain open. Executed checks are recorded in +[VALIDATION](VALIDATION.md); this document alone is not a passing result. diff --git a/docs/engineering/M12_REMAINING_GATES.md b/docs/engineering/M12_REMAINING_GATES.md index c61076f..2f4336e 100644 --- a/docs/engineering/M12_REMAINING_GATES.md +++ b/docs/engineering/M12_REMAINING_GATES.md @@ -25,7 +25,7 @@ or authorize production enablement. |---|---|---| | 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. | | 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. | -| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. | +| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. | | 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. | | 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. | | 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. | @@ -42,11 +42,13 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics. The engineering constitution requires stopping expansion at this architectural boundary and documenting the resolution. That does not prevent the independent worker scheduling correction or the dedicated exact population adapter above. -Biological migration escrow is the next M12.4 boundary; none of the rows above is -closed by adding a placeholder service or relabeling the non-biological probe. +Durable biological migration escrow is implemented through ADR-0147–0149 below; +none of the rows above is closed by adding a placeholder service or relabeling +the non-biological probe. ADR-0144's [portable transfer candidates](M12_EXACT_BIOLOGICAL_TRANSFER.md) resolve the closed-owner/foreign-identity conflict with explicit flows and nonresident -history. They do not yet supply durable paired head publication or activation. +history. Those portable candidates alone grant no durable publication or +activation authority; the dedicated workflow below supplies that boundary. Paired transfer trace 1 now supplies bounded, independently replayable input and member/material evidence, including raw/cohort separate-process reproduction. It does not grant ownership or close database escrow/retry/partition qualification. @@ -80,6 +82,12 @@ Independent database observation proves unchanged or atomically committed heads; the healthy peer keeps its epoch and retained pause. See [the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network partitions, object-store/orchestrator failures and linked ecology remain open. +M12.4l adds local object directory-sync failures before publication and actual +read-permission denial after commit. Current-result I/O failure pauses complete +recovery for retry; unavailable counterpart replay evidence quarantines until +explicit replacement. Both preserve the original receipt and heads. See +[storage scope](M12_MIGRATION_STORAGE_FAULTS.md). This does not qualify a remote +storage-host outage or physical media/power failure. The transfer tests also expose the current rules-8 maturation gap: buds retain their subadult realization while reproduction requires a fully realized parent. Finite extra food alone does not close repeated wildlife generations. A versioned diff --git a/docs/engineering/TEST_STRATEGY.md b/docs/engineering/TEST_STRATEGY.md index 69dd2a5..644fdcd 100644 --- a/docs/engineering/TEST_STRATEGY.md +++ b/docs/engineering/TEST_STRATEGY.md @@ -68,6 +68,17 @@ in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix Multi-host/kernel-network failures, object-store outages and orchestrator drills remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md). +M12.4l adds source/destination capsule and result directory-sync failures, plus +actual object read-permission denial after paired commit. Both representations +and transfer directions must preserve the independently observed heads and +pending roles before publication. Current-result I/O failure pauses both hosts +for complete retry under the same epochs; unavailable counterpart replay evidence +quarantines and restoring access cannot clear quarantine. Explicit lease retirement +and replacement must recover exact bytes under newer epochs, and exact retry must +return the original receipt. Runtime +checks require real I/O errors, without changing immutable bytes or skipping +under elevated filesystem privileges. See [storage scope](M12_MIGRATION_STORAGE_FAULTS.md). + Tests defend contracts, not directory coverage. All checks remain active in Release builds. Current proof has no actual network, biology or distributed recovery, so those categories below are milestone gates rather than passing diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 75043ab..c3df943 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -1,5 +1,55 @@ # Validation report +## 2026-10-18 - M12.4l migration object sync/read failures + +Test-only qualification extends ADR-0147–0149 using the existing filesystem +adapter's directory-sync fault seam and actual read-permission denial under the +unprivileged disposable-cluster user. Both raw and cohort representations use +real player-origin wildlife, outbound and return transfers, unchanged complete +payload oracles and independent database head/pointer observations. See +[storage fixture and limits](M12_MIGRATION_STORAGE_FAULTS.md). + +Executed locally: + +- GCC 14 optional persistence/process targets rebuilt. The final private + population/migration suite passed **17 s** (15 s population phase). +- Linux Clang 19 **ASan/UBSan** storage and TLS fault matrix passed **64 s** + (60 s population phase), with no sanitizer reports. +- Explicit clang-tidy 19 on the changed publication tests passed with no owned + warnings. Repository clang-format 19 passed. +- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**, + **546 s**: C++ integration/process/load/history (434 s), combined TLS/storage + fault matrix (38 s), life reservations, backup/restore, host-loss DR, upgrade, + migration history/rollback, runtime permission audit and checksum/future-version + rejection. The detached runner recorded its completion code. + +Failed source capsule sync cannot prepare a freeze; failed destination capsule +sync cannot stage a ghost. Failure of either result sync leaves both heads and +both pending roles unchanged. Existing identical objects are reverified and +synced on retry; visible bytes do not grant ownership. + +Post-commit direct reads report actual I/O errors while the database retains +both exact result heads. Current-payload I/O failure pauses complete recovery and +retries under the same epochs after access returns. Unavailable counterpart +replay evidence quarantines both owners; restoring access leaves quarantine +sticky. Explicit current-lease retirement and replacement recover exact bytes +under strictly newer epochs. The original paired receipt survives both paths. + +The first test draft incorrectly expected current-payload I/O failure to +quarantine. Inspection confirmed the existing retriable `RecoveryError::store` +contract. The test now distinguishes this from unavailable paired replay evidence +(`RecoveryError::tail`) and verifies both behaviors without changing production +error handling or weakening readiness checks. + +Limits: sync failure uses an existing test seam; permission denial affects +selected local objects, not a storage-host or physical media/power failure. Test +coordinator lease retirement is explicit, not a new automatic quarantine policy. +No new production/thread behavior, SQL, dependency, biological rule/format, +golden, live database, World0 or deployment changed. Native/TSan results from +M12.4k remain separate; this test-only segment does not claim new runs of those +checks or a Docker image build/remote CI result. Linked ecology and later M12 +integration/soak gates remain open. + ## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and diff --git a/schemas/ecology/cellular-migration-commit-v1.md b/schemas/ecology/cellular-migration-commit-v1.md index d405222..7a84d65 100644 --- a/schemas/ecology/cellular-migration-commit-v1.md +++ b/schemas/ecology/cellular-migration-commit-v1.md @@ -52,6 +52,9 @@ Unknown abort that loses to commit reconciles the actual committed head. M12.4k's [loopback TCP/TLS matrix](../../docs/engineering/M12_MIGRATION_TRANSPORT_FAULTS.md) qualifies request/receipt loss and independent owner expiry/replacement/heal. +M12.4l's [local storage matrix](../../docs/engineering/M12_MIGRATION_STORAGE_FAULTS.md) +checks capsule/result sync failures, paused current-payload I/O retry and sticky +quarantine for unavailable counterpart replay evidence, with exact recovery. Multi-host/kernel-network failures, linked geometry/resources, world-wide cuts/census, target-host costs and actual elapsed-day ecology remain separate gates. Backup/DR regression retains its supported fixtures and does not qualify cellular history diff --git a/tests/population_publication_tests.cpp b/tests/population_publication_tests.cpp index e8a634f..29a26ee 100644 --- a/tests/population_publication_tests.cpp +++ b/tests/population_publication_tests.cpp @@ -98,6 +98,25 @@ struct Fixture { head.head.end_tick + count}; } }; +// Deny actual reads by the unprivileged fixture user, then restore permissions +// even if a runtime requirement throws. This never changes immutable bytes. +class UnreadableObject { + public: + explicit UnreadableObject(std::filesystem::path path) + : path_(std::move(path)), permissions_(std::filesystem::status(path_).permissions()) { + std::filesystem::permissions(path_, std::filesystem::perms::none); + } + ~UnreadableObject() { + std::error_code error; + std::filesystem::permissions(path_, permissions_, error); + } + UnreadableObject(const UnreadableObject&) = delete; + UnreadableObject& operator=(const UnreadableObject&) = delete; + + private: + std::filesystem::path path_; + std::filesystem::perms permissions_; +}; std::uint16_t migration_backend_port() { const auto port = environment("EVO_MIGRATION_TCP_PORT"); unsigned value{}; @@ -1072,6 +1091,25 @@ void migration_publication(Fixture& f, const RegionKey destination_key, {sk.region, sh.head.commit, f.id(), sh.head.digest}, {dk.region, dh.head.commit, f.id(), dh.head.digest}, genome}; + ObjectStoreFaults source_faults, destination_faults; + auto failing_source = + take(FilesystemObjectStore::open(environment("PGHOST"), sk, + max_cellular_population_bytes, &source_faults), + "migration source sync fault store"); + auto failing_destination = take( + FilesystemObjectStore::open(environment("PGHOST"), dk, + max_cellular_population_bytes, &destination_faults), + "migration destination sync fault store"); + source_faults.fail_directory_syncs = 1; + const auto unsynced_preparation = prepare_cellular_migration_freeze( + *f.db, *failing_source, dst, sl, sh, dh, request, 5s); + require(!unsynced_preparation && + unsynced_preparation.error().kind == PublicationError::upload && + unsynced_preparation.error().object == ObjectError::io && + source_faults.fail_directory_syncs == 0 && + !take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s), + "unsynced preparation pointer"), + "failed capsule sync durably prepared a migration"); take(source_host->prepare_migration_freeze(dh, dst, request), "owning paired source freeze"); const auto frozen = take(take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s), @@ -1081,7 +1119,41 @@ void migration_publication(Fixture& f, const RegionKey destination_key, require(!publish_cellular_migration(*f.db, so, dst, sl, dl, sh, dh, frozen.trace, f.build, 5s), "unstaged migration published"); + destination_faults.fail_directory_syncs = 1; + const auto unsynced_stage = stage_cellular_migration(*f.db, so, *failing_destination, + dl, sh, dh, frozen.trace, 5s); + require(!unsynced_stage && unsynced_stage.error().kind == PublicationError::upload && + unsynced_stage.error().object == ObjectError::io && + destination_faults.fail_directory_syncs == 0 && + !take(load_cellular_migration_stage(*f.db, dst, dk, dl, 5s), + "unsynced destination stage pointer") && + take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s), + "source freeze after stage sync failure") + .has_value(), + "failed stage capsule sync activated a ghost or released source ownership"); take(destination_host->stage_migration(sh, so, frozen.trace), "owning paired stage"); + for (unsigned endpoint = 0; endpoint < 2; ++endpoint) { + auto& faults = endpoint == 0 ? source_faults : destination_faults; + faults.fail_directory_syncs = 1; + const auto unsynced_pair = + publish_cellular_migration(*f.db, *failing_source, *failing_destination, sl, dl, + sh, dh, frozen.trace, f.build, 5s); + require( + !unsynced_pair && unsynced_pair.error().kind == PublicationError::upload && + unsynced_pair.error().object == ObjectError::io && + faults.fail_directory_syncs == 0 && + take(load_region_checkpoint(*f.db, sk, 5s), "unsynced source head").head == + source_before.head && + take(load_region_checkpoint(*f.db, dk, 5s), "unsynced destination head") + .head == destination_before.head && + take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s), + "unsynced source freeze") + .has_value() && + take(load_cellular_migration_stage(*f.db, dst, dk, dl, 5s), + "unsynced destination stage") + .has_value(), + "failed result sync published a half or released retained work"); + } std::vector expected_source, expected_destination; if (representation == 0) { auto a = take(restore_cellular_payload(sb, s), "paired raw source oracle"); @@ -1234,6 +1306,63 @@ void migration_publication(Fixture& f, const RegionKey destination_key, require(!conflict && conflict.error().store.kind == LeaseStoreError::commit_conflict && !retry_cellular_migration_commit(*f.db, stale, dl, prepared, 5s), "changed pair or stale source authority was accepted"); + const auto sm = take(decode_manifest(sh.manifest), "unavailable source result"); + const auto dm = take(decode_manifest(dh.manifest), "unavailable destination result"); + // Outbound: deny current results. Return: deny retained counterpart + // capsules. Both are required before either paused host may resume. + const VerifiedObject unavailable_source = + direction == 0 ? VerifiedObject{sm.payload_sha256, sm.payload_bytes} + : VerifiedObject{d.payload_sha256, d.payload_bytes}; + const VerifiedObject unavailable_destination = + direction == 0 ? VerifiedObject{dm.payload_sha256, dm.payload_bytes} + : VerifiedObject{s.payload_sha256, s.payload_bytes}; + { + UnreadableObject source_denial(so.directory() / + hexadecimal(unavailable_source.sha256, false)); + UnreadableObject destination_denial( + dst.directory() / hexadecimal(unavailable_destination.sha256, false)); + const auto denied_source = so.get(unavailable_source); + const auto denied_destination = dst.get(unavailable_destination); + require(!denied_source && denied_source.error() == ObjectError::io && + !denied_destination && denied_destination.error() == ObjectError::io, + "object read denial did not produce actual I/O failures"); + const auto source_failures = source_host->status().store_failures; + const auto destination_failures = destination_host->status().store_failures; + const auto observed = [&] { + if (direction == 0) + return source_host->status().store_failures > source_failures && + destination_host->status().store_failures > destination_failures; + return source_host->status().phase == HostPhase::quarantined && + destination_host->status().phase == HostPhase::quarantined; + }; + for (unsigned attempt = 0; attempt < 200 && !observed(); ++attempt) { + source_host->step(); + destination_host->step(); + std::this_thread::sleep_for(10ms); + } + require( + observed() && !source_host->status().ready && + !destination_host->status().ready && !source_host->advance_unattended(1) && + !destination_host->advance_unattended(1) && + take(load_region_checkpoint(*f.db, sk, 5s), "unavailable source head") + .head == sh.head && + take(load_region_checkpoint(*f.db, dk, 5s), "unavailable destination head") + .head == dh.head, + "unreadable migration evidence resumed old ownership or changed a head"); + } + if (direction != 0) { + require(!source_host->step().ready && !destination_host->step().ready, + "restoring storage silently cleared quarantine"); + source_host->drain(); + destination_host->drain(); + // The quarantined local authorities cannot release themselves. This + // test coordinator explicitly retires their still-current DB leases. + take(release_region_lease(*f.db, sk, sl, 5s), "retire quarantined source lease"); + take(release_region_lease(*f.db, dk, dl, 5s), + "retire quarantined destination lease"); + source_host = make_host(sk); + destination_host = make_host(dk); + } for (unsigned attempt = 0; attempt < 200 && (!source_host->step().ready || !destination_host->step().ready); ++attempt) @@ -1242,6 +1371,17 @@ void migration_publication(Fixture& f, const RegionKey destination_key, take(source_host->committed(), "adopted source").head == sh.head && take(destination_host->committed(), "adopted destination").head == dh.head, "pointer release resumed a stale owner instead of recovering the pair"); + const auto old_source_epoch = sl.epoch, old_destination_epoch = dl.epoch; + sl = take(source_host->authority(), "restored source object authority"); + dl = take(destination_host->authority(), "restored destination object authority"); + const bool expected_epochs = + direction == 0 ? sl.epoch == old_source_epoch && dl.epoch == old_destination_epoch + : sl.epoch > old_source_epoch && dl.epoch > old_destination_epoch; + require(expected_epochs && + take(retry_cellular_migration_commit(*f.db, sl, dl, prepared, 5s), + "restored storage exact paired receipt") + .source.head == sh.head, + "storage recovery changed a committed pair or reused authority"); const auto check_bytes = [&](const CommittedCheckpoint& h, RegionKey key, const FilesystemObjectStore& objects, const LeaseGrant& lease) { @@ -1290,6 +1430,7 @@ void migration_publication(Fixture& f, const RegionKey destination_key, migration_owner_partitions(f, destination_key, destination_objects, destination_lease, destination_head, genome); } + std::puts("migration object sync/read failures, quarantine and exact recovery passed"); // Race the actual two-head transaction against abort on independent connections. const auto prepare_pair = [&](bool reverse) { auto& sh = reverse ? destination_head : f.head;