M12.4: qualify migration storage failures and exact recovery
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 19m42s
Foundation / checks (checks) (push) Canceled after 19m48s
Foundation / checks (tsan) (push) Canceled after 19m49s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 19m42s
Foundation / checks (checks) (push) Canceled after 19m48s
Foundation / checks (tsan) (push) Canceled after 19m49s
This commit is contained in:
@@ -1,5 +1,22 @@
|
||||
# Project status and next steps
|
||||
|
||||
## M12.4l local migration storage failures - 2026-10-18
|
||||
|
||||
The private matrix now refuses preparation, staging or paired publication after
|
||||
capsule/result directory-sync failures. Real read-permission denial after commit
|
||||
pauses current-payload recovery for retry or quarantines unavailable counterpart
|
||||
replay evidence. Both paths preserve the exact database heads and receipt; full
|
||||
recovery restores exact biology, with newer authority after quarantine.
|
||||
|
||||
GCC private population/migration checks pass in 17 s, ASan/UBSan storage/TLS
|
||||
checks in 64 s, and Clang analysis/format checks pass.
|
||||
Complete private persistence/process/backup/DR and migration/permission
|
||||
qualification, including the combined fault matrix, passes in 546 s.
|
||||
[Evidence and limits](engineering/VALIDATION.md) distinguish local sync/read
|
||||
failures from physical media, storage-host and orchestrator drills. Linked ecology,
|
||||
maturation and later M12 integration/soak gates remain open. This segment changes
|
||||
tests/documentation only.
|
||||
|
||||
## M12.4k independent migration TLS path faults - 2026-10-18
|
||||
|
||||
Test-only relays now exercise real loopback TCP/end-to-end TLS request/receipt
|
||||
|
||||
@@ -35,6 +35,12 @@ owner's actual lease expiry, replacement and healed stale fencing in both
|
||||
representations. Independent database observation checks the outcomes. See
|
||||
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
|
||||
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
|
||||
M12.4l extends local qualification with capsule/result directory-sync failures and
|
||||
actual object read-permission denial after commit. Current-payload I/O failure
|
||||
pauses complete recovery for retry; unavailable retained counterpart evidence
|
||||
quarantines. Replacement fully verifies the original pair under newer authority. See
|
||||
[storage scope](../engineering/M12_MIGRATION_STORAGE_FAULTS.md). Remote storage-host,
|
||||
physical media and orchestrator failures remain separate evidence.
|
||||
|
||||
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
|
||||
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
|
||||
|
||||
@@ -208,3 +208,17 @@ application-boundary byte loss over real local TCP/TLS, not multi-host/kernel
|
||||
networking, database/object-store loss or orchestration. Linked ecology and later
|
||||
M12 gates remain open. No SQL, production code, biological rule/format or deployment
|
||||
changes; the added relays and credential tools belong to tests/CI only.
|
||||
|
||||
## M12.4l local storage fault qualification
|
||||
|
||||
The private migration matrix now fails source/destination capsule and result
|
||||
directory syncs before publication and denies actual reads after paired commit.
|
||||
Current-result I/O failures pause verified recovery for retry under the same
|
||||
epochs. Unavailable counterpart replay evidence quarantines; restoring access
|
||||
leaves that quarantine sticky. Explicit lease retirement and replacement recover
|
||||
the exact original pair under newer authority. Independent database observations
|
||||
prove unchanged predecessor heads before publication and unchanged committed heads
|
||||
after it; exact retries never transfer stock twice. Both representations and
|
||||
outbound/return directions run these cases without changing immutable bytes.
|
||||
See [the storage scope](M12_MIGRATION_STORAGE_FAULTS.md). Physical media, remote
|
||||
storage-host and orchestrator failures remain separate evidence.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# M12.4l migration storage fault qualification
|
||||
|
||||
This extends the ADR-0147–0149 private PostgreSQL migration matrix. It changes
|
||||
tests and documentation only, with the existing ADR-0061 filesystem adapter and
|
||||
its test-only directory-sync seam. No SQL, production API, biological rule,
|
||||
format, dependency, deployment or live world changes.
|
||||
|
||||
## Matrix and command
|
||||
|
||||
```sh
|
||||
bash tools/ci/postgres-check.sh --population-only "$PWD/build/m97-postgres/evo-postgres-tests"
|
||||
```
|
||||
|
||||
The full, sanitizer and TLS qualification paths also run these cases as part of
|
||||
the population suite. Both raw/profile-4 and cohort/profile-5 cases use real
|
||||
player-origin wildlife and outbound/return transfers.
|
||||
|
||||
- Failure of the source capsule's directory sync refuses preparation without a
|
||||
pending source pointer. A subsequent ordinary preparation verifies and syncs
|
||||
the retained object before using it.
|
||||
- Failure of the destination capsule's directory sync refuses staging. The
|
||||
source remains frozen and no destination pending pointer appears.
|
||||
- Failure of either result object's directory sync refuses paired publication.
|
||||
Independent database observations prove both heads unchanged and both pending
|
||||
roles retained. A visible object alone never grants ownership; later retries
|
||||
verify and sync identical bytes again.
|
||||
- After a real paired commit, the unprivileged fixture user loses read permission
|
||||
on both current result objects (outbound) or both local counterpart predecessor
|
||||
capsules (return). Direct adapter reads must report actual I/O errors. Paused
|
||||
owners refuse advancement while independent database observation proves the
|
||||
committed heads remain unchanged. Current-payload I/O failure is retriable and
|
||||
increments the store-failure counter; unavailable paired replay evidence is a
|
||||
tail-verification failure and quarantines. Pointer settlement cannot resume
|
||||
stale source stock or activate an unverified destination.
|
||||
- A scope guard restores original permissions without changing immutable bytes.
|
||||
Temporary current-payload I/O failures reconcile the complete pair under the
|
||||
same epochs. Tail-verification quarantine remains sticky: the test coordinator
|
||||
explicitly retires the captured current database leases, then replacement hosts
|
||||
obtain strictly newer epochs and recover the exact committed payloads. Exact
|
||||
paired retries return the original receipt without a second transfer.
|
||||
|
||||
The read-permission fixture requires an unprivileged Linux user, as does the
|
||||
disposable PostgreSQL cluster. It fails if permissions do not actually deny
|
||||
reads; it does not silently skip under elevated filesystem privileges.
|
||||
|
||||
## Limits
|
||||
|
||||
Directory-sync failures use the existing adapter test seam; they are not physical
|
||||
disk failure, a power-loss durability experiment or a filesystem remount. Read
|
||||
denial is a real filesystem permission failure on selected retained objects,
|
||||
without deleting, corrupting or rewriting their contents. It does not establish
|
||||
remote object-storage availability, multi-host failover, capacity/performance or
|
||||
orchestrator recovery. Explicit lease retirement in the test is a coordinator
|
||||
action under current captured authority, not an automatic quarantine policy.
|
||||
|
||||
Multi-host/kernel-network faults, storage-host failure, orchestrator drills,
|
||||
linked ecology, maturation, world census/cuts, target-host costs and actual
|
||||
elapsed-day ecology remain open. Executed checks are recorded in
|
||||
[VALIDATION](VALIDATION.md); this document alone is not a passing result.
|
||||
@@ -25,7 +25,7 @@ or authorize production enablement.
|
||||
|---|---|---|
|
||||
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
|
||||
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
|
||||
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
|
||||
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
|
||||
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
|
||||
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
|
||||
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
|
||||
@@ -42,11 +42,13 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics.
|
||||
The engineering constitution requires stopping expansion at this architectural
|
||||
boundary and documenting the resolution. That does not prevent the independent
|
||||
worker scheduling correction or the dedicated exact population adapter above.
|
||||
Biological migration escrow is the next M12.4 boundary; none of the rows above is
|
||||
closed by adding a placeholder service or relabeling the non-biological probe.
|
||||
Durable biological migration escrow is implemented through ADR-0147–0149 below;
|
||||
none of the rows above is closed by adding a placeholder service or relabeling
|
||||
the non-biological probe.
|
||||
ADR-0144's [portable transfer candidates](M12_EXACT_BIOLOGICAL_TRANSFER.md) resolve
|
||||
the closed-owner/foreign-identity conflict with explicit flows and nonresident
|
||||
history. They do not yet supply durable paired head publication or activation.
|
||||
history. Those portable candidates alone grant no durable publication or
|
||||
activation authority; the dedicated workflow below supplies that boundary.
|
||||
Paired transfer trace 1 now supplies bounded, independently replayable input and
|
||||
member/material evidence, including raw/cohort separate-process reproduction.
|
||||
It does not grant ownership or close database escrow/retry/partition qualification.
|
||||
@@ -80,6 +82,12 @@ Independent database observation proves unchanged or atomically committed heads;
|
||||
the healthy peer keeps its epoch and retained pause. See
|
||||
[the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network
|
||||
partitions, object-store/orchestrator failures and linked ecology remain open.
|
||||
M12.4l adds local object directory-sync failures before publication and actual
|
||||
read-permission denial after commit. Current-result I/O failure pauses complete
|
||||
recovery for retry; unavailable counterpart replay evidence quarantines until
|
||||
explicit replacement. Both preserve the original receipt and heads. See
|
||||
[storage scope](M12_MIGRATION_STORAGE_FAULTS.md). This does not qualify a remote
|
||||
storage-host outage or physical media/power failure.
|
||||
The transfer tests also expose the current rules-8 maturation gap: buds retain
|
||||
their subadult realization while reproduction requires a fully realized parent.
|
||||
Finite extra food alone does not close repeated wildlife generations. A versioned
|
||||
|
||||
@@ -68,6 +68,17 @@ in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix
|
||||
Multi-host/kernel-network failures, object-store outages and orchestrator drills
|
||||
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
|
||||
|
||||
M12.4l adds source/destination capsule and result directory-sync failures, plus
|
||||
actual object read-permission denial after paired commit. Both representations
|
||||
and transfer directions must preserve the independently observed heads and
|
||||
pending roles before publication. Current-result I/O failure pauses both hosts
|
||||
for complete retry under the same epochs; unavailable counterpart replay evidence
|
||||
quarantines and restoring access cannot clear quarantine. Explicit lease retirement
|
||||
and replacement must recover exact bytes under newer epochs, and exact retry must
|
||||
return the original receipt. Runtime
|
||||
checks require real I/O errors, without changing immutable bytes or skipping
|
||||
under elevated filesystem privileges. See [storage scope](M12_MIGRATION_STORAGE_FAULTS.md).
|
||||
|
||||
Tests defend contracts, not directory coverage. All checks remain active in
|
||||
Release builds. Current proof has no actual network, biology or distributed
|
||||
recovery, so those categories below are milestone gates rather than passing
|
||||
|
||||
@@ -1,5 +1,55 @@
|
||||
# Validation report
|
||||
|
||||
## 2026-10-18 - M12.4l migration object sync/read failures
|
||||
|
||||
Test-only qualification extends ADR-0147–0149 using the existing filesystem
|
||||
adapter's directory-sync fault seam and actual read-permission denial under the
|
||||
unprivileged disposable-cluster user. Both raw and cohort representations use
|
||||
real player-origin wildlife, outbound and return transfers, unchanged complete
|
||||
payload oracles and independent database head/pointer observations. See
|
||||
[storage fixture and limits](M12_MIGRATION_STORAGE_FAULTS.md).
|
||||
|
||||
Executed locally:
|
||||
|
||||
- GCC 14 optional persistence/process targets rebuilt. The final private
|
||||
population/migration suite passed **17 s** (15 s population phase).
|
||||
- Linux Clang 19 **ASan/UBSan** storage and TLS fault matrix passed **64 s**
|
||||
(60 s population phase), with no sanitizer reports.
|
||||
- Explicit clang-tidy 19 on the changed publication tests passed with no owned
|
||||
warnings. Repository clang-format 19 passed.
|
||||
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
|
||||
**546 s**: C++ integration/process/load/history (434 s), combined TLS/storage
|
||||
fault matrix (38 s), life reservations, backup/restore, host-loss DR, upgrade,
|
||||
migration history/rollback, runtime permission audit and checksum/future-version
|
||||
rejection. The detached runner recorded its completion code.
|
||||
|
||||
Failed source capsule sync cannot prepare a freeze; failed destination capsule
|
||||
sync cannot stage a ghost. Failure of either result sync leaves both heads and
|
||||
both pending roles unchanged. Existing identical objects are reverified and
|
||||
synced on retry; visible bytes do not grant ownership.
|
||||
|
||||
Post-commit direct reads report actual I/O errors while the database retains
|
||||
both exact result heads. Current-payload I/O failure pauses complete recovery and
|
||||
retries under the same epochs after access returns. Unavailable counterpart
|
||||
replay evidence quarantines both owners; restoring access leaves quarantine
|
||||
sticky. Explicit current-lease retirement and replacement recover exact bytes
|
||||
under strictly newer epochs. The original paired receipt survives both paths.
|
||||
|
||||
The first test draft incorrectly expected current-payload I/O failure to
|
||||
quarantine. Inspection confirmed the existing retriable `RecoveryError::store`
|
||||
contract. The test now distinguishes this from unavailable paired replay evidence
|
||||
(`RecoveryError::tail`) and verifies both behaviors without changing production
|
||||
error handling or weakening readiness checks.
|
||||
|
||||
Limits: sync failure uses an existing test seam; permission denial affects
|
||||
selected local objects, not a storage-host or physical media/power failure. Test
|
||||
coordinator lease retirement is explicit, not a new automatic quarantine policy.
|
||||
No new production/thread behavior, SQL, dependency, biological rule/format,
|
||||
golden, live database, World0 or deployment changed. Native/TSan results from
|
||||
M12.4k remain separate; this test-only segment does not claim new runs of those
|
||||
checks or a Docker image build/remote CI result. Linked ecology and later M12
|
||||
integration/soak gates remain open.
|
||||
|
||||
## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions
|
||||
|
||||
Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and
|
||||
|
||||
@@ -52,6 +52,9 @@ Unknown abort that loses to commit reconciles the actual committed head.
|
||||
|
||||
M12.4k's [loopback TCP/TLS matrix](../../docs/engineering/M12_MIGRATION_TRANSPORT_FAULTS.md)
|
||||
qualifies request/receipt loss and independent owner expiry/replacement/heal.
|
||||
M12.4l's [local storage matrix](../../docs/engineering/M12_MIGRATION_STORAGE_FAULTS.md)
|
||||
checks capsule/result sync failures, paused current-payload I/O retry and sticky
|
||||
quarantine for unavailable counterpart replay evidence, with exact recovery.
|
||||
Multi-host/kernel-network failures, linked geometry/resources, world-wide cuts/census,
|
||||
target-host costs and actual elapsed-day ecology remain separate gates. Backup/DR
|
||||
regression retains its supported fixtures and does not qualify cellular history
|
||||
|
||||
@@ -98,6 +98,25 @@ struct Fixture {
|
||||
head.head.end_tick + count};
|
||||
}
|
||||
};
|
||||
// Deny actual reads by the unprivileged fixture user, then restore permissions
|
||||
// even if a runtime requirement throws. This never changes immutable bytes.
|
||||
class UnreadableObject {
|
||||
public:
|
||||
explicit UnreadableObject(std::filesystem::path path)
|
||||
: path_(std::move(path)), permissions_(std::filesystem::status(path_).permissions()) {
|
||||
std::filesystem::permissions(path_, std::filesystem::perms::none);
|
||||
}
|
||||
~UnreadableObject() {
|
||||
std::error_code error;
|
||||
std::filesystem::permissions(path_, permissions_, error);
|
||||
}
|
||||
UnreadableObject(const UnreadableObject&) = delete;
|
||||
UnreadableObject& operator=(const UnreadableObject&) = delete;
|
||||
|
||||
private:
|
||||
std::filesystem::path path_;
|
||||
std::filesystem::perms permissions_;
|
||||
};
|
||||
std::uint16_t migration_backend_port() {
|
||||
const auto port = environment("EVO_MIGRATION_TCP_PORT");
|
||||
unsigned value{};
|
||||
@@ -1072,6 +1091,25 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
{sk.region, sh.head.commit, f.id(), sh.head.digest},
|
||||
{dk.region, dh.head.commit, f.id(), dh.head.digest},
|
||||
genome};
|
||||
ObjectStoreFaults source_faults, destination_faults;
|
||||
auto failing_source =
|
||||
take(FilesystemObjectStore::open(environment("PGHOST"), sk,
|
||||
max_cellular_population_bytes, &source_faults),
|
||||
"migration source sync fault store");
|
||||
auto failing_destination = take(
|
||||
FilesystemObjectStore::open(environment("PGHOST"), dk,
|
||||
max_cellular_population_bytes, &destination_faults),
|
||||
"migration destination sync fault store");
|
||||
source_faults.fail_directory_syncs = 1;
|
||||
const auto unsynced_preparation = prepare_cellular_migration_freeze(
|
||||
*f.db, *failing_source, dst, sl, sh, dh, request, 5s);
|
||||
require(!unsynced_preparation &&
|
||||
unsynced_preparation.error().kind == PublicationError::upload &&
|
||||
unsynced_preparation.error().object == ObjectError::io &&
|
||||
source_faults.fail_directory_syncs == 0 &&
|
||||
!take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s),
|
||||
"unsynced preparation pointer"),
|
||||
"failed capsule sync durably prepared a migration");
|
||||
take(source_host->prepare_migration_freeze(dh, dst, request),
|
||||
"owning paired source freeze");
|
||||
const auto frozen = take(take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s),
|
||||
@@ -1081,7 +1119,41 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
require(!publish_cellular_migration(*f.db, so, dst, sl, dl, sh, dh, frozen.trace,
|
||||
f.build, 5s),
|
||||
"unstaged migration published");
|
||||
destination_faults.fail_directory_syncs = 1;
|
||||
const auto unsynced_stage = stage_cellular_migration(*f.db, so, *failing_destination,
|
||||
dl, sh, dh, frozen.trace, 5s);
|
||||
require(!unsynced_stage && unsynced_stage.error().kind == PublicationError::upload &&
|
||||
unsynced_stage.error().object == ObjectError::io &&
|
||||
destination_faults.fail_directory_syncs == 0 &&
|
||||
!take(load_cellular_migration_stage(*f.db, dst, dk, dl, 5s),
|
||||
"unsynced destination stage pointer") &&
|
||||
take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s),
|
||||
"source freeze after stage sync failure")
|
||||
.has_value(),
|
||||
"failed stage capsule sync activated a ghost or released source ownership");
|
||||
take(destination_host->stage_migration(sh, so, frozen.trace), "owning paired stage");
|
||||
for (unsigned endpoint = 0; endpoint < 2; ++endpoint) {
|
||||
auto& faults = endpoint == 0 ? source_faults : destination_faults;
|
||||
faults.fail_directory_syncs = 1;
|
||||
const auto unsynced_pair =
|
||||
publish_cellular_migration(*f.db, *failing_source, *failing_destination, sl, dl,
|
||||
sh, dh, frozen.trace, f.build, 5s);
|
||||
require(
|
||||
!unsynced_pair && unsynced_pair.error().kind == PublicationError::upload &&
|
||||
unsynced_pair.error().object == ObjectError::io &&
|
||||
faults.fail_directory_syncs == 0 &&
|
||||
take(load_region_checkpoint(*f.db, sk, 5s), "unsynced source head").head ==
|
||||
source_before.head &&
|
||||
take(load_region_checkpoint(*f.db, dk, 5s), "unsynced destination head")
|
||||
.head == destination_before.head &&
|
||||
take(load_cellular_migration_freeze(*f.db, so, sk, sl, 5s),
|
||||
"unsynced source freeze")
|
||||
.has_value() &&
|
||||
take(load_cellular_migration_stage(*f.db, dst, dk, dl, 5s),
|
||||
"unsynced destination stage")
|
||||
.has_value(),
|
||||
"failed result sync published a half or released retained work");
|
||||
}
|
||||
std::vector<std::uint8_t> expected_source, expected_destination;
|
||||
if (representation == 0) {
|
||||
auto a = take(restore_cellular_payload(sb, s), "paired raw source oracle");
|
||||
@@ -1234,6 +1306,63 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
require(!conflict && conflict.error().store.kind == LeaseStoreError::commit_conflict &&
|
||||
!retry_cellular_migration_commit(*f.db, stale, dl, prepared, 5s),
|
||||
"changed pair or stale source authority was accepted");
|
||||
const auto sm = take(decode_manifest(sh.manifest), "unavailable source result");
|
||||
const auto dm = take(decode_manifest(dh.manifest), "unavailable destination result");
|
||||
// Outbound: deny current results. Return: deny retained counterpart
|
||||
// capsules. Both are required before either paused host may resume.
|
||||
const VerifiedObject unavailable_source =
|
||||
direction == 0 ? VerifiedObject{sm.payload_sha256, sm.payload_bytes}
|
||||
: VerifiedObject{d.payload_sha256, d.payload_bytes};
|
||||
const VerifiedObject unavailable_destination =
|
||||
direction == 0 ? VerifiedObject{dm.payload_sha256, dm.payload_bytes}
|
||||
: VerifiedObject{s.payload_sha256, s.payload_bytes};
|
||||
{
|
||||
UnreadableObject source_denial(so.directory() /
|
||||
hexadecimal(unavailable_source.sha256, false));
|
||||
UnreadableObject destination_denial(
|
||||
dst.directory() / hexadecimal(unavailable_destination.sha256, false));
|
||||
const auto denied_source = so.get(unavailable_source);
|
||||
const auto denied_destination = dst.get(unavailable_destination);
|
||||
require(!denied_source && denied_source.error() == ObjectError::io &&
|
||||
!denied_destination && denied_destination.error() == ObjectError::io,
|
||||
"object read denial did not produce actual I/O failures");
|
||||
const auto source_failures = source_host->status().store_failures;
|
||||
const auto destination_failures = destination_host->status().store_failures;
|
||||
const auto observed = [&] {
|
||||
if (direction == 0)
|
||||
return source_host->status().store_failures > source_failures &&
|
||||
destination_host->status().store_failures > destination_failures;
|
||||
return source_host->status().phase == HostPhase::quarantined &&
|
||||
destination_host->status().phase == HostPhase::quarantined;
|
||||
};
|
||||
for (unsigned attempt = 0; attempt < 200 && !observed(); ++attempt) {
|
||||
source_host->step();
|
||||
destination_host->step();
|
||||
std::this_thread::sleep_for(10ms);
|
||||
}
|
||||
require(
|
||||
observed() && !source_host->status().ready &&
|
||||
!destination_host->status().ready && !source_host->advance_unattended(1) &&
|
||||
!destination_host->advance_unattended(1) &&
|
||||
take(load_region_checkpoint(*f.db, sk, 5s), "unavailable source head")
|
||||
.head == sh.head &&
|
||||
take(load_region_checkpoint(*f.db, dk, 5s), "unavailable destination head")
|
||||
.head == dh.head,
|
||||
"unreadable migration evidence resumed old ownership or changed a head");
|
||||
}
|
||||
if (direction != 0) {
|
||||
require(!source_host->step().ready && !destination_host->step().ready,
|
||||
"restoring storage silently cleared quarantine");
|
||||
source_host->drain();
|
||||
destination_host->drain();
|
||||
// The quarantined local authorities cannot release themselves. This
|
||||
// test coordinator explicitly retires their still-current DB leases.
|
||||
take(release_region_lease(*f.db, sk, sl, 5s), "retire quarantined source lease");
|
||||
take(release_region_lease(*f.db, dk, dl, 5s),
|
||||
"retire quarantined destination lease");
|
||||
source_host = make_host(sk);
|
||||
destination_host = make_host(dk);
|
||||
}
|
||||
for (unsigned attempt = 0;
|
||||
attempt < 200 && (!source_host->step().ready || !destination_host->step().ready);
|
||||
++attempt)
|
||||
@@ -1242,6 +1371,17 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
take(source_host->committed(), "adopted source").head == sh.head &&
|
||||
take(destination_host->committed(), "adopted destination").head == dh.head,
|
||||
"pointer release resumed a stale owner instead of recovering the pair");
|
||||
const auto old_source_epoch = sl.epoch, old_destination_epoch = dl.epoch;
|
||||
sl = take(source_host->authority(), "restored source object authority");
|
||||
dl = take(destination_host->authority(), "restored destination object authority");
|
||||
const bool expected_epochs =
|
||||
direction == 0 ? sl.epoch == old_source_epoch && dl.epoch == old_destination_epoch
|
||||
: sl.epoch > old_source_epoch && dl.epoch > old_destination_epoch;
|
||||
require(expected_epochs &&
|
||||
take(retry_cellular_migration_commit(*f.db, sl, dl, prepared, 5s),
|
||||
"restored storage exact paired receipt")
|
||||
.source.head == sh.head,
|
||||
"storage recovery changed a committed pair or reused authority");
|
||||
const auto check_bytes = [&](const CommittedCheckpoint& h, RegionKey key,
|
||||
const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease) {
|
||||
@@ -1290,6 +1430,7 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
migration_owner_partitions(f, destination_key, destination_objects, destination_lease,
|
||||
destination_head, genome);
|
||||
}
|
||||
std::puts("migration object sync/read failures, quarantine and exact recovery passed");
|
||||
// Race the actual two-head transaction against abort on independent connections.
|
||||
const auto prepare_pair = [&](bool reverse) {
|
||||
auto& sh = reverse ? destination_head : f.head;
|
||||
|
||||
Reference in New Issue
Block a user