M12.4: qualify migration storage failures and exact recovery
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 19m42s
Foundation / checks (checks) (push) Canceled after 19m48s
Foundation / checks (tsan) (push) Canceled after 19m49s

This commit is contained in:
dingo
2026-10-18 13:38:05 -04:00
parent e0a7c13f6b
commit df680810af
9 changed files with 313 additions and 4 deletions
@@ -208,3 +208,17 @@ application-boundary byte loss over real local TCP/TLS, not multi-host/kernel
networking, database/object-store loss or orchestration. Linked ecology and later
M12 gates remain open. No SQL, production code, biological rule/format or deployment
changes; the added relays and credential tools belong to tests/CI only.
## M12.4l local storage fault qualification
The private migration matrix now fails source/destination capsule and result
directory syncs before publication and denies actual reads after paired commit.
Current-result I/O failures pause verified recovery for retry under the same
epochs. Unavailable counterpart replay evidence quarantines; restoring access
leaves that quarantine sticky. Explicit lease retirement and replacement recover
the exact original pair under newer authority. Independent database observations
prove unchanged predecessor heads before publication and unchanged committed heads
after it; exact retries never transfer stock twice. Both representations and
outbound/return directions run these cases without changing immutable bytes.
See [the storage scope](M12_MIGRATION_STORAGE_FAULTS.md). Physical media, remote
storage-host and orchestrator failures remain separate evidence.
@@ -0,0 +1,59 @@
# M12.4l migration storage fault qualification
This extends the ADR-0147–0149 private PostgreSQL migration matrix. It changes
tests and documentation only, with the existing ADR-0061 filesystem adapter and
its test-only directory-sync seam. No SQL, production API, biological rule,
format, dependency, deployment or live world changes.
## Matrix and command
```sh
bash tools/ci/postgres-check.sh --population-only "$PWD/build/m97-postgres/evo-postgres-tests"
```
The full, sanitizer and TLS qualification paths also run these cases as part of
the population suite. Both raw/profile-4 and cohort/profile-5 cases use real
player-origin wildlife and outbound/return transfers.
- Failure of the source capsule's directory sync refuses preparation without a
pending source pointer. A subsequent ordinary preparation verifies and syncs
the retained object before using it.
- Failure of the destination capsule's directory sync refuses staging. The
source remains frozen and no destination pending pointer appears.
- Failure of either result object's directory sync refuses paired publication.
Independent database observations prove both heads unchanged and both pending
roles retained. A visible object alone never grants ownership; later retries
verify and sync identical bytes again.
- After a real paired commit, the unprivileged fixture user loses read permission
on both current result objects (outbound) or both local counterpart predecessor
capsules (return). Direct adapter reads must report actual I/O errors. Paused
owners refuse advancement while independent database observation proves the
committed heads remain unchanged. Current-payload I/O failure is retriable and
increments the store-failure counter; unavailable paired replay evidence is a
tail-verification failure and quarantines. Pointer settlement cannot resume
stale source stock or activate an unverified destination.
- A scope guard restores original permissions without changing immutable bytes.
Temporary current-payload I/O failures reconcile the complete pair under the
same epochs. Tail-verification quarantine remains sticky: the test coordinator
explicitly retires the captured current database leases, then replacement hosts
obtain strictly newer epochs and recover the exact committed payloads. Exact
paired retries return the original receipt without a second transfer.
The read-permission fixture requires an unprivileged Linux user, as does the
disposable PostgreSQL cluster. It fails if permissions do not actually deny
reads; it does not silently skip under elevated filesystem privileges.
## Limits
Directory-sync failures use the existing adapter test seam; they are not physical
disk failure, a power-loss durability experiment or a filesystem remount. Read
denial is a real filesystem permission failure on selected retained objects,
without deleting, corrupting or rewriting their contents. It does not establish
remote object-storage availability, multi-host failover, capacity/performance or
orchestrator recovery. Explicit lease retirement in the test is a coordinator
action under current captured authority, not an automatic quarantine policy.
Multi-host/kernel-network faults, storage-host failure, orchestrator drills,
linked ecology, maturation, world census/cuts, target-host costs and actual
elapsed-day ecology remain open. Executed checks are recorded in
[VALIDATION](VALIDATION.md); this document alone is not a passing result.
+12 -4
View File
@@ -25,7 +25,7 @@ or authorize production enablement.
|---|---|---|
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
@@ -42,11 +42,13 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics.
The engineering constitution requires stopping expansion at this architectural
boundary and documenting the resolution. That does not prevent the independent
worker scheduling correction or the dedicated exact population adapter above.
Biological migration escrow is the next M12.4 boundary; none of the rows above is
closed by adding a placeholder service or relabeling the non-biological probe.
Durable biological migration escrow is implemented through ADR-0147–0149 below;
none of the rows above is closed by adding a placeholder service or relabeling
the non-biological probe.
ADR-0144's [portable transfer candidates](M12_EXACT_BIOLOGICAL_TRANSFER.md) resolve
the closed-owner/foreign-identity conflict with explicit flows and nonresident
history. They do not yet supply durable paired head publication or activation.
history. Those portable candidates alone grant no durable publication or
activation authority; the dedicated workflow below supplies that boundary.
Paired transfer trace 1 now supplies bounded, independently replayable input and
member/material evidence, including raw/cohort separate-process reproduction.
It does not grant ownership or close database escrow/retry/partition qualification.
@@ -80,6 +82,12 @@ Independent database observation proves unchanged or atomically committed heads;
the healthy peer keeps its epoch and retained pause. See
[the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network
partitions, object-store/orchestrator failures and linked ecology remain open.
M12.4l adds local object directory-sync failures before publication and actual
read-permission denial after commit. Current-result I/O failure pauses complete
recovery for retry; unavailable counterpart replay evidence quarantines until
explicit replacement. Both preserve the original receipt and heads. See
[storage scope](M12_MIGRATION_STORAGE_FAULTS.md). This does not qualify a remote
storage-host outage or physical media/power failure.
The transfer tests also expose the current rules-8 maturation gap: buds retain
their subadult realization while reproduction requires a fully realized parent.
Finite extra food alone does not close repeated wildlife generations. A versioned
+11
View File
@@ -68,6 +68,17 @@ in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix
Multi-host/kernel-network failures, object-store outages and orchestrator drills
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
M12.4l adds source/destination capsule and result directory-sync failures, plus
actual object read-permission denial after paired commit. Both representations
and transfer directions must preserve the independently observed heads and
pending roles before publication. Current-result I/O failure pauses both hosts
for complete retry under the same epochs; unavailable counterpart replay evidence
quarantines and restoring access cannot clear quarantine. Explicit lease retirement
and replacement must recover exact bytes under newer epochs, and exact retry must
return the original receipt. Runtime
checks require real I/O errors, without changing immutable bytes or skipping
under elevated filesystem privileges. See [storage scope](M12_MIGRATION_STORAGE_FAULTS.md).
Tests defend contracts, not directory coverage. All checks remain active in
Release builds. Current proof has no actual network, biology or distributed
recovery, so those categories below are milestone gates rather than passing
+50
View File
@@ -1,5 +1,55 @@
# Validation report
## 2026-10-18 - M12.4l migration object sync/read failures
Test-only qualification extends ADR-0147–0149 using the existing filesystem
adapter's directory-sync fault seam and actual read-permission denial under the
unprivileged disposable-cluster user. Both raw and cohort representations use
real player-origin wildlife, outbound and return transfers, unchanged complete
payload oracles and independent database head/pointer observations. See
[storage fixture and limits](M12_MIGRATION_STORAGE_FAULTS.md).
Executed locally:
- GCC 14 optional persistence/process targets rebuilt. The final private
population/migration suite passed **17 s** (15 s population phase).
- Linux Clang 19 **ASan/UBSan** storage and TLS fault matrix passed **64 s**
(60 s population phase), with no sanitizer reports.
- Explicit clang-tidy 19 on the changed publication tests passed with no owned
warnings. Repository clang-format 19 passed.
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
**546 s**: C++ integration/process/load/history (434 s), combined TLS/storage
fault matrix (38 s), life reservations, backup/restore, host-loss DR, upgrade,
migration history/rollback, runtime permission audit and checksum/future-version
rejection. The detached runner recorded its completion code.
Failed source capsule sync cannot prepare a freeze; failed destination capsule
sync cannot stage a ghost. Failure of either result sync leaves both heads and
both pending roles unchanged. Existing identical objects are reverified and
synced on retry; visible bytes do not grant ownership.
Post-commit direct reads report actual I/O errors while the database retains
both exact result heads. Current-payload I/O failure pauses complete recovery and
retries under the same epochs after access returns. Unavailable counterpart
replay evidence quarantines both owners; restoring access leaves quarantine
sticky. Explicit current-lease retirement and replacement recover exact bytes
under strictly newer epochs. The original paired receipt survives both paths.
The first test draft incorrectly expected current-payload I/O failure to
quarantine. Inspection confirmed the existing retriable `RecoveryError::store`
contract. The test now distinguishes this from unavailable paired replay evidence
(`RecoveryError::tail`) and verifies both behaviors without changing production
error handling or weakening readiness checks.
Limits: sync failure uses an existing test seam; permission denial affects
selected local objects, not a storage-host or physical media/power failure. Test
coordinator lease retirement is explicit, not a new automatic quarantine policy.
No new production/thread behavior, SQL, dependency, biological rule/format,
golden, live database, World0 or deployment changed. Native/TSan results from
M12.4k remain separate; this test-only segment does not claim new runs of those
checks or a Docker image build/remote CI result. Linked ecology and later M12
integration/soak gates remain open.
## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions
Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and