M12.4: qualify migration storage failures and exact recovery
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 19m42s
Foundation / checks (checks) (push) Canceled after 19m48s
Foundation / checks (tsan) (push) Canceled after 19m49s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m12s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 19m42s
Foundation / checks (checks) (push) Canceled after 19m48s
Foundation / checks (tsan) (push) Canceled after 19m49s
This commit is contained in:
@@ -1,5 +1,22 @@
|
||||
# Project status and next steps
|
||||
|
||||
## M12.4l local migration storage failures - 2026-10-18
|
||||
|
||||
The private matrix now refuses preparation, staging or paired publication after
|
||||
capsule/result directory-sync failures. Real read-permission denial after commit
|
||||
pauses current-payload recovery for retry or quarantines unavailable counterpart
|
||||
replay evidence. Both paths preserve the exact database heads and receipt; full
|
||||
recovery restores exact biology, with newer authority after quarantine.
|
||||
|
||||
GCC private population/migration checks pass in 17 s, ASan/UBSan storage/TLS
|
||||
checks in 64 s, and Clang analysis/format checks pass.
|
||||
Complete private persistence/process/backup/DR and migration/permission
|
||||
qualification, including the combined fault matrix, passes in 546 s.
|
||||
[Evidence and limits](engineering/VALIDATION.md) distinguish local sync/read
|
||||
failures from physical media, storage-host and orchestrator drills. Linked ecology,
|
||||
maturation and later M12 integration/soak gates remain open. This segment changes
|
||||
tests/documentation only.
|
||||
|
||||
## M12.4k independent migration TLS path faults - 2026-10-18
|
||||
|
||||
Test-only relays now exercise real loopback TCP/end-to-end TLS request/receipt
|
||||
|
||||
@@ -35,6 +35,12 @@ owner's actual lease expiry, replacement and healed stale fencing in both
|
||||
representations. Independent database observation checks the outcomes. See
|
||||
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
|
||||
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
|
||||
M12.4l extends local qualification with capsule/result directory-sync failures and
|
||||
actual object read-permission denial after commit. Current-payload I/O failure
|
||||
pauses complete recovery for retry; unavailable retained counterpart evidence
|
||||
quarantines. Replacement fully verifies the original pair under newer authority. See
|
||||
[storage scope](../engineering/M12_MIGRATION_STORAGE_FAULTS.md). Remote storage-host,
|
||||
physical media and orchestrator failures remain separate evidence.
|
||||
|
||||
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
|
||||
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
|
||||
|
||||
@@ -208,3 +208,17 @@ application-boundary byte loss over real local TCP/TLS, not multi-host/kernel
|
||||
networking, database/object-store loss or orchestration. Linked ecology and later
|
||||
M12 gates remain open. No SQL, production code, biological rule/format or deployment
|
||||
changes; the added relays and credential tools belong to tests/CI only.
|
||||
|
||||
## M12.4l local storage fault qualification
|
||||
|
||||
The private migration matrix now fails source/destination capsule and result
|
||||
directory syncs before publication and denies actual reads after paired commit.
|
||||
Current-result I/O failures pause verified recovery for retry under the same
|
||||
epochs. Unavailable counterpart replay evidence quarantines; restoring access
|
||||
leaves that quarantine sticky. Explicit lease retirement and replacement recover
|
||||
the exact original pair under newer authority. Independent database observations
|
||||
prove unchanged predecessor heads before publication and unchanged committed heads
|
||||
after it; exact retries never transfer stock twice. Both representations and
|
||||
outbound/return directions run these cases without changing immutable bytes.
|
||||
See [the storage scope](M12_MIGRATION_STORAGE_FAULTS.md). Physical media, remote
|
||||
storage-host and orchestrator failures remain separate evidence.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
# M12.4l migration storage fault qualification
|
||||
|
||||
This extends the ADR-0147–0149 private PostgreSQL migration matrix. It changes
|
||||
tests and documentation only, with the existing ADR-0061 filesystem adapter and
|
||||
its test-only directory-sync seam. No SQL, production API, biological rule,
|
||||
format, dependency, deployment or live world changes.
|
||||
|
||||
## Matrix and command
|
||||
|
||||
```sh
|
||||
bash tools/ci/postgres-check.sh --population-only "$PWD/build/m97-postgres/evo-postgres-tests"
|
||||
```
|
||||
|
||||
The full, sanitizer and TLS qualification paths also run these cases as part of
|
||||
the population suite. Both raw/profile-4 and cohort/profile-5 cases use real
|
||||
player-origin wildlife and outbound/return transfers.
|
||||
|
||||
- Failure of the source capsule's directory sync refuses preparation without a
|
||||
pending source pointer. A subsequent ordinary preparation verifies and syncs
|
||||
the retained object before using it.
|
||||
- Failure of the destination capsule's directory sync refuses staging. The
|
||||
source remains frozen and no destination pending pointer appears.
|
||||
- Failure of either result object's directory sync refuses paired publication.
|
||||
Independent database observations prove both heads unchanged and both pending
|
||||
roles retained. A visible object alone never grants ownership; later retries
|
||||
verify and sync identical bytes again.
|
||||
- After a real paired commit, the unprivileged fixture user loses read permission
|
||||
on both current result objects (outbound) or both local counterpart predecessor
|
||||
capsules (return). Direct adapter reads must report actual I/O errors. Paused
|
||||
owners refuse advancement while independent database observation proves the
|
||||
committed heads remain unchanged. Current-payload I/O failure is retriable and
|
||||
increments the store-failure counter; unavailable paired replay evidence is a
|
||||
tail-verification failure and quarantines. Pointer settlement cannot resume
|
||||
stale source stock or activate an unverified destination.
|
||||
- A scope guard restores original permissions without changing immutable bytes.
|
||||
Temporary current-payload I/O failures reconcile the complete pair under the
|
||||
same epochs. Tail-verification quarantine remains sticky: the test coordinator
|
||||
explicitly retires the captured current database leases, then replacement hosts
|
||||
obtain strictly newer epochs and recover the exact committed payloads. Exact
|
||||
paired retries return the original receipt without a second transfer.
|
||||
|
||||
The read-permission fixture requires an unprivileged Linux user, as does the
|
||||
disposable PostgreSQL cluster. It fails if permissions do not actually deny
|
||||
reads; it does not silently skip under elevated filesystem privileges.
|
||||
|
||||
## Limits
|
||||
|
||||
Directory-sync failures use the existing adapter test seam; they are not physical
|
||||
disk failure, a power-loss durability experiment or a filesystem remount. Read
|
||||
denial is a real filesystem permission failure on selected retained objects,
|
||||
without deleting, corrupting or rewriting their contents. It does not establish
|
||||
remote object-storage availability, multi-host failover, capacity/performance or
|
||||
orchestrator recovery. Explicit lease retirement in the test is a coordinator
|
||||
action under current captured authority, not an automatic quarantine policy.
|
||||
|
||||
Multi-host/kernel-network faults, storage-host failure, orchestrator drills,
|
||||
linked ecology, maturation, world census/cuts, target-host costs and actual
|
||||
elapsed-day ecology remain open. Executed checks are recorded in
|
||||
[VALIDATION](VALIDATION.md); this document alone is not a passing result.
|
||||
@@ -25,7 +25,7 @@ or authorize production enablement.
|
||||
|---|---|---|
|
||||
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
|
||||
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
|
||||
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
|
||||
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
|
||||
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
|
||||
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
|
||||
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
|
||||
@@ -42,11 +42,13 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics.
|
||||
The engineering constitution requires stopping expansion at this architectural
|
||||
boundary and documenting the resolution. That does not prevent the independent
|
||||
worker scheduling correction or the dedicated exact population adapter above.
|
||||
Biological migration escrow is the next M12.4 boundary; none of the rows above is
|
||||
closed by adding a placeholder service or relabeling the non-biological probe.
|
||||
Durable biological migration escrow is implemented through ADR-0147–0149 below;
|
||||
none of the rows above is closed by adding a placeholder service or relabeling
|
||||
the non-biological probe.
|
||||
ADR-0144's [portable transfer candidates](M12_EXACT_BIOLOGICAL_TRANSFER.md) resolve
|
||||
the closed-owner/foreign-identity conflict with explicit flows and nonresident
|
||||
history. They do not yet supply durable paired head publication or activation.
|
||||
history. Those portable candidates alone grant no durable publication or
|
||||
activation authority; the dedicated workflow below supplies that boundary.
|
||||
Paired transfer trace 1 now supplies bounded, independently replayable input and
|
||||
member/material evidence, including raw/cohort separate-process reproduction.
|
||||
It does not grant ownership or close database escrow/retry/partition qualification.
|
||||
@@ -80,6 +82,12 @@ Independent database observation proves unchanged or atomically committed heads;
|
||||
the healthy peer keeps its epoch and retained pause. See
|
||||
[the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network
|
||||
partitions, object-store/orchestrator failures and linked ecology remain open.
|
||||
M12.4l adds local object directory-sync failures before publication and actual
|
||||
read-permission denial after commit. Current-result I/O failure pauses complete
|
||||
recovery for retry; unavailable counterpart replay evidence quarantines until
|
||||
explicit replacement. Both preserve the original receipt and heads. See
|
||||
[storage scope](M12_MIGRATION_STORAGE_FAULTS.md). This does not qualify a remote
|
||||
storage-host outage or physical media/power failure.
|
||||
The transfer tests also expose the current rules-8 maturation gap: buds retain
|
||||
their subadult realization while reproduction requires a fully realized parent.
|
||||
Finite extra food alone does not close repeated wildlife generations. A versioned
|
||||
|
||||
@@ -68,6 +68,17 @@ in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix
|
||||
Multi-host/kernel-network failures, object-store outages and orchestrator drills
|
||||
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
|
||||
|
||||
M12.4l adds source/destination capsule and result directory-sync failures, plus
|
||||
actual object read-permission denial after paired commit. Both representations
|
||||
and transfer directions must preserve the independently observed heads and
|
||||
pending roles before publication. Current-result I/O failure pauses both hosts
|
||||
for complete retry under the same epochs; unavailable counterpart replay evidence
|
||||
quarantines and restoring access cannot clear quarantine. Explicit lease retirement
|
||||
and replacement must recover exact bytes under newer epochs, and exact retry must
|
||||
return the original receipt. Runtime
|
||||
checks require real I/O errors, without changing immutable bytes or skipping
|
||||
under elevated filesystem privileges. See [storage scope](M12_MIGRATION_STORAGE_FAULTS.md).
|
||||
|
||||
Tests defend contracts, not directory coverage. All checks remain active in
|
||||
Release builds. Current proof has no actual network, biology or distributed
|
||||
recovery, so those categories below are milestone gates rather than passing
|
||||
|
||||
@@ -1,5 +1,55 @@
|
||||
# Validation report
|
||||
|
||||
## 2026-10-18 - M12.4l migration object sync/read failures
|
||||
|
||||
Test-only qualification extends ADR-0147–0149 using the existing filesystem
|
||||
adapter's directory-sync fault seam and actual read-permission denial under the
|
||||
unprivileged disposable-cluster user. Both raw and cohort representations use
|
||||
real player-origin wildlife, outbound and return transfers, unchanged complete
|
||||
payload oracles and independent database head/pointer observations. See
|
||||
[storage fixture and limits](M12_MIGRATION_STORAGE_FAULTS.md).
|
||||
|
||||
Executed locally:
|
||||
|
||||
- GCC 14 optional persistence/process targets rebuilt. The final private
|
||||
population/migration suite passed **17 s** (15 s population phase).
|
||||
- Linux Clang 19 **ASan/UBSan** storage and TLS fault matrix passed **64 s**
|
||||
(60 s population phase), with no sanitizer reports.
|
||||
- Explicit clang-tidy 19 on the changed publication tests passed with no owned
|
||||
warnings. Repository clang-format 19 passed.
|
||||
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
|
||||
**546 s**: C++ integration/process/load/history (434 s), combined TLS/storage
|
||||
fault matrix (38 s), life reservations, backup/restore, host-loss DR, upgrade,
|
||||
migration history/rollback, runtime permission audit and checksum/future-version
|
||||
rejection. The detached runner recorded its completion code.
|
||||
|
||||
Failed source capsule sync cannot prepare a freeze; failed destination capsule
|
||||
sync cannot stage a ghost. Failure of either result sync leaves both heads and
|
||||
both pending roles unchanged. Existing identical objects are reverified and
|
||||
synced on retry; visible bytes do not grant ownership.
|
||||
|
||||
Post-commit direct reads report actual I/O errors while the database retains
|
||||
both exact result heads. Current-payload I/O failure pauses complete recovery and
|
||||
retries under the same epochs after access returns. Unavailable counterpart
|
||||
replay evidence quarantines both owners; restoring access leaves quarantine
|
||||
sticky. Explicit current-lease retirement and replacement recover exact bytes
|
||||
under strictly newer epochs. The original paired receipt survives both paths.
|
||||
|
||||
The first test draft incorrectly expected current-payload I/O failure to
|
||||
quarantine. Inspection confirmed the existing retriable `RecoveryError::store`
|
||||
contract. The test now distinguishes this from unavailable paired replay evidence
|
||||
(`RecoveryError::tail`) and verifies both behaviors without changing production
|
||||
error handling or weakening readiness checks.
|
||||
|
||||
Limits: sync failure uses an existing test seam; permission denial affects
|
||||
selected local objects, not a storage-host or physical media/power failure. Test
|
||||
coordinator lease retirement is explicit, not a new automatic quarantine policy.
|
||||
No new production/thread behavior, SQL, dependency, biological rule/format,
|
||||
golden, live database, World0 or deployment changed. Native/TSan results from
|
||||
M12.4k remain separate; this test-only segment does not claim new runs of those
|
||||
checks or a Docker image build/remote CI result. Linked ecology and later M12
|
||||
integration/soak gates remain open.
|
||||
|
||||
## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions
|
||||
|
||||
Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and
|
||||
|
||||
Reference in New Issue
Block a user