M12.4: qualify migration ownership across TLS path loss
Foundation / checks (ecs) (push) Successful in 2m29s
Foundation / checks (deploy) (push) Successful in 4m8s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 18m9s
Foundation / checks (client) (push) Canceled after 18m8s
Foundation / checks (network) (push) Canceled after 18m7s
Foundation / checks (tsan) (push) Canceled after 15m39s
Foundation / checks (postgres) (push) Canceled after 18m13s

This commit is contained in:
dingo
2026-10-18 13:19:59 -04:00
parent 0f071f3f8b
commit e0a7c13f6b
16 changed files with 707 additions and 14 deletions
+18
View File
@@ -1,5 +1,23 @@
# Project status and next steps
## M12.4k independent migration TLS path faults - 2026-10-18
Test-only relays now exercise real loopback TCP/end-to-end TLS request/receipt
loss for raw and cohort migration. Independent observation proves unchanged or
atomically committed heads. Owning hosts retain lost prepare/stage/abort work,
fence after actual partitioned lease expiry, recover under replacement authority
and refuse healed stale owners. Their healthy peer keeps its epoch/freeze.
Wrong-hostname TLS connections fail; production adapter settings are unchanged.
Native Release passes 78/78; GCC, ASan/UBSan, TSan, Clang analysis, format and
shell checks pass. Complete private persistence, process-fault, backup/DR and
migration/permission qualification, including the TLS matrix, passes in 540 s.
[Evidence and limits](engineering/VALIDATION.md) distinguish
these local transport faults from multi-host/kernel-network, database/object-store
and orchestrator drills. Linked ecology and later M12 integration/soak gates
remain open. No SQL, production code, biological rule/format, deployment or live
region changed. Full/sanitizer CI paths include the new disposable TLS matrix.
## M12.4j atomic paired biological publication - 2026-10-18
ADR-0149/schema 49 commits both exact migration results and one immutable common
@@ -30,7 +30,11 @@ second-insert failure rolls back both heads and pointer settlement. Recovery
replays both cuts and binds both results to the stage. Frozen hosts reload and
verify a changed head before readiness, rather than inferring activation from
pointer absence. See [ADR-0149](../adr/0149-atomic-cellular-migration-publication.md).
Independent network partitions and linked ecological boundaries remain separate.
M12.4k qualifies independent loopback TCP/TLS request/receipt loss and either
owner's actual lease expiry, replacement and healed stale fencing in both
representations. Independent database observation checks the outcomes. See
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
@@ -190,3 +190,21 @@ See [the paired commit contract](../../schemas/ecology/cellular-migration-commit
Independent network partitions, linked geometry/environment, maturation, world
census/cuts, actual elapsed-day ecology and target-host capacity remain open.
No automatic route, biological rule/format, golden or deployment changes.
## M12.4k independent TLS transport fault qualification
A test-owned opaque relay and separate disposable loopback TLS cluster exercise
the unchanged production adapter. Both representations lose real requests and
receipts; independent observation distinguishes unchanged heads from an exact
committed pair. Lost preparation/staging/abort replies retain durable work and
host pauses. Either owner's path can remain isolated through actual lease expiry
while its peer retains the same epoch/freeze. Replacement recovers the same work;
stale publication fails, explicit abort releases both pauses and healing does not
revive the old fence. Wrong-hostname TLS is refused. ASan/UBSan and TSan exercise
the matrix; the full/sanitizer CI paths include it in a separate cluster.
See [fixture and qualification limits](M12_MIGRATION_TRANSPORT_FAULTS.md). This is
application-boundary byte loss over real local TCP/TLS, not multi-host/kernel
networking, database/object-store loss or orchestration. Linked ecology and later
M12 gates remain open. No SQL, production code, biological rule/format or deployment
changes; the added relays and credential tools belong to tests/CI only.
@@ -0,0 +1,79 @@
# M12.4k migration transport fault qualification
This qualifies ADR-0147–0149 over real loopback TCP with end-to-end PostgreSQL TLS.
It changes tests and CI only: no SQL, biological format/rule, production adapter,
world configuration, dependency in the headless library or deployment changes.
## Fixture and command
```sh
bash tools/ci/postgres-check.sh --migration-faults "$PWD/build/m97-postgres/evo-postgres-tests"
```
The runner creates a fresh 0700 `/tmp/evo-pg-test.*` cluster and temporary TLS
certificate/key. Only this mode listens on `127.0.0.1` at an ephemeral port. The
owner stays on its protected Unix socket; only the least-privileged test login is
admitted over TLS. Ordinary fixtures retain disabled TCP. The full and sanitizer
qualification paths run this mode in a separate cluster, never a supplied/live
database. Setup failure is a failed test, without a silent skip.
The test-owned relay forwards opaque encrypted bytes. It binds only loopback,
owns and joins its worker, bounds each direction's queue to 64 KiB, and records
discarded-byte counts. It can discard client requests or server responses
independently. The unchanged adapter still uses `sslmode=verify-full`; a wrong
hostname must fail. The fixture neither terminates TLS nor changes SQL results,
leases, clocks or biological stock. Timeouts retain the adapter's real unknown
outcome behavior.
## Executed matrix
Both raw owner-7/profile-4 and exact cohort/profile-5 cases use actual funded
player-origin wildlife from the existing publication fixture.
- **Request path lost:** a paired request is discarded after TLS setup. The
caller gets unknown with the exact two manifests; independent Unix-socket
observation proves both heads stayed at their predecessors.
- **Receipt path lost:** the database receives and commits the pair while the
relay discards its reply. The caller gets unknown; independent observation
proves both exact result manifests committed. Existing exact retry/recovery
checks then prove no second transfer.
- **Preparation/staging replies lost:** the owning source/destination hosts
pause, retain exact work and reconnect to their durable freeze/ghost. No
candidate is activated from an unconfirmed reply.
- **Either owner isolated:** that owner's request path stays unavailable through
actual three-second lease expiry. The independent healthy peer keeps its same
epoch and retained pause. A replacement acquires a newer isolated-region epoch
and recovers the same preparation/stage. Old-fence publication is refused and
both heads remain unchanged.
- **Explicit abort reply lost:** current source authority decides abort despite
a lost response. Exact reconciliation releases both pointers. The healthy
owner resumes identical biology; the isolated old owner remains fenced when
its path heals while the replacement owns the region.
The matrix runs alongside existing process-kill, second-insert rollback,
commit/abort race, full-payload replay and durable continuation tests. Numerical
stock/head equality is checked independently of relay counters and client errors.
## Tooling evidence and limits
The relay uses standard C++/POSIX facilities and the already approved PostgreSQL
TLS adapter. Certificate generation uses the existing OpenSSL tool family;
Python uses only its standard-library `socket` module to select a loopback port.
The CI-only image installs Ubuntu's `openssl`/`python3` packages and prints their
versions. Production image package lists do not change.
Local evidence: OpenSSL 3.0.13 (`3.0.13-0ubuntu3.16`), Python 3.12.3
(`python3 3.12.3-0ubuntu2.1`, `python3.12 3.12.3-1ubuntu0.17`). Packaged copyright
evidence is `/usr/share/doc/openssl/copyright` (Apache-2.0 for upstream OpenSSL)
and `/usr/share/doc/python3.12/copyright` (Python licensing and bundled notices).
These are distro-managed test utilities, without vendored code, a new package
registry, configure-time fetch or selected game dependency. CI records its actual
installed versions; local package versions do not promise future CI versions.
The relay discards bytes at an application transport boundary on one machine.
This is not kernel packet loss, a multi-host routing/asymmetric firewall drill,
remote database/object-store failure, or a production orchestrator failover test.
The local object stores remain available. Linked geometry/resources, maturation,
global census/cuts, target-host cost and actual elapsed-day ecology remain open.
Executed compiler/sanitizer results and environmental limits belong in
[VALIDATION](VALIDATION.md). No World0 action or cellular clone/restore claim.
+8 -3
View File
@@ -25,7 +25,7 @@ or authorize production enablement.
|---|---|---|
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Owned bounded intervals; exact transfer/accounting; durable preparation/staging, atomic paired publication and replayed host recovery | Independent network-partition drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
@@ -73,8 +73,13 @@ second inserts roll back both heads; commit/abort races have one exclusive
outcome. Exact retries preserve both manifests under fresh authority. Hosts
replay and verify changed heads before readiness. Preparation remains an
optimistic destination observation; staging/publication recheck captured heads.
Independent network-partition drills and linked ecological boundaries remain
open; shared-database fault tests do not supply those proofs.
M12.4k now exercises actual loopback TCP/end-to-end TLS with independent request
and response loss, lost prepare/stage/abort receipts, either owner's actual lease
expiry, replacement and healed stale-owner fencing in both representations.
Independent database observation proves unchanged or atomically committed heads;
the healthy peer keeps its epoch and retained pause. See
[the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network
partitions, object-store/orchestrator failures and linked ecology remain open.
The transfer tests also expose the current rules-8 maturation gap: buds retain
their subadult realization while reproduction requires a fully realized parent.
Finite extra food alone does not close repeated wildlife generations. A versioned
+10
View File
@@ -58,6 +58,16 @@ retain complete recovery after replacement and continue biology afterwards.
Independent network partitions, linked ecology and actual elapsed-day stability
remain separate evidence; full schema/permission/backup/DR regression is required.
M12.4k adds an explicit disposable loopback TLS mode and test-owned opaque relays.
Both representations exercise request loss, committed receipt loss, wrong-host
rejection, lost preparation/staging/abort replies, either owner's actual lease
expiry, an unchanged healthy-peer epoch, replacement and healed stale-owner
fencing. Independent Unix-socket observations prove the database outcome; relay
byte counters alone are insufficient. Full and sanitizer CI paths run this mode
in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix.
Multi-host/kernel-network failures, object-store outages and orchestrator drills
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
Tests defend contracts, not directory coverage. All checks remain active in
Release builds. Current proof has no actual network, biology or distributed
recovery, so those categories below are milestone gates rather than passing
+55
View File
@@ -1,5 +1,60 @@
# Validation report
## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions
Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and
the unchanged adapter's end-to-end `verify-full` PostgreSQL TLS. A separate
disposable cluster mode admits only the test runtime login over loopback TLS;
ordinary fixtures keep TCP disabled. Full/sanitizer CI paths run this matrix.
See [fixture, tooling and limits](M12_MIGRATION_TRANSPORT_FAULTS.md).
Executed locally:
- Native MSVC Release **78/78 passed**, 12.63 s.
- Actual GCC 14 optional targets rebuilt; final TLS matrix passed **35 s**
(34 s population phase), with both raw/cohort markers for request/reply loss
and independent owner partition/expiry/replacement/heal.
- Linux Clang 19 **ASan/UBSan** TLS matrix passed **62 s** (58 s population phase),
with no sanitizer reports.
- Separate Linux Clang 19 **ThreadSanitizer** optional PostgreSQL build and TLS
matrix passed **63 s** (60 s population phase), with no race reports.
- Explicit clang-tidy 19 on the relay and publication/fault tests passed with no
owned warnings. Repository clang-format 19 and shell syntax checks passed.
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
**540 s**: C++ integration/process/load/history (432 s), the separate TLS fault
matrix (36 s), life reservations, backup/restore, host-loss DR, upgrade,
migration history/rollback, runtime permission audit and checksum/future-version
rejection. The detached runner recorded its completion code.
Wrong-hostname TLS connections fail. Discarded requests return unknown with the
exact prepared manifests while independent observation proves both heads stayed
unchanged. Discarded commit replies return unknown while independent observation
proves both exact results committed. Source preparation, destination staging and
explicit abort also lose real replies without losing retained work or activating
ghosts. Either owner remains isolated through actual lease expiry, while its
healthy peer keeps the same epoch and freeze. Replacement gets a newer epoch and
the same retained work; stale publication fails with both heads unchanged.
Explicit current-source abort reconciles lost receipts. The healthy host resumes
identical biology; a healed old owner stays fenced while replacement authority
is held. All of these cases run for raw and cohort representations with actual
player-origin wildlife and the existing full replay/continuation matrix.
The initial analysis found avoidable stop-token copies in the test relay; both
callback/run arguments now use const references. No warning suppression or
production retry/TLS escape hatch was added. The new relay is test-owned and its
worker is joined; no production thread/service is introduced.
Limits: byte discard at an application TCP boundary on one machine is not kernel
packet loss, a multi-host routing/firewall failure, a database host outage, object
store unavailability or an orchestrator drill. Existing local objects stay
available. The CI-only image adds distro OpenSSL/Python utilities for ephemeral
credentials/port selection; local versions/licensing evidence are recorded in
the fixture document. No Docker image build or remote CI result is claimed here.
Linked ecology, maturation, global census/cuts, target-host costs and actual
elapsed-day soak remain open. No SQL, production adapter, biological format/rule,
golden, live database, World0 or deployment changed. `Build_Errors.txt` and unrelated
local work remain untracked; prior captured CI runner failures remain separate.
## 2026-10-18 - M12.4j atomic paired biological publication and recovery
ADR-0149/schema 49 commits both exact bundle-10 no-tick results and one immutable
+10 -4
View File
@@ -9,8 +9,9 @@ distributed tooling. No custom hashing/cryptography is introduced.
## Local isolated verification
Linux prerequisites: GCC14, CMake>=3.28, Ninja, make, bison, flex, curl, bzip2,
OpenSSL development headers and zlib development headers. Optional clang19 tools
support analysis/ASan. Run as a normal user, not root:
OpenSSL development headers, the OpenSSL CLI, Python 3 (standard library only)
and zlib development headers. Optional clang19 tools support analysis/ASan/TSan.
Run as a normal user, not root:
```sh
bash tools/prepare-postgres.sh
@@ -18,13 +19,18 @@ cmake --preset release -B build/postgres -DCMAKE_CXX_COMPILER=g++-14 \
-DEVO_BUILD_POSTGRES=ON -DPostgreSQL_ROOT="$PWD/.cache/postgres/install"
cmake --build build/postgres --target evo-postgres-tests --parallel 2
bash tools/ci/postgres-check.sh "$PWD/build/postgres/evo-postgres-tests"
# Focused raw/cohort migration faults over real loopback TLS:
bash tools/ci/postgres-check.sh --migration-faults "$PWD/build/postgres/evo-postgres-tests"
```
The test script overrides all connection coordinates and creates a new private
0700 /tmp directory, private Unix socket and disposable local cluster. It creates
the `evo_runtime_test` group and the `evo_runtime_login` member, then runs fencing
tests as that real runtime login. TCP is off;
local trust applies only inside that private directory. Cluster stops on exit;
tests as that real runtime login. Ordinary fixtures disable TCP; local trust
applies only inside that private directory. The migration-fault mode creates a
separate private cluster with ephemeral loopback TLS for the runtime login only;
the full and sanitizer paths include it. See [scope and prerequisites](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md).
Clusters stop on exit;
logs/data are retained there for diagnosis. It never uses production PG credentials
or deletes a pre-existing database. Do not point this script at a shared server.