M12.4: qualify migration ownership across TLS path loss
Foundation / checks (ecs) (push) Successful in 2m29s
Foundation / checks (deploy) (push) Successful in 4m8s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 18m9s
Foundation / checks (client) (push) Canceled after 18m8s
Foundation / checks (network) (push) Canceled after 18m7s
Foundation / checks (tsan) (push) Canceled after 15m39s
Foundation / checks (postgres) (push) Canceled after 18m13s

This commit is contained in:
dingo
2026-10-18 13:19:59 -04:00
parent 0f071f3f8b
commit e0a7c13f6b
16 changed files with 707 additions and 14 deletions
+1
View File
@@ -307,6 +307,7 @@ if(EVO_BUILD_POSTGRES)
add_executable(evo-postgres-tests tests/postgres_tests.cpp tests/region_lease_store_tests.cpp
tests/cellular_publication_tests.cpp
tests/population_publication_tests.cpp
tests/fixtures/postgres_relay.cpp
tests/account_control_store_tests.cpp tests/probe_handoff_store_tests.cpp
tests/life_reservation_store_tests.cpp
tests/primitive_publication_tests.cpp
+5 -2
View File
@@ -140,8 +140,11 @@ worker/session path. Opt-in exact population profile 5 is described in
replacement owners; biological migration, environmental coupling and production
qualification remain in progress. The
portable [biological transfer candidates](docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md)
preserve exact identity/history and regional boundary budgets; their durable
escrow is still required. The microscope client is the current M11 playtest path.
preserve exact identity/history and regional boundary budgets. Their scoped
durable preparation/staging, atomic paired publication and verified recovery are
implemented; [TLS transport fault checks](docs/engineering/M12_MIGRATION_TRANSPORT_FAULTS.md)
cover independent local path loss and owner replacement. Linked ecology and
deployment qualification remain open. The microscope client is the M11 playtest path.
`evo::simulation` remains
graphics-, network- and database-independent.
+5 -1
View File
@@ -156,7 +156,11 @@ RUN cmake --preset asan -B build/postgres-asan -DCMAKE_CXX_COMPILER=clang++-19 \
&& cmake --build build/postgres-asan --target evo-postgres-tests --parallel "${PG_BUILD_JOBS}"
FROM postgres-test-build AS ci-postgres
RUN useradd --uid 10001 --no-create-home evo
# Test-only ephemeral TLS credentials and a standard-library loopback port picker.
RUN apt-get update && apt-get install -y --no-install-recommends openssl python3 \
&& rm -rf /var/lib/apt/lists/* \
&& openssl version && python3 --version \
&& useradd --uid 10001 --no-create-home evo
USER 10001:10001
# Full qualification once on the release build. ASan/UBSan reruns everything that
# executes its instrumented binaries; the uninstrumented clang-tidy build reruns only
+18
View File
@@ -1,5 +1,23 @@
# Project status and next steps
## M12.4k independent migration TLS path faults - 2026-10-18
Test-only relays now exercise real loopback TCP/end-to-end TLS request/receipt
loss for raw and cohort migration. Independent observation proves unchanged or
atomically committed heads. Owning hosts retain lost prepare/stage/abort work,
fence after actual partitioned lease expiry, recover under replacement authority
and refuse healed stale owners. Their healthy peer keeps its epoch/freeze.
Wrong-hostname TLS connections fail; production adapter settings are unchanged.
Native Release passes 78/78; GCC, ASan/UBSan, TSan, Clang analysis, format and
shell checks pass. Complete private persistence, process-fault, backup/DR and
migration/permission qualification, including the TLS matrix, passes in 540 s.
[Evidence and limits](engineering/VALIDATION.md) distinguish
these local transport faults from multi-host/kernel-network, database/object-store
and orchestrator drills. Linked ecology and later M12 integration/soak gates
remain open. No SQL, production code, biological rule/format, deployment or live
region changed. Full/sanitizer CI paths include the new disposable TLS matrix.
## M12.4j atomic paired biological publication - 2026-10-18
ADR-0149/schema 49 commits both exact migration results and one immutable common
@@ -30,7 +30,11 @@ second-insert failure rolls back both heads and pointer settlement. Recovery
replays both cuts and binds both results to the stage. Frozen hosts reload and
verify a changed head before readiness, rather than inferring activation from
pointer absence. See [ADR-0149](../adr/0149-atomic-cellular-migration-publication.md).
Independent network partitions and linked ecological boundaries remain separate.
M12.4k qualifies independent loopback TCP/TLS request/receipt loss and either
owner's actual lease expiry, replacement and healed stale fencing in both
representations. Independent database observation checks the outcomes. See
[transport scope](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host
network/kernel, object-store/orchestrator failures and linked ecology remain separate.
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
@@ -190,3 +190,21 @@ See [the paired commit contract](../../schemas/ecology/cellular-migration-commit
Independent network partitions, linked geometry/environment, maturation, world
census/cuts, actual elapsed-day ecology and target-host capacity remain open.
No automatic route, biological rule/format, golden or deployment changes.
## M12.4k independent TLS transport fault qualification
A test-owned opaque relay and separate disposable loopback TLS cluster exercise
the unchanged production adapter. Both representations lose real requests and
receipts; independent observation distinguishes unchanged heads from an exact
committed pair. Lost preparation/staging/abort replies retain durable work and
host pauses. Either owner's path can remain isolated through actual lease expiry
while its peer retains the same epoch/freeze. Replacement recovers the same work;
stale publication fails, explicit abort releases both pauses and healing does not
revive the old fence. Wrong-hostname TLS is refused. ASan/UBSan and TSan exercise
the matrix; the full/sanitizer CI paths include it in a separate cluster.
See [fixture and qualification limits](M12_MIGRATION_TRANSPORT_FAULTS.md). This is
application-boundary byte loss over real local TCP/TLS, not multi-host/kernel
networking, database/object-store loss or orchestration. Linked ecology and later
M12 gates remain open. No SQL, production code, biological rule/format or deployment
changes; the added relays and credential tools belong to tests/CI only.
@@ -0,0 +1,79 @@
# M12.4k migration transport fault qualification
This qualifies ADR-0147–0149 over real loopback TCP with end-to-end PostgreSQL TLS.
It changes tests and CI only: no SQL, biological format/rule, production adapter,
world configuration, dependency in the headless library or deployment changes.
## Fixture and command
```sh
bash tools/ci/postgres-check.sh --migration-faults "$PWD/build/m97-postgres/evo-postgres-tests"
```
The runner creates a fresh 0700 `/tmp/evo-pg-test.*` cluster and temporary TLS
certificate/key. Only this mode listens on `127.0.0.1` at an ephemeral port. The
owner stays on its protected Unix socket; only the least-privileged test login is
admitted over TLS. Ordinary fixtures retain disabled TCP. The full and sanitizer
qualification paths run this mode in a separate cluster, never a supplied/live
database. Setup failure is a failed test, without a silent skip.
The test-owned relay forwards opaque encrypted bytes. It binds only loopback,
owns and joins its worker, bounds each direction's queue to 64 KiB, and records
discarded-byte counts. It can discard client requests or server responses
independently. The unchanged adapter still uses `sslmode=verify-full`; a wrong
hostname must fail. The fixture neither terminates TLS nor changes SQL results,
leases, clocks or biological stock. Timeouts retain the adapter's real unknown
outcome behavior.
## Executed matrix
Both raw owner-7/profile-4 and exact cohort/profile-5 cases use actual funded
player-origin wildlife from the existing publication fixture.
- **Request path lost:** a paired request is discarded after TLS setup. The
caller gets unknown with the exact two manifests; independent Unix-socket
observation proves both heads stayed at their predecessors.
- **Receipt path lost:** the database receives and commits the pair while the
relay discards its reply. The caller gets unknown; independent observation
proves both exact result manifests committed. Existing exact retry/recovery
checks then prove no second transfer.
- **Preparation/staging replies lost:** the owning source/destination hosts
pause, retain exact work and reconnect to their durable freeze/ghost. No
candidate is activated from an unconfirmed reply.
- **Either owner isolated:** that owner's request path stays unavailable through
actual three-second lease expiry. The independent healthy peer keeps its same
epoch and retained pause. A replacement acquires a newer isolated-region epoch
and recovers the same preparation/stage. Old-fence publication is refused and
both heads remain unchanged.
- **Explicit abort reply lost:** current source authority decides abort despite
a lost response. Exact reconciliation releases both pointers. The healthy
owner resumes identical biology; the isolated old owner remains fenced when
its path heals while the replacement owns the region.
The matrix runs alongside existing process-kill, second-insert rollback,
commit/abort race, full-payload replay and durable continuation tests. Numerical
stock/head equality is checked independently of relay counters and client errors.
## Tooling evidence and limits
The relay uses standard C++/POSIX facilities and the already approved PostgreSQL
TLS adapter. Certificate generation uses the existing OpenSSL tool family;
Python uses only its standard-library `socket` module to select a loopback port.
The CI-only image installs Ubuntu's `openssl`/`python3` packages and prints their
versions. Production image package lists do not change.
Local evidence: OpenSSL 3.0.13 (`3.0.13-0ubuntu3.16`), Python 3.12.3
(`python3 3.12.3-0ubuntu2.1`, `python3.12 3.12.3-1ubuntu0.17`). Packaged copyright
evidence is `/usr/share/doc/openssl/copyright` (Apache-2.0 for upstream OpenSSL)
and `/usr/share/doc/python3.12/copyright` (Python licensing and bundled notices).
These are distro-managed test utilities, without vendored code, a new package
registry, configure-time fetch or selected game dependency. CI records its actual
installed versions; local package versions do not promise future CI versions.
The relay discards bytes at an application transport boundary on one machine.
This is not kernel packet loss, a multi-host routing/asymmetric firewall drill,
remote database/object-store failure, or a production orchestrator failover test.
The local object stores remain available. Linked geometry/resources, maturation,
global census/cuts, target-host cost and actual elapsed-day ecology remain open.
Executed compiler/sanitizer results and environmental limits belong in
[VALIDATION](VALIDATION.md). No World0 action or cellular clone/restore claim.
+8 -3
View File
@@ -25,7 +25,7 @@ or authorize production enablement.
|---|---|---|
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Owned bounded intervals; exact transfer/accounting; durable preparation/staging, atomic paired publication and replayed host recovery | Independent network-partition drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal | Multi-host/kernel-network and object-store/orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
@@ -73,8 +73,13 @@ second inserts roll back both heads; commit/abort races have one exclusive
outcome. Exact retries preserve both manifests under fresh authority. Hosts
replay and verify changed heads before readiness. Preparation remains an
optimistic destination observation; staging/publication recheck captured heads.
Independent network-partition drills and linked ecological boundaries remain
open; shared-database fault tests do not supply those proofs.
M12.4k now exercises actual loopback TCP/end-to-end TLS with independent request
and response loss, lost prepare/stage/abort receipts, either owner's actual lease
expiry, replacement and healed stale-owner fencing in both representations.
Independent database observation proves unchanged or atomically committed heads;
the healthy peer keeps its epoch and retained pause. See
[the transport scope](M12_MIGRATION_TRANSPORT_FAULTS.md). Multi-host/kernel-network
partitions, object-store/orchestrator failures and linked ecology remain open.
The transfer tests also expose the current rules-8 maturation gap: buds retain
their subadult realization while reproduction requires a fully realized parent.
Finite extra food alone does not close repeated wildlife generations. A versioned
+10
View File
@@ -58,6 +58,16 @@ retain complete recovery after replacement and continue biology afterwards.
Independent network partitions, linked ecology and actual elapsed-day stability
remain separate evidence; full schema/permission/backup/DR regression is required.
M12.4k adds an explicit disposable loopback TLS mode and test-owned opaque relays.
Both representations exercise request loss, committed receipt loss, wrong-host
rejection, lost preparation/staging/abort replies, either owner's actual lease
expiry, an unchanged healthy-peer epoch, replacement and healed stale-owner
fencing. Independent Unix-socket observations prove the database outcome; relay
byte counters alone are insufficient. Full and sanitizer CI paths run this mode
in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix.
Multi-host/kernel-network failures, object-store outages and orchestrator drills
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
Tests defend contracts, not directory coverage. All checks remain active in
Release builds. Current proof has no actual network, biology or distributed
recovery, so those categories below are milestone gates rather than passing
+55
View File
@@ -1,5 +1,60 @@
# Validation report
## 2026-10-18 - M12.4k TLS migration request/receipt and owner partitions
Test-only qualification extends ADR-0147–0149 with opaque loopback TCP relays and
the unchanged adapter's end-to-end `verify-full` PostgreSQL TLS. A separate
disposable cluster mode admits only the test runtime login over loopback TLS;
ordinary fixtures keep TCP disabled. Full/sanitizer CI paths run this matrix.
See [fixture, tooling and limits](M12_MIGRATION_TRANSPORT_FAULTS.md).
Executed locally:
- Native MSVC Release **78/78 passed**, 12.63 s.
- Actual GCC 14 optional targets rebuilt; final TLS matrix passed **35 s**
(34 s population phase), with both raw/cohort markers for request/reply loss
and independent owner partition/expiry/replacement/heal.
- Linux Clang 19 **ASan/UBSan** TLS matrix passed **62 s** (58 s population phase),
with no sanitizer reports.
- Separate Linux Clang 19 **ThreadSanitizer** optional PostgreSQL build and TLS
matrix passed **63 s** (60 s population phase), with no race reports.
- Explicit clang-tidy 19 on the relay and publication/fault tests passed with no
owned warnings. Repository clang-format 19 and shell syntax checks passed.
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
**540 s**: C++ integration/process/load/history (432 s), the separate TLS fault
matrix (36 s), life reservations, backup/restore, host-loss DR, upgrade,
migration history/rollback, runtime permission audit and checksum/future-version
rejection. The detached runner recorded its completion code.
Wrong-hostname TLS connections fail. Discarded requests return unknown with the
exact prepared manifests while independent observation proves both heads stayed
unchanged. Discarded commit replies return unknown while independent observation
proves both exact results committed. Source preparation, destination staging and
explicit abort also lose real replies without losing retained work or activating
ghosts. Either owner remains isolated through actual lease expiry, while its
healthy peer keeps the same epoch and freeze. Replacement gets a newer epoch and
the same retained work; stale publication fails with both heads unchanged.
Explicit current-source abort reconciles lost receipts. The healthy host resumes
identical biology; a healed old owner stays fenced while replacement authority
is held. All of these cases run for raw and cohort representations with actual
player-origin wildlife and the existing full replay/continuation matrix.
The initial analysis found avoidable stop-token copies in the test relay; both
callback/run arguments now use const references. No warning suppression or
production retry/TLS escape hatch was added. The new relay is test-owned and its
worker is joined; no production thread/service is introduced.
Limits: byte discard at an application TCP boundary on one machine is not kernel
packet loss, a multi-host routing/firewall failure, a database host outage, object
store unavailability or an orchestrator drill. Existing local objects stay
available. The CI-only image adds distro OpenSSL/Python utilities for ephemeral
credentials/port selection; local versions/licensing evidence are recorded in
the fixture document. No Docker image build or remote CI result is claimed here.
Linked ecology, maturation, global census/cuts, target-host costs and actual
elapsed-day soak remain open. No SQL, production adapter, biological format/rule,
golden, live database, World0 or deployment changed. `Build_Errors.txt` and unrelated
local work remain untracked; prior captured CI runner failures remain separate.
## 2026-10-18 - M12.4j atomic paired biological publication and recovery
ADR-0149/schema 49 commits both exact bundle-10 no-tick results and one immutable
+10 -4
View File
@@ -9,8 +9,9 @@ distributed tooling. No custom hashing/cryptography is introduced.
## Local isolated verification
Linux prerequisites: GCC14, CMake>=3.28, Ninja, make, bison, flex, curl, bzip2,
OpenSSL development headers and zlib development headers. Optional clang19 tools
support analysis/ASan. Run as a normal user, not root:
OpenSSL development headers, the OpenSSL CLI, Python 3 (standard library only)
and zlib development headers. Optional clang19 tools support analysis/ASan/TSan.
Run as a normal user, not root:
```sh
bash tools/prepare-postgres.sh
@@ -18,13 +19,18 @@ cmake --preset release -B build/postgres -DCMAKE_CXX_COMPILER=g++-14 \
-DEVO_BUILD_POSTGRES=ON -DPostgreSQL_ROOT="$PWD/.cache/postgres/install"
cmake --build build/postgres --target evo-postgres-tests --parallel 2
bash tools/ci/postgres-check.sh "$PWD/build/postgres/evo-postgres-tests"
# Focused raw/cohort migration faults over real loopback TLS:
bash tools/ci/postgres-check.sh --migration-faults "$PWD/build/postgres/evo-postgres-tests"
```
The test script overrides all connection coordinates and creates a new private
0700 /tmp directory, private Unix socket and disposable local cluster. It creates
the `evo_runtime_test` group and the `evo_runtime_login` member, then runs fencing
tests as that real runtime login. TCP is off;
local trust applies only inside that private directory. Cluster stops on exit;
tests as that real runtime login. Ordinary fixtures disable TCP; local trust
applies only inside that private directory. The migration-fault mode creates a
separate private cluster with ephemeral loopback TLS for the runtime login only;
the full and sanitizer paths include it. See [scope and prerequisites](../engineering/M12_MIGRATION_TRANSPORT_FAULTS.md).
Clusters stop on exit;
logs/data are retained there for diagnosis. It never uses production PG credentials
or deletes a pre-existing database. Do not point this script at a shared server.
@@ -50,7 +50,9 @@ requires full recovery before readiness or advancement, including after restart.
Pointer absence alone cannot resume the old source or activate an uploaded ghost.
Unknown abort that loses to commit reconciles the actual committed head.
Independent network partitions, linked geometry/resources, world-wide cuts/census,
M12.4k's [loopback TCP/TLS matrix](../../docs/engineering/M12_MIGRATION_TRANSPORT_FAULTS.md)
qualifies request/receipt loss and independent owner expiry/replacement/heal.
Multi-host/kernel-network failures, linked geometry/resources, world-wide cuts/census,
target-host costs and actual elapsed-day ecology remain separate gates. Backup/DR
regression retains its supported fixtures and does not qualify cellular history
cloning or a new cellular restore procedure. No automatic migration or deployment.
+175
View File
@@ -0,0 +1,175 @@
#include "postgres_relay.hpp"
#include <algorithm>
#include <arpa/inet.h>
#include <array>
#include <atomic>
#include <cerrno>
#include <fcntl.h>
#include <poll.h>
#include <stdexcept>
#include <sys/socket.h>
#include <thread>
#include <unistd.h>
#include <utility>
#include <vector>
namespace evo::test {
namespace {
struct Socket {
int fd{-1};
Socket() = default;
explicit Socket(int value) : fd(value) {}
~Socket() {
if (fd >= 0)
::close(fd);
}
Socket(const Socket&) = delete;
Socket& operator=(const Socket&) = delete;
Socket(Socket&& other) noexcept : fd(std::exchange(other.fd, -1)) {}
Socket& operator=(Socket&& other) noexcept {
if (fd >= 0)
::close(fd);
fd = std::exchange(other.fd, -1);
return *this;
}
};
sockaddr_in loopback(std::uint16_t port) {
sockaddr_in address{};
address.sin_family = AF_INET;
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
address.sin_port = htons(port);
return address;
}
bool nonblocking(int fd) {
const int flags = ::fcntl(fd, F_GETFL);
return flags >= 0 && ::fcntl(fd, F_SETFL, flags | O_NONBLOCK) == 0;
}
bool receive(int fd, std::vector<char>& queued, bool drop, std::atomic<std::uint64_t>& discarded) {
if (queued.size() >= 65536)
return true;
std::array<char, 8192> bytes{};
const auto count = ::recv(fd, bytes.data(), std::min(bytes.size(), 65536 - queued.size()), 0);
if (count < 0)
return errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR;
if (count == 0)
return false;
if (drop)
discarded.fetch_add(static_cast<std::uint64_t>(count));
else
queued.insert(queued.end(), bytes.begin(), bytes.begin() + count);
return true;
}
bool transmit(int fd, std::vector<char>& queued) {
if (queued.empty())
return true;
const auto count = ::send(fd, queued.data(), queued.size(), MSG_NOSIGNAL);
if (count < 0)
return errno == EAGAIN || errno == EWOULDBLOCK || errno == EINTR;
if (count == 0)
return false;
queued.erase(queued.begin(), queued.begin() + count);
return true;
}
} // namespace
struct PostgresRelay::Impl {
Socket listener;
std::uint16_t port{}, backend{};
std::atomic<bool> requests{}, responses{}, failed{};
std::atomic<std::uint64_t> request_bytes{}, response_bytes{};
std::jthread worker;
explicit Impl(std::uint16_t target)
: listener(::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0)), backend(target) {
auto address = loopback(0);
socklen_t size = sizeof(address);
if (target == 0 || listener.fd < 0 ||
::bind(listener.fd, reinterpret_cast<const sockaddr*>(&address), sizeof(address)) !=
0 ||
::listen(listener.fd, 8) != 0 || !nonblocking(listener.fd) ||
::getsockname(listener.fd, reinterpret_cast<sockaddr*>(&address), &size) != 0)
throw std::runtime_error("private PostgreSQL relay listener");
port = ntohs(address.sin_port);
worker = std::jthread([this](const std::stop_token& stop) {
try {
run(stop);
} catch (...) {
failed = true;
}
});
}
void run(const std::stop_token& stop) {
Socket client, server;
std::vector<char> upstream, downstream;
while (!stop.stop_requested()) {
if (requests.load()) {
request_bytes.fetch_add(upstream.size());
upstream.clear();
}
if (responses.load()) {
response_bytes.fetch_add(downstream.size());
downstream.clear();
}
std::array<pollfd, 3> fds{
{{listener.fd, static_cast<short>(client.fd < 0 ? POLLIN : 0), 0},
{client.fd, static_cast<short>(POLLIN | (downstream.empty() ? 0 : POLLOUT)), 0},
{server.fd, static_cast<short>(POLLIN | (upstream.empty() ? 0 : POLLOUT)), 0}}};
const auto count = ::poll(fds.data(), fds.size(), 10);
if (count < 0) {
if (errno == EINTR)
continue;
failed = true;
return;
}
if (fds[0].revents & POLLIN) {
client = Socket(::accept4(listener.fd, nullptr, nullptr, SOCK_CLOEXEC));
server = Socket(::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0));
const auto target = loopback(backend);
if (client.fd < 0 || server.fd < 0 ||
::connect(server.fd, reinterpret_cast<const sockaddr*>(&target),
sizeof(target)) != 0 ||
!nonblocking(client.fd) || !nonblocking(server.fd)) {
failed = true;
return;
}
}
bool alive = true;
if (fds[1].revents & POLLIN)
alive = receive(client.fd, upstream, requests.load(), request_bytes);
if (alive && (fds[2].revents & POLLIN))
alive = receive(server.fd, downstream, responses.load(), response_bytes);
if (alive && (fds[1].revents & POLLOUT))
alive = transmit(client.fd, downstream);
if (alive && (fds[2].revents & POLLOUT))
alive = transmit(server.fd, upstream);
if (!alive || (fds[1].revents & (POLLERR | POLLHUP | POLLNVAL)) ||
(fds[2].revents & (POLLERR | POLLHUP | POLLNVAL))) {
client = Socket{};
server = Socket{};
upstream.clear();
downstream.clear();
}
}
}
};
PostgresRelay::PostgresRelay(std::uint16_t backend_port)
: impl_(std::make_unique<Impl>(backend_port)) {}
PostgresRelay::~PostgresRelay() = default;
std::string PostgresRelay::port() const {
return std::to_string(impl_->port);
}
void PostgresRelay::drop_requests(bool value) {
impl_->requests = value;
}
void PostgresRelay::drop_responses(bool value) {
impl_->responses = value;
}
std::uint64_t PostgresRelay::dropped_requests() const {
return impl_->request_bytes.load();
}
std::uint64_t PostgresRelay::dropped_responses() const {
return impl_->response_bytes.load();
}
bool PostgresRelay::healthy() const {
return !impl_->failed.load();
}
} // namespace evo::test
+26
View File
@@ -0,0 +1,26 @@
#pragma once
// Test-only opaque TCP relay. TLS remains end-to-end between libpq and PostgreSQL.
#include <cstdint>
#include <memory>
#include <string>
namespace evo::test {
class PostgresRelay {
public:
explicit PostgresRelay(std::uint16_t backend_port);
~PostgresRelay();
PostgresRelay(const PostgresRelay&) = delete;
PostgresRelay& operator=(const PostgresRelay&) = delete;
[[nodiscard]] std::string port() const;
void drop_requests(bool);
void drop_responses(bool);
[[nodiscard]] std::uint64_t dropped_requests() const;
[[nodiscard]] std::uint64_t dropped_responses() const;
[[nodiscard]] bool healthy() const;
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace evo::test
+264
View File
@@ -4,8 +4,10 @@
#include "evo/persistence/sha256.hpp"
#include "evo/server/cellular_region_host.hpp"
#include "fixtures/cellular_ecology.hpp"
#include "fixtures/postgres_relay.hpp"
#include <barrier>
#include <charconv>
#include <csignal>
#include <cstdlib>
#include <filesystem>
@@ -96,6 +98,73 @@ struct Fixture {
head.head.end_tick + count};
}
};
std::uint16_t migration_backend_port() {
const auto port = environment("EVO_MIGRATION_TCP_PORT");
unsigned value{};
const auto [end, error] = std::from_chars(port.data(), port.data() + port.size(), value);
require(error == std::errc{} && end == port.data() + port.size() && value > 0 && value <= 65535,
"invalid private migration TLS port");
return static_cast<std::uint16_t>(value);
}
PgOptions migration_tls_options(const test::PostgresRelay& relay) {
return {"localhost",
"127.0.0.1",
relay.port(),
environment("PGDATABASE"),
environment("EVO_PG_RUNTIME_USER"),
{},
environment("EVO_MIGRATION_CA")};
}
std::unique_ptr<Postgres> migration_tls_connection(const test::PostgresRelay& relay) {
auto db = take(Postgres::connect(migration_tls_options(relay), 2s),
"migration verify-full TLS connection");
take(db->query("SELECT 1", {}, 2s), "migration TLS warmup");
return db;
}
void migration_transport_commit(Fixture& f, const CellularMigrationCommitPrepared& prepared,
RegionKey sk, RegionKey dk, const LeaseGrant& sl,
const LeaseGrant& dl, const RegionHead& source_before,
const RegionHead& destination_before) {
{
test::PostgresRelay relay(migration_backend_port());
auto wrong = migration_tls_options(relay);
wrong.host = "untrusted.evolution.test";
require(!Postgres::connect(wrong, 1s), "relay bypassed TLS hostname verification");
}
{
test::PostgresRelay relay(migration_backend_port());
auto db = migration_tls_connection(relay);
relay.drop_requests(true);
const auto lost = retry_cellular_migration_commit(*db, sl, dl, prepared, 300ms);
require(
!lost && lost.error().kind == PublicationError::unknown &&
lost.error().prepared.source_manifest == prepared.source_manifest &&
lost.error().prepared.destination_manifest == prepared.destination_manifest &&
relay.dropped_requests() > 0 && relay.healthy() &&
take(load_region_checkpoint(*f.db, sk, 5s), "lost request source head").head ==
source_before &&
take(load_region_checkpoint(*f.db, dk, 5s), "lost request destination head").head ==
destination_before,
"a dropped TLS request changed ownership or lost the exact pair");
}
{
test::PostgresRelay relay(migration_backend_port());
auto db = migration_tls_connection(relay);
relay.drop_responses(true);
const auto lost = retry_cellular_migration_commit(*db, sl, dl, prepared, 500ms);
require(
!lost && lost.error().kind == PublicationError::unknown &&
lost.error().prepared.source_manifest == prepared.source_manifest &&
lost.error().prepared.destination_manifest == prepared.destination_manifest &&
relay.dropped_responses() > 0 && relay.healthy() &&
take(load_region_checkpoint(*f.db, sk, 5s), "lost reply source head").manifest ==
prepared.source_manifest &&
take(load_region_checkpoint(*f.db, dk, 5s), "lost reply destination head")
.manifest == prepared.destination_manifest,
"lost TLS receipt was treated as rollback or committed only one head");
}
std::puts("migration TLS request/reply loss and hostname verification passed");
}
void funded_source(Fixture& f) {
f.build.fill(0xDA);
const std::array<std::string, 1> build{hexadecimal(f.build)};
@@ -746,6 +815,196 @@ void boundary_upgrades(Fixture& f) {
require(f.raw->format_version() == cellular_lives_migration_format,
"reverse transition downgraded accounting owner");
}
void migration_owner_partitions(Fixture& f, RegionKey destination_key,
FilesystemObjectStore& destination_objects,
LeaseGrant& destination_lease,
CommittedCheckpoint& destination_head, std::uint64_t genome) {
if (!std::getenv("EVO_MIGRATION_TCP_PORT"))
return;
const std::array<Sha256Digest, 1> builds{f.build};
for (unsigned lost_source = 0; lost_source < 2; ++lost_source) {
test::PostgresRelay source_relay(migration_backend_port()),
destination_relay(migration_backend_port());
const auto make_host = [&](RegionKey key, const test::PostgresRelay& relay) {
auto objects = take(FilesystemObjectStore::open(environment("PGHOST"), key,
max_cellular_population_bytes),
"partition owner objects");
CellularHostConfig config{key, 1, {}, f.build, {f.build}};
config.lease = {3000ms, 400ms, 300ms, 150ms, 250ms};
const auto options = migration_tls_options(relay);
return take(CellularRegionHost::create(
config, [options] { return Postgres::connect(options, 500ms); },
std::move(objects)),
"partition owner host");
};
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release source to TLS owner");
take(release_region_lease(*f.db, destination_key, destination_lease, 5s),
"release destination to TLS owner");
auto source_host = make_host(f.key, source_relay),
destination_host = make_host(destination_key, destination_relay);
require(source_host->step().ready && destination_host->step().ready,
"TLS owners before freeze");
const auto sl = take(source_host->authority(), "TLS source fence"),
dl = take(destination_host->authority(), "TLS destination fence");
const auto source_bytes = source_host->state()->snapshot(),
destination_bytes = destination_host->state()->snapshot();
const auto source_before = f.head, destination_before = destination_head;
const CellularMigrationRequest request{
f.id(),
{f.key.region, f.head.head.commit, f.id(), f.head.head.digest},
{destination_key.region, destination_head.head.commit, f.id(),
destination_head.head.digest},
genome};
source_relay.drop_responses(true);
const auto unknown_prepare =
source_host->prepare_migration_freeze(destination_head, destination_objects, request);
require(!unknown_prepare &&
unknown_prepare.error().publication == PublicationError::unknown &&
!source_host->status().ready && source_relay.dropped_responses() > 0,
"lost preparation receipt did not pause source ownership");
const auto frozen =
take(take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, sl, 5s),
"independent committed preparation observation"),
"missing preparation after TLS reply loss");
source_relay.drop_responses(false);
const auto reconcile_source_until = LeaseClock::now() + 2s;
while (!source_host->status().migration_frozen &&
LeaseClock::now() < reconcile_source_until) {
source_host->step();
destination_host->step();
std::this_thread::sleep_for(25ms);
}
require(source_host->status().migration_frozen && !source_host->status().ready,
"source lost the exact prepared request after TLS reconnect");
destination_relay.drop_responses(true);
const auto unknown_stage =
destination_host->stage_migration(f.head, *f.objects, frozen.trace);
require(!unknown_stage && unknown_stage.error().publication == PublicationError::unknown &&
!destination_host->status().ready &&
destination_relay.dropped_responses() > 0 &&
take(load_cellular_migration_stage(*f.db, destination_objects, destination_key,
dl, 5s),
"independent committed stage observation")
.has_value(),
"lost stage receipt activated the ghost or lost retained work");
destination_relay.drop_responses(false);
const auto reconcile_destination_until = LeaseClock::now() + 2s;
while (!destination_host->status().migration_frozen &&
LeaseClock::now() < reconcile_destination_until) {
source_host->step();
destination_host->step();
std::this_thread::sleep_for(25ms);
}
require(source_host->status().migration_frozen &&
destination_host->status().migration_frozen,
"TLS owners failed exact stage reconciliation");
auto& isolated = lost_source ? source_host : destination_host;
auto& healthy = lost_source ? destination_host : source_host;
auto& relay = lost_source ? source_relay : destination_relay;
const auto isolated_key = lost_source ? f.key : destination_key;
auto& replacement = lost_source ? f.lease : destination_lease;
const auto healthy_key = lost_source ? destination_key : f.key;
auto& healthy_lease = lost_source ? destination_lease : f.lease;
relay.drop_requests(true);
const auto expire_until = LeaseClock::now() + 3500ms;
while (LeaseClock::now() < expire_until) {
isolated->step();
healthy->step();
std::this_thread::sleep_for(25ms);
}
require(!isolated->status().ready && !isolated->authority().has_value() &&
!isolated->advance_unattended(64) && relay.dropped_requests() > 0 &&
healthy->status().phase == HostPhase::running &&
healthy->status().epoch == (lost_source ? dl.epoch : sl.epoch) &&
healthy->status().migration_frozen,
"isolated owner retained authority or healthy peer released a frozen ghost");
replacement = take(acquire_region_lease(*f.db, isolated_key, f.build, 60000ms, 5s),
"replacement after actual TLS partition expiry");
require(replacement.epoch > (lost_source ? sl.epoch : dl.epoch),
"replacement reused partitioned epoch");
if (lost_source)
require(take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, replacement, 5s),
"replacement source retains freeze")
.has_value(),
"source lease expiry implicitly aborted");
else
require(take(load_cellular_migration_stage(*f.db, destination_objects, destination_key,
replacement, 5s),
"replacement destination retains stage")
.has_value(),
"destination lease expiry activated ghost");
const auto stale =
publish_cellular_migration(*f.db, *f.objects, destination_objects, sl, dl, f.head,
destination_head, frozen.trace, f.build, 5s);
require(!stale && stale.error().store.kind == LeaseStoreError::fenced &&
take(load_region_checkpoint(*f.db, f.key, 5s), "partition source head").head ==
source_before.head &&
take(load_region_checkpoint(*f.db, destination_key, 5s),
"partition destination head")
.head == destination_before.head,
"stale partitioned fence published a migration half");
healthy->step();
const auto current_source = lost_source ? replacement : sl;
if (lost_source) {
test::PostgresRelay abort_relay(migration_backend_port());
auto db = migration_tls_connection(abort_relay);
abort_relay.drop_responses(true);
const auto unknown =
abort_cellular_migration_freeze(*db, f.key, current_source, request.request, 300ms);
require(!unknown && unknown.error().kind == LeaseStoreError::unknown &&
abort_relay.dropped_responses() > 0 && abort_relay.healthy(),
"lost replacement abort receipt was not unknown");
take(abort_cellular_migration_freeze(*f.db, f.key, current_source, request.request, 5s),
"exact replacement abort reconciliation");
} else {
source_relay.drop_responses(true);
const auto unknown = source_host->abort_migration_freeze(request.request);
require(!unknown && !source_host->status().ready &&
source_relay.dropped_responses() > 0,
"lost owning-source abort receipt resumed the region");
source_relay.drop_responses(false);
}
require(!take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, current_source, 5s),
"independent abort source observation")
.has_value(),
"lost abort receipt reopened source stock");
const auto resume_until = LeaseClock::now() + 2s;
while (!healthy->step().ready && LeaseClock::now() < resume_until)
std::this_thread::sleep_for(25ms);
require(healthy->status().ready && healthy->state()->snapshot() ==
(lost_source ? destination_bytes : source_bytes),
"healthy owner resumed different stock after abort");
relay.drop_requests(false);
isolated->step();
require(!isolated->status().ready && !isolated->advance_unattended(64),
"healed stale owner resumed despite a replacement fence");
const auto isolated_manifest =
take(decode_manifest(lost_source ? f.head.manifest : destination_head.manifest),
"isolated head profile");
const auto& objects = lost_source ? *f.objects : destination_objects;
std::vector<std::uint8_t> recovered;
if (isolated_manifest.profile == CheckpointProfile::cellular_population_v1)
recovered = take(recover_population_checkpoint(*f.db, objects, isolated_key,
replacement, builds, 5s),
"replacement population replay")
.state->state()
.snapshot();
else
recovered = take(recover_cellular_checkpoint(*f.db, objects, isolated_key, replacement,
builds, 5s),
"replacement raw replay")
.state->snapshot();
require(recovered == (lost_source ? source_bytes : destination_bytes) &&
source_relay.healthy() && destination_relay.healthy(),
"replacement or relay lost exact stock");
source_host->drain();
destination_host->drain();
healthy_lease = take(acquire_region_lease(*f.db, healthy_key, f.build, 60000ms, 5s),
"return healthy TLS owner authority");
}
std::puts("migration independent TLS owner partitions, expiry, replacement and heal passed");
}
void migration_publication(Fixture& f, const RegionKey destination_key,
FilesystemObjectStore& destination_objects,
LeaseGrant& destination_lease, CommittedCheckpoint& destination_head) {
@@ -920,6 +1179,9 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
take(load_region_checkpoint(*f.db, dk, 5s), "killed pair destination").head ==
destination_before.head,
"pre-transaction paired death changed a head");
if (direction == 0 && std::getenv("EVO_MIGRATION_TCP_PORT"))
migration_transport_commit(f, prepared, sk, dk, sl, dl, source_before.head,
destination_before.head);
if (direction == 0) {
auto locker = connection(true), timed = connection();
take(locker->query("BEGIN", {}, 5s), "pair ambiguity transaction");
@@ -1025,6 +1287,8 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
dl = take(acquire_region_lease(*f.db, dk, f.build, 60000ms, 5s),
"return paired destination authority");
}
migration_owner_partitions(f, destination_key, destination_objects, destination_lease,
destination_head, genome);
}
// Race the actual two-head transaction against abort on independent connections.
const auto prepare_pair = [&](bool reverse) {
+25 -2
View File
@@ -3,6 +3,7 @@ set -euo pipefail
# Usage: postgres-check.sh [--full|--sanitizer|--suite-only] <evo-postgres-tests>
# postgres-check.sh --life-reservations-only [<evo-postgres-tests>]
# postgres-check.sh --population-only <evo-postgres-tests>
# postgres-check.sh --migration-faults <evo-postgres-tests> (private loopback TLS)
# --full (default) is the complete qualification: the C++ suite, the backup, DR and
# upgrade process checks, then the SQL-only migration, permission and checksum checks.
# --sanitizer keeps everything that runs the binaries built beside the test binary
@@ -12,6 +13,7 @@ set -euo pipefail
root="$(cd "$(dirname "$0")/../.." && pwd)"
mode=full
case "${1:-}" in
--migration-faults) mode=migration-faults; shift ;;
--population-only) mode=population; shift ;;
--full) shift ;;
--sanitizer) mode=sanitizer; shift ;;
@@ -41,9 +43,26 @@ test_dir="$(mktemp -d /tmp/evo-pg-test.XXXXXXXX)"
chmod 700 "$test_dir"
export PGHOST="$test_dir" PGPORT=55432 PGDATABASE=postgres PGUSER="$(id -un)"
unset PGHOSTADDR PGPASSWORD PGSERVICE PGSERVICEFILE
unset EVO_MIGRATION_TCP_PORT EVO_MIGRATION_CA
initdb -D "$test_dir/data" --no-locale --encoding=UTF8 --auth-local=trust --auth-host=reject > "$test_dir/init.log"
trap 'rc=$?; timing_summary "exit $rc" >&2; pg_ctl -D "$test_dir/data" -m immediate stop > /dev/null || true' EXIT
pg_ctl -D "$test_dir/data" -l "$test_dir/server.log" -o "-k $test_dir -p 55432 -c listen_addresses=''" -w start
if [ "$mode" = migration-faults ]; then
# One disposable cluster only; the owner remains Unix-socket-only. No production
# credential or adapter TLS exception. The relay forwards encrypted bytes.
openssl req -x509 -newkey rsa:2048 -nodes -days 1 -subj /CN=localhost \
-addext 'subjectAltName=DNS:localhost,IP:127.0.0.1' \
-keyout "$test_dir/server.key" -out "$test_dir/server.crt" > "$test_dir/tls.log" 2>&1
chmod 600 "$test_dir/server.key"
PGPORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1])')"
export PGPORT EVO_MIGRATION_TCP_PORT="$PGPORT" EVO_MIGRATION_CA="$test_dir/server.crt"
printf '%s\n' 'local all all trust' \
'hostssl postgres evo_runtime_login 127.0.0.1/32 trust' \
'host all all 127.0.0.1/32 reject' 'host all all ::1/128 reject' > "$test_dir/data/pg_hba.conf"
pg_ctl -D "$test_dir/data" -l "$test_dir/server.log" \
-o "-k $test_dir -p $PGPORT -c listen_addresses=127.0.0.1 -c ssl=on -c ssl_cert_file=$test_dir/server.crt -c ssl_key_file=$test_dir/server.key" -w start
else
pg_ctl -D "$test_dir/data" -l "$test_dir/server.log" -o "-k $test_dir -p 55432 -c listen_addresses=''" -w start
fi
# Runtime group role receives grants; a separate login inherits only that role.
psql -X -v ON_ERROR_STOP=1 -c 'CREATE ROLE evo_runtime_test NOLOGIN' \
-c 'CREATE ROLE evo_runtime_login LOGIN IN ROLE evo_runtime_test'
@@ -58,7 +77,7 @@ if [ "$mode" = life-reservations ]; then
bash "$root/tools/ci/life-reservations-concurrency.sh"
exit 0
fi
if [ "$mode" = population ]; then
if [ "$mode" = population ] || [ "$mode" = migration-faults ]; then
phase population
"$1" --population-only
exit 0
@@ -85,6 +104,10 @@ if [ "$mode" = suite-only ]; then
echo "postgres-check[suite-only]: $1 passed; process and SQL checks run in --full"
exit 0
fi
if [ "$mode" = full ] || [ "$mode" = sanitizer ]; then
phase migration-partitions
bash "$root/tools/ci/postgres-check.sh" --migration-faults "$1"
fi
if [ "$mode" = full ]; then
phase life-reservations
psql -X -v ON_ERROR_STOP=1 -f "$root/tools/ci/life-reservations-check.sql" > /dev/null