Add M10.9c-2 lease loss under load process test
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 3m26s
Foundation / checks (tsan) (push) Successful in 8m14s
Foundation / checks (checks) (push) Successful in 32m35s
Foundation / checks (postgres) (push) Successful in 46m6s
Foundation / server-image (push) Successful in 4s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m37s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 3m26s
Foundation / checks (tsan) (push) Successful in 8m14s
Foundation / checks (checks) (push) Successful in 32m35s
Foundation / checks (postgres) (push) Successful in 46m6s
Foundation / server-image (push) Successful in 4s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m37s
ADR-0097. 24 evo-load bots against a real evo-server on a fixed loopback port: a SIGKILL (the successor waits out the lease) and then a SIGTERM drain, each followed by a successor under a new epoch. Every bot logs in again within 15 s of the successor being ready (development reference: 5.0 s after SIGKILL, 2.4 s after drain), with 48 drops and no failed logins counted over the hold. Scope narrowed: no standalone raw-UDP sender; abuse stays in-process or on loopback, and the per-connection datagram cap's eviction cost is analysed in the ADR. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -12,7 +12,7 @@ spec:
|
||||
env:
|
||||
- {name: REGION_SESSION, value: "dev"}
|
||||
- {name: REGION_SESSION_LISTEN, value: "0.0.0.0:7400"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261008T015315-20832:1791427995"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261008T100924-e9f8:1791455364"}
|
||||
- {name: REGION_M10_MAX_GRANT_LIFETIME, value: "14400"}
|
||||
ports:
|
||||
# QUIC (ADR-0090): UDP only; nothing listens on TCP 7400.
|
||||
|
||||
+4
-3
@@ -546,9 +546,10 @@ runs qualify): 10.9a per-tick work measurement, load governor (refuse logins, th
|
||||
replication with hysteresis) and its metrics done; 10.9b evo-load multi-client tool
|
||||
(per-client distributions, starvation check, step verdicts, server gauges) and its process
|
||||
smoke done; 10.9c-1 session fuzz target and smoke, in-process Hello and datagram floods,
|
||||
slow peer and connect/disconnect storm with honest-service checks done; 10.9c-2 lease loss
|
||||
with 20+ bots and a live raw-UDP abuse sender, and 10.9d dev-cluster drill harness and
|
||||
reference results follow.
|
||||
slow peer and connect/disconnect storm with honest-service checks done; 10.9c-2 SIGKILL and
|
||||
drain lease loss with 24 bots (all back within 5 s of the successor) done; no standalone
|
||||
raw-UDP sender (abuse stays in-process/loopback; datagram-cap eviction cost analysed).
|
||||
10.9d dev-cluster drill harness and reference results follow.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+5
-2
@@ -120,8 +120,11 @@ refusing at 32 (work p95 about 14 ms), with responses about 103 ms and outcomes
|
||||
which an honest session kept every snapshot and receipt.
|
||||
|
||||
`evo-session-probe` can now log in from a renewal's grant bundle. A dev login was welcomed
|
||||
and its intention admitted (the dev region does not simulate outside drills). Next:
|
||||
10.9c-2, lease loss with 20+ bots and a live raw-UDP sender.
|
||||
and its intention admitted (the dev region does not simulate outside drills). M10.9c-2
|
||||
kills and drains a real evo-server under 24 `evo-load` bots. All of them log in to the
|
||||
successor within 5.0 s (SIGKILL, after a 3.5 s lease wait) or 2.4 s (drain), with no failed
|
||||
logins, and responses stay at 106 ms p95. Next: 10.9d, the evolution-dev drill harness and
|
||||
the reference-worker run.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -195,3 +195,30 @@ snapshots.
|
||||
and a live raw-UDP sender of mutated datagrams. Our client endpoint caps its own send
|
||||
rate, so it cannot load the server's per-connection datagram cap (64, newest wins, with
|
||||
an O(queue) eviction scan); the raw sender measures that.
|
||||
|
||||
## 10.9c-2 authority loss under load (2026-10-08)
|
||||
- **Lease loss under load** (`region_m10_session_tests.cpp`): 24 `evo-load` bots on a real
|
||||
evo-server (loopback, fixed port). First the server is SIGKILLed, so its lease must
|
||||
expire before a successor runs. Then it is drained with SIGTERM: sessions close with
|
||||
`shutdown` and the lease is released. Each time the successor takes the same port under
|
||||
a new epoch, and every bot must log in again within 15 s of it being ready.
|
||||
|
||||
Development reference:
|
||||
|
||||
| Loss | Successor ready | All 24 bots back after ready |
|
||||
|---|---|---|
|
||||
| SIGKILL | 3.5 s | 5.0 s |
|
||||
| SIGTERM drain | 0.18 s | 2.4 s |
|
||||
|
||||
Over the 100 s hold the bots made 72 logins (24 + 2 × 24) and counted 48 drops, with no
|
||||
failed or refused login, and response p95 stayed at 106 ms. After a SIGKILL, clients only
|
||||
notice the loss through the QUIC idle timeout (5 s), which dominates the SIGKILL figure.
|
||||
- **Scope narrowed:** no standalone tool sends raw UDP traffic at a network endpoint.
|
||||
Abuse testing stays in-process or on loopback inside the test suite (10.9c-1). The
|
||||
server's per-connection datagram cap is analysed here instead of being measured with a
|
||||
flood tool. Each connection may hold 64 queued datagrams (newest wins). Evicting the
|
||||
oldest scans the endpoint's event queue (at most 4096 events), which happens only when
|
||||
one connection queues more than 64 datagrams between two server polls (normally under a
|
||||
millisecond apart). That cost is bounded per datagram but can add up at very high rates
|
||||
from one peer. Making eviction O(1) without letting tombstones outgrow the queue bound is
|
||||
recorded as a follow-up, not done.
|
||||
|
||||
@@ -5063,3 +5063,25 @@ Executed:
|
||||
|
||||
Not done (10.9c-2): lease loss with 20 or more bots, and a live raw-UDP sender of mutated
|
||||
datagrams that loads the server's per-connection datagram cap.
|
||||
|
||||
## 2026-10-08 — M10.9c-2 lease loss under load (ADR-0097)
|
||||
|
||||
Implemented:
|
||||
- `region_m10_session_tests` `lease_loss_under_load`: 24 `evo-load` bots against a real
|
||||
evo-server on a fixed loopback port. It runs a SIGKILL and then a SIGTERM drain, each
|
||||
followed by a successor under a new epoch.
|
||||
- The test fixture's `Setup` gains a `listen` setting (default `127.0.0.1:0`).
|
||||
- Scope: no standalone raw-UDP sender was built. The per-connection datagram cap's eviction
|
||||
cost is analysed in ADR-0097, and making eviction O(1) is a follow-up.
|
||||
|
||||
Results (development reference, WSL, Release):
|
||||
- **SIGKILL:** the successor was ready after 3.5 s, and all 24 bots were back 5.0 s
|
||||
later.
|
||||
- **Drain:** the successor was ready after 0.18 s, and all 24 bots were back 2.4 s later.
|
||||
- **Bots over the 100 s hold:** 72 logins, 48 drops, no failed or refused logins; response
|
||||
p95 106 ms; login p99 62 ms.
|
||||
|
||||
Executed:
|
||||
- **Linux GCC 14:** `postgres-check.sh` with QUIC, all suites.
|
||||
- **Linux Clang 19:** clang-tidy 19 on the M10 process tests (clean); clang-format 19.
|
||||
- **Windows:** not applicable (the PostgreSQL process tests are Linux only).
|
||||
|
||||
@@ -100,10 +100,26 @@ std::optional<std::uint64_t> metric(std::uint16_t port, const std::string& serie
|
||||
return std::stoull(raw.substr(at + series.size() + 2));
|
||||
}
|
||||
|
||||
// A UDP port that was free a moment ago.
|
||||
std::uint16_t free_port() {
|
||||
const int fd = ::socket(AF_INET, SOCK_DGRAM | SOCK_CLOEXEC, 0);
|
||||
expect(fd >= 0, "socket");
|
||||
sockaddr_in a{};
|
||||
a.sin_family = AF_INET;
|
||||
a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
||||
socklen_t size = sizeof a;
|
||||
const bool ok = ::bind(fd, reinterpret_cast<const sockaddr*>(&a), sizeof a) == 0 &&
|
||||
::getsockname(fd, reinterpret_cast<sockaddr*>(&a), &size) == 0;
|
||||
::close(fd);
|
||||
expect(ok, "free port");
|
||||
return ntohs(a.sin_port);
|
||||
}
|
||||
|
||||
// One region's credentials, issuer key and server process arguments.
|
||||
struct Setup {
|
||||
ServerRegion region;
|
||||
std::string ca_pem;
|
||||
std::string listen{"127.0.0.1:0"}; // A fixed port lets clients reconnect after a restart.
|
||||
std::optional<GrantSigningKey> issuer;
|
||||
explicit Setup(std::uint64_t r, std::uint8_t genesis) : region{provision(r, genesis)} {
|
||||
const auto made = generate_dev_credentials(DeploymentEnvironment::dev, name, 1,
|
||||
@@ -133,7 +149,7 @@ struct Setup {
|
||||
"--ops-listen",
|
||||
"127.0.0.1:0",
|
||||
"--m10-listen",
|
||||
"127.0.0.1:0",
|
||||
listen,
|
||||
"--m10-cert",
|
||||
(region.root / "server.pem").string(),
|
||||
"--m10-key",
|
||||
@@ -523,10 +539,105 @@ void load_smoke() {
|
||||
server.child->signal(SIGTERM);
|
||||
CHECK(server.child->wait_exit(30s) == 0);
|
||||
}
|
||||
// M10.9c-2 (ADR-0097): region authority lost under load. 24 evo-load bots on a real
|
||||
// evo-server; the server is SIGKILLed (its lease must expire before a successor runs), then
|
||||
// later drained with SIGTERM (sessions close with shutdown). Each time the successor takes
|
||||
// the same port under a new epoch and every bot logs in again within 15 s of it being ready,
|
||||
// and the bots count the drops. Loopback only; the bound is a development reference.
|
||||
void lease_loss_under_load() {
|
||||
Setup s{153, 0xC7};
|
||||
s.listen = "127.0.0.1:" + std::to_string(free_port());
|
||||
auto server = std::make_unique<Server>(s);
|
||||
constexpr std::uint64_t bots = 24;
|
||||
const auto tool = fs::path(env("EVO_SERVER_BIN")).parent_path() / "evo-load";
|
||||
Child load{{tool.string(),
|
||||
"--connect",
|
||||
"127.0.0.1:" + std::to_string(server->port),
|
||||
"--server-name",
|
||||
name,
|
||||
"--ca",
|
||||
(s.region.root / "ca.pem").string(),
|
||||
"--issuer-key",
|
||||
(s.region.root / "issuer.key").string(),
|
||||
"--world",
|
||||
"7",
|
||||
"--frame",
|
||||
std::to_string(region_identity.frame),
|
||||
"--region",
|
||||
"153",
|
||||
"--steps",
|
||||
std::to_string(bots),
|
||||
"--hold",
|
||||
"100",
|
||||
"--warmup",
|
||||
"2",
|
||||
"--ramp-ms",
|
||||
"50",
|
||||
"--threads",
|
||||
"4",
|
||||
"--lifetime",
|
||||
"900"}};
|
||||
const auto pump = [&] {
|
||||
server->pump();
|
||||
load.pump(5ms);
|
||||
};
|
||||
// Milliseconds until the region reports `n` sessions, or nothing within `limit`.
|
||||
const auto sessions_after =
|
||||
[&](std::chrono::milliseconds limit) -> std::optional<std::chrono::milliseconds> {
|
||||
const auto started = Clock::now();
|
||||
for (int i = 0; Clock::now() - started < limit; ++i) {
|
||||
pump();
|
||||
if (i % 20 == 0 && metric(server->ops, "evo_m10_sessions") == bots)
|
||||
return std::chrono::duration_cast<std::chrono::milliseconds>(Clock::now() -
|
||||
started);
|
||||
}
|
||||
return std::nullopt;
|
||||
};
|
||||
CHECK(sessions_after(60s).has_value());
|
||||
// A successor on the same port; how long until it is ready and every bot is back.
|
||||
const auto successor = [&](const char* how) {
|
||||
const auto lost = Clock::now();
|
||||
server = std::make_unique<Server>(s); // Waits out the old lease, then "ready 1".
|
||||
const auto ready =
|
||||
std::chrono::duration_cast<std::chrono::milliseconds>(Clock::now() - lost);
|
||||
const auto back = sessions_after(15s);
|
||||
std::printf("lease loss %s ready_ms=%lld all_back_ms=%lld\n", how,
|
||||
static_cast<long long>(ready.count()),
|
||||
static_cast<long long>(back ? back->count() : -1));
|
||||
return back.has_value();
|
||||
};
|
||||
server->kill();
|
||||
CHECK(successor("sigkill"));
|
||||
server->child->signal(SIGTERM);
|
||||
CHECK(server->child->wait_exit(30s) == 0);
|
||||
CHECK(successor("sigterm"));
|
||||
// The bots measured both losses within their hold.
|
||||
int code = -1;
|
||||
for (const auto end = Clock::now() + 180s; Clock::now() < end && code < 0;) {
|
||||
pump();
|
||||
if (load.printed("load result "))
|
||||
code = load.wait_exit(30s);
|
||||
}
|
||||
std::string step;
|
||||
for (const auto& line : load.lines()) {
|
||||
if (starts(line, "load step ") || starts(line, "load response_ms") ||
|
||||
starts(line, "load login_ms"))
|
||||
std::printf("%s\n", line.c_str());
|
||||
if (starts(line, "load step "))
|
||||
step = line;
|
||||
}
|
||||
CHECK(code == 0 && starts(step, "load step N=24 established=24 "));
|
||||
const auto drops_at = step.find(" drops=");
|
||||
CHECK(drops_at != std::string::npos &&
|
||||
std::stoull(step.substr(drops_at + 7)) >= 2 * bots); // Both losses dropped every bot.
|
||||
server->child->signal(SIGTERM);
|
||||
CHECK(server->child->wait_exit(30s) == 0);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_m10_session_tests() {
|
||||
end_to_end();
|
||||
load_smoke();
|
||||
lease_loss_under_load();
|
||||
std::puts("region m10 session integration passed");
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user