Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s

ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md)
runs the evolution-dev reference qualification: synthetic inputs on for
the run (the ADR-0073 drill-time override, restored on exit), bundle-mode
evo-load bots through the public route, and each step's hold matched with
the server's reports. evo-server logs "m10 load-report" every 10 s, since
the agent cannot reach the ops port. evo-load gains --grants FILE
--grants-per-bot C, exhausted=, hold_utc= and close_reasons=;
evo-session-probe grants gains --accounts N.

Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on
different threads verifying their first connections at once were refused
as untrusted (two per smoke run). One process-wide recursive crypto lock
now guards every network-library entry point that touches them; the
smoke went from 2/2 failing to 3/3 clean. quic-transport adds a
concurrent-connect test. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-08 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent 47b84bce74
commit ada77fd761
16 changed files with 609 additions and 70 deletions
+62
View File
@@ -5,12 +5,15 @@
#include "evo/network/dev_credentials.hpp"
#include "udp_relay.hpp"
#include <algorithm>
#include <array>
#include <atomic>
#include <chrono>
#include <cstdio>
#include <cstdlib>
#include <filesystem>
#include <fstream>
#include <functional>
#include <latch>
#include <set>
#include <source_location>
#include <stdexcept>
@@ -558,8 +561,67 @@ void garbage_and_close() {
}
} // namespace
// M10.9d (ADR-0097): client endpoints on several threads connecting at the same moment all
// connect and pass the owned pin check. Mbed TLS runs without its threading layer; evo-load
// saw two concurrent first verifications refused as `untrusted` before the shared crypto
// lock. This test covers concurrent connects; it does not reliably reproduce that race (PSA
// is already initialised here by the credential generation), which evo-load's smoke does.
void concurrent_clients() {
const auto f = credentials("concurrent");
Limits limits;
limits.max_connections = 64; // 5 rounds x 8, whatever the server has not yet reaped.
auto server = serve(f, limits);
constexpr int threads = 8;
for (int round = 0; round < 5; ++round) {
std::atomic<int> connected{0}, finished{0};
std::array<std::atomic<int>, 8> causes{};
std::latch start{threads};
std::vector<std::thread> clients;
clients.reserve(threads);
for (int t = 0; t < threads; ++t)
clients.emplace_back([&] {
start.arrive_and_wait();
auto client = dial(f, server->port());
const auto end = std::chrono::steady_clock::now() + std::chrono::seconds{10};
bool done = false;
while (!done && std::chrono::steady_clock::now() < end) {
for (const auto& e : client->poll()) {
if (e.kind == EventKind::connected) {
++connected;
done = true;
} else if (e.kind == EventKind::closed) {
causes[static_cast<std::size_t>(e.cause)] += 1;
done = true;
}
}
std::this_thread::sleep_for(std::chrono::milliseconds{1});
}
++finished;
});
while (finished < threads) {
(void)server->poll();
std::this_thread::sleep_for(std::chrono::milliseconds{1});
}
for (auto& c : clients)
c.join();
if (connected != threads) {
std::printf("concurrent round %d connected=%d", round, connected.load());
for (std::size_t i = 0; i < causes.size(); ++i)
std::printf(" cause%zu=%d", i, causes.at(i).load());
std::printf("\n");
}
CHECK(connected == threads);
for (int i = 0; i < 200; ++i) { // Let the server see the closes.
(void)server->poll();
std::this_thread::sleep_for(std::chrono::milliseconds{1});
}
}
std::filesystem::remove_all(f.dir);
}
int main() try {
framing();
concurrent_clients();
channels();
pinning();
impaired();