Add M10.9d load drill harness and serialize Mbed TLS use
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
Foundation / checks (deploy) (push) Successful in 3m24s
Foundation / checks (ecs) (push) Successful in 1m56s
Foundation / checks (client) (push) Successful in 12m18s
Foundation / checks (network) (push) Successful in 13m30s
Foundation / checks (tsan) (push) Successful in 8m0s
Foundation / checks (checks) (push) Successful in 32m39s
Foundation / checks (postgres) (push) Successful in 45m47s
Foundation / postgres-tools (push) Successful in 17s
Foundation / image (push) Successful in 2m30s
Foundation / server-image (push) Successful in 4m8s
ADR-0097. tools/ops/load-drill.sh (runbook docs/networking/M10_LOAD_DRILL.md) runs the evolution-dev reference qualification: synthetic inputs on for the run (the ADR-0073 drill-time override, restored on exit), bundle-mode evo-load bots through the public route, and each step's hold matched with the server's reports. evo-server logs "m10 load-report" every 10 s, since the agent cannot reach the ops port. evo-load gains --grants FILE --grants-per-bot C, exhausted=, hold_utc= and close_reasons=; evo-session-probe grants gains --accounts N. Finding: Mbed TLS/PSA are built without MBEDTLS_THREADING_C, and bots on different threads verifying their first connections at once were refused as untrusted (two per smoke run). One process-wide recursive crypto lock now guards every network-library entry point that touches them; the smoke went from 2/2 failing to 3/3 clean. quic-transport adds a concurrent-connect test. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -5,12 +5,15 @@
|
||||
#include "evo/network/dev_credentials.hpp"
|
||||
#include "udp_relay.hpp"
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <functional>
|
||||
#include <latch>
|
||||
#include <set>
|
||||
#include <source_location>
|
||||
#include <stdexcept>
|
||||
@@ -558,8 +561,67 @@ void garbage_and_close() {
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// M10.9d (ADR-0097): client endpoints on several threads connecting at the same moment all
|
||||
// connect and pass the owned pin check. Mbed TLS runs without its threading layer; evo-load
|
||||
// saw two concurrent first verifications refused as `untrusted` before the shared crypto
|
||||
// lock. This test covers concurrent connects; it does not reliably reproduce that race (PSA
|
||||
// is already initialised here by the credential generation), which evo-load's smoke does.
|
||||
void concurrent_clients() {
|
||||
const auto f = credentials("concurrent");
|
||||
Limits limits;
|
||||
limits.max_connections = 64; // 5 rounds x 8, whatever the server has not yet reaped.
|
||||
auto server = serve(f, limits);
|
||||
constexpr int threads = 8;
|
||||
for (int round = 0; round < 5; ++round) {
|
||||
std::atomic<int> connected{0}, finished{0};
|
||||
std::array<std::atomic<int>, 8> causes{};
|
||||
std::latch start{threads};
|
||||
std::vector<std::thread> clients;
|
||||
clients.reserve(threads);
|
||||
for (int t = 0; t < threads; ++t)
|
||||
clients.emplace_back([&] {
|
||||
start.arrive_and_wait();
|
||||
auto client = dial(f, server->port());
|
||||
const auto end = std::chrono::steady_clock::now() + std::chrono::seconds{10};
|
||||
bool done = false;
|
||||
while (!done && std::chrono::steady_clock::now() < end) {
|
||||
for (const auto& e : client->poll()) {
|
||||
if (e.kind == EventKind::connected) {
|
||||
++connected;
|
||||
done = true;
|
||||
} else if (e.kind == EventKind::closed) {
|
||||
causes[static_cast<std::size_t>(e.cause)] += 1;
|
||||
done = true;
|
||||
}
|
||||
}
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds{1});
|
||||
}
|
||||
++finished;
|
||||
});
|
||||
while (finished < threads) {
|
||||
(void)server->poll();
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds{1});
|
||||
}
|
||||
for (auto& c : clients)
|
||||
c.join();
|
||||
if (connected != threads) {
|
||||
std::printf("concurrent round %d connected=%d", round, connected.load());
|
||||
for (std::size_t i = 0; i < causes.size(); ++i)
|
||||
std::printf(" cause%zu=%d", i, causes.at(i).load());
|
||||
std::printf("\n");
|
||||
}
|
||||
CHECK(connected == threads);
|
||||
for (int i = 0; i < 200; ++i) { // Let the server see the closes.
|
||||
(void)server->poll();
|
||||
std::this_thread::sleep_for(std::chrono::milliseconds{1});
|
||||
}
|
||||
}
|
||||
std::filesystem::remove_all(f.dir);
|
||||
}
|
||||
|
||||
int main() try {
|
||||
framing();
|
||||
concurrent_clients();
|
||||
channels();
|
||||
pinning();
|
||||
impaired();
|
||||
|
||||
Reference in New Issue
Block a user