evo-quic-hol-measure sends the same 20 Hz state on the DATAGRAM channel and the ordered reliable stream plus 16 KiB reliable messages through the impairment relay across RTT x loss. Under 5-10 % loss the stream's p95 is 3-5.5x the datagrams' and exceeds the 250 ms target at 100 ms RTT, confirming the ADR-0090 channel split. The relay and tool request 1 ms timer resolution on Windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9.8 KiB
ADR-0090: M10 transport — QUIC with msquic and OpenSSL
Status
Accepted for M10.2 (2026-10-03). The selection, pins and build integration (10.2a) are decided here; bounded channels with the impairment relay (10.2b) and loss/head-of-line measurements (10.2c) implement and test it and may amend the channel parameters. Resolves the transport half of ADR-0017; its no-custom-cryptography rule stands.
Context
ADR-0089 fixes the budgets the transport must meet: 1,200-byte realtime payloads, superseding state separate from reliable actions, 1 s heartbeat, 5 s liveness and handshake, bounded queues. The M9 dev session is TLS 1.3 over TCP (ADR-0082), where one lost segment delays everything behind it.
Candidates surveyed 2026-10-03 (GitHub releases and licences):
| Candidate | Latest release | Licence | Fit |
|---|---|---|---|
| msquic (Microsoft) | v2.6.2, 2026-09-30 (v2.6.1 2026-08-28) | MIT | Complete QUIC: TLS 1.3, streams, RFC 9221 DATAGRAM, congestion control, PMTUD, own threads; C API; Windows and Linux |
| ngtcp2 | v1.25.0, 2026-07-26 | MIT | Protocol library; we own UDP I/O, timers, loop and a crypto backend |
| picoquic | date-tagged builds (2026-10-01) | MIT | Research-oriented, picotls |
| lsquic | v4.10.0, 2026-09-20 | MIT | Requires BoringSSL |
| quiche | 0.30.0, 2026-09-17 | BSD-2-Clause | Rust toolchain dependency |
| GameNetworkingSockets | v1.6.1, 2026-10-03 | BSD-3-Clause | Game messages and fragmentation; its own non-TLS encryption and certificates |
| Mbed TLS DTLS + owned reliability | 4.2.0 / 4.1.1 | Apache-2.0 option | DTLS 1.2 only here; we would own reliability, fragmentation and congestion |
| wolfSSL | v5.9.4 | GPL-3.0 | Excluded by licence |
Decision (maintainer, 2026-10-03)
- msquic 2.6.2 (commit 819ab74f, MIT) with its OpenSSL TLS provider on Windows
and Linux, OpenSSL 3.5.9 LTS (Apache-2.0), both from pinned, checksum-verified
release archives (
tools/msquic-pin.cmake,tools/prepare-msquic.cmake). msquic's OpenSSL submodule points at an untagged openssl-3.5 commit (453eaaa9, 2026-07-29); the official 3.5.9 release on the same branch replaces it for provenance and later fixes, and the smoke test proves the combination. - msquic builds OpenSSL statically into its shared library with TLS 1.3 only and legacy
protocols/algorithms disabled (
no-tls1_2,no-dtls,no-legacy,no-deprecatedand others in msquic's submodule build). Logging, tests, tools and perf are off; the library ismsquic.dll/libmsquic.so. Windows builds need Strawberry Perl and NASM (installed with winget on the development host, 2026-10-03;tools/windows-check.ps1reads them from the registry PATH and pinsCC/CXXtoclbecause Strawberry also puts MinGW compilers on PATH) plusnmake. - msquic always compiles its Windows XDP datapath (optional at run time), which needs the
headers of its
xdp-for-windowssubmodule: pinned at msquic's commit d372b525 (MIT), archive SHA-256 9a52e1b5..., only those headers are used. - Upstream builds itself with warnings as errors and MSVC
/analyze; MSVC 19.44 reports C28020 in unmodifiedcrypto_tls.c. As for Mbed TLS,cmake/Quic.cmakeappends/WX-(-Wno-errorelsewhere) to msquic's ownwarningsinterface; owned code keeps warnings as errors. Windows consumers also linkntdllfor msquic.h's inline address helpers. - Trust: servers present a certificate chain from environment provisioning (dev: the
ADR-0083 dev CA). Clients load the environment CA with
CaCertificateFileand use OpenSSL's verifier on every platform (USE_TLS_BUILTIN_CERTIFICATE_VALIDATION; msquic forces it on Linux, and on Windows its default platform verifier ignores the CA file), verify the server name, and never setNO_CERTIFICATE_VALIDATION. ALPNevo-m10. Known gap: OpenSSL's built-in verification also loads its default verify paths (OPENSSLDIR,SSL_CERT_FILE,SSL_CERT_DIR), so trust is the pinned CA plus those locations, not the pinned CA alone. 10.2b closes this with exact pinning: the adapter requests the peer chain (INDICATE_CERTIFICATE_RECEIVED/deferred validation) and accepts only a chain to the configured CA with the expected name, checked in owned code. Grants and session admission stay application-level (ADR-0083/10.3). - Threading: msquic calls back on its own worker threads. The transport adapter only copies bounded data into bounded queues owned by the region host's thread; callbacks never touch simulation or presentation state and never block.
- Channels (implemented in 10.2b): realtime intentions and superseding snapshots on QUIC DATAGRAM frames (unreliable, congestion-controlled, payload <= 1,200 bytes, newest-wins, never retransmitted by us); reliable actions and outcomes on bidirectional streams with bounded send/receive windows; liveness from QUIC keep-alive (1 s) and idle timeout (5 s); handshake idle timeout 5 s; congestion control msquic's default (CUBIC) unless 10.2c measurements justify BBR.
- M9 path: the TLS-over-TCP dev session remains the dev/engineering fallback; M10 features build on QUIC only.
- Impairment tests: an owned, seeded user-space UDP relay (loss, reorder, duplication, delay, jitter, MTU), no root; packet-level TCP comparison is not run.
Evidence (10.2a, 2026-10-03)
evo-quic-smoke-tests (label quic, preset quic): loopback handshake with the dev CA,
one DATAGRAM and one stream message with FIN delivered, an untrusted CA refused (TLS
alert 51) and a wrong server name refused (alert 42), on Windows MSVC 19.44 and Linux
GCC 14; clang-tidy clean; repeated runs stable. Linux standalone build 280 s; full
results in engineering/VALIDATION.md.
Alternatives considered
See the table. ngtcp2 needs more owned networking code; GameNetworkingSockets uses non-TLS cryptography; DTLS plus owned reliability carries the retransmission and congestion risk ADR-0017 warns about; Schannel would split TLS stacks by platform and exclude older Windows; prebuilt msquic binaries are not built from reviewed sources.
Consequences
Two TLS stacks exist while the M9 fallback remains (Mbed TLS in the session library,
OpenSSL inside msquic). Clean builds take about 5 minutes longer on Linux (OpenSSL).
When the client adopts QUIC, the Windows package (ADR-0086) adds msquic.dll to its
allowlist and the msquic (MIT) and OpenSSL (Apache-2.0) notices. OpenSSL and msquic
security releases update the pin, checksums and this ADR. UDP-blocked networks are
served only by the dev fallback for now.
M10.2b implementation (2026-10-03)
evo_network_quic (include/evo/network/quic_transport.hpp) provides one Endpoint
(server or client) owned by one thread that calls poll():
- Channels: realtime DATAGRAMs (payload <= min(1,200, the peer's current maximum);
inbound newest-wins per connection, outbound in-flight cap refuses) and one
client-opened bidirectional stream carrying 32-bit length-framed messages (<= 64 KiB,
once, in order). Outbound reliable bytes not yet completed and inbound bytes awaiting
poll()are bounded; inbound overflow closes the connection (overloaded), an oversized or malformed frame closes it (protocol), a second peer stream is refused. Closures carry a cause: local, peer, idle, untrusted, overloaded, protocol, transport. - Exact pinning closes the 10.2a gap: the client defers OpenSSL's verdict, receives the
peer leaf as DER and returns pending;
poll()verifies it with Mbed TLS against only the configured CA and name and completes validation. The owned check is the sole authority; OpenSSL's verdict is advisory (counted when it disagrees). Reason: dev CAs share a subject name, so with another CA inSSL_CERT_FILEOpenSSL's issuer lookup failed a genuine server; the environment must neither add nor remove trust. - Threading rule: callbacks touch state only under one mutex and never call msquic while holding it (msquic may run callbacks inline); handles are closed only by the owning thread. Mbed TLS runs only on the owning thread.
- Path MTU: msquic assumes a 1,288-byte IP MTU by default (1,260-byte IPv4 UDP
payloads), which a compliant 1,280-byte path drops.
MinimumMtuis set to 1,280; path discovery only probes upward, and the 1,200-byte datagram budget fits that path. - Settings: idle 5 s, keep-alive 1 s, handshake 5 s, flow-control windows equal to the inbound reliable bound, one peer bidirectional stream, no unidirectional streams.
- Impairment relay (
tools/network/impair, tests and measurements only): seeded loss, duplication, reordering, corruption, delay/jitter, payload-size limit and live reconfiguration (outage).
M10.2c measurements (2026-10-03)
evo-quic-hol-measure sent the same 20 Hz, 300-byte state on the DATAGRAM channel and
the ordered reliable stream, plus 16 KiB reliable messages, through the relay across
RTT {2, 50, 100} ms x loss {0, 1, 5, 10} % with 2 % reordering and 1 % duplication, on
Windows and Linux (results).
- The channel split is confirmed. Under 5-10 % loss the ordered stream's p95 latency is 3-5.5x the datagram channel's (head-of-line blocking). At 100 ms RTT it exceeds the 250 ms responsiveness target (10 % loss: 258 / 310 ms on Linux / Windows), while datagram p95 stays <= 67 ms and newest-state staleness p99 <= 195 ms. Superseding state and intentions must not use the ordered stream.
- Reliable actions and outcomes fit their budget: the slowest 16 KiB message took 510 ms, inside the 1.5 s final-outcome budget; no reliable message was lost.
- Congestion control stays CUBIC: nothing measured here motivates BBR. Bandwidth limits, competing flows and 20+ clients are 10.9 work.
- Limits: user-space relay on one host (no kernel TCP, netem or real paths), development hardware. The relay and tool request 1 ms timer resolution on Windows.