Files
aevolution/docs/adr/0090-m10-quic-transport.md
T
dingoandClaude Opus 5.5 1055cc8c8f
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m46s
Foundation / checks (network) (push) Successful in 8m16s
Foundation / checks (client) (push) Successful in 12m6s
Foundation / checks (tsan) (push) Successful in 7m46s
Foundation / checks (checks) (push) Successful in 31m39s
Foundation / checks (postgres) (push) Successful in 33m28s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m15s
Foundation / server-image (push) Successful in 18m31s
Measure M10.2c loss and head-of-line behaviour of the QUIC channels
evo-quic-hol-measure sends the same 20 Hz state on the DATAGRAM channel and the
ordered reliable stream plus 16 KiB reliable messages through the impairment relay
across RTT x loss. Under 5-10 % loss the stream's p95 is 3-5.5x the datagrams' and
exceeds the 250 ms target at 100 ms RTT, confirming the ADR-0090 channel split.
The relay and tool request 1 ms timer resolution on Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-03 17:25:38 -04:00

9.8 KiB

ADR-0090: M10 transport — QUIC with msquic and OpenSSL

Status

Accepted for M10.2 (2026-10-03). The selection, pins and build integration (10.2a) are decided here; bounded channels with the impairment relay (10.2b) and loss/head-of-line measurements (10.2c) implement and test it and may amend the channel parameters. Resolves the transport half of ADR-0017; its no-custom-cryptography rule stands.

Context

ADR-0089 fixes the budgets the transport must meet: 1,200-byte realtime payloads, superseding state separate from reliable actions, 1 s heartbeat, 5 s liveness and handshake, bounded queues. The M9 dev session is TLS 1.3 over TCP (ADR-0082), where one lost segment delays everything behind it.

Candidates surveyed 2026-10-03 (GitHub releases and licences):

Candidate Latest release Licence Fit
msquic (Microsoft) v2.6.2, 2026-09-30 (v2.6.1 2026-08-28) MIT Complete QUIC: TLS 1.3, streams, RFC 9221 DATAGRAM, congestion control, PMTUD, own threads; C API; Windows and Linux
ngtcp2 v1.25.0, 2026-07-26 MIT Protocol library; we own UDP I/O, timers, loop and a crypto backend
picoquic date-tagged builds (2026-10-01) MIT Research-oriented, picotls
lsquic v4.10.0, 2026-09-20 MIT Requires BoringSSL
quiche 0.30.0, 2026-09-17 BSD-2-Clause Rust toolchain dependency
GameNetworkingSockets v1.6.1, 2026-10-03 BSD-3-Clause Game messages and fragmentation; its own non-TLS encryption and certificates
Mbed TLS DTLS + owned reliability 4.2.0 / 4.1.1 Apache-2.0 option DTLS 1.2 only here; we would own reliability, fragmentation and congestion
wolfSSL v5.9.4 GPL-3.0 Excluded by licence

Decision (maintainer, 2026-10-03)

  • msquic 2.6.2 (commit 819ab74f, MIT) with its OpenSSL TLS provider on Windows and Linux, OpenSSL 3.5.9 LTS (Apache-2.0), both from pinned, checksum-verified release archives (tools/msquic-pin.cmake, tools/prepare-msquic.cmake). msquic's OpenSSL submodule points at an untagged openssl-3.5 commit (453eaaa9, 2026-07-29); the official 3.5.9 release on the same branch replaces it for provenance and later fixes, and the smoke test proves the combination.
  • msquic builds OpenSSL statically into its shared library with TLS 1.3 only and legacy protocols/algorithms disabled (no-tls1_2, no-dtls, no-legacy, no-deprecated and others in msquic's submodule build). Logging, tests, tools and perf are off; the library is msquic.dll / libmsquic.so. Windows builds need Strawberry Perl and NASM (installed with winget on the development host, 2026-10-03; tools/windows-check.ps1 reads them from the registry PATH and pins CC/CXX to cl because Strawberry also puts MinGW compilers on PATH) plus nmake.
  • msquic always compiles its Windows XDP datapath (optional at run time), which needs the headers of its xdp-for-windows submodule: pinned at msquic's commit d372b525 (MIT), archive SHA-256 9a52e1b5..., only those headers are used.
  • Upstream builds itself with warnings as errors and MSVC /analyze; MSVC 19.44 reports C28020 in unmodified crypto_tls.c. As for Mbed TLS, cmake/Quic.cmake appends /WX- (-Wno-error elsewhere) to msquic's own warnings interface; owned code keeps warnings as errors. Windows consumers also link ntdll for msquic.h's inline address helpers.
  • Trust: servers present a certificate chain from environment provisioning (dev: the ADR-0083 dev CA). Clients load the environment CA with CaCertificateFile and use OpenSSL's verifier on every platform (USE_TLS_BUILTIN_CERTIFICATE_VALIDATION; msquic forces it on Linux, and on Windows its default platform verifier ignores the CA file), verify the server name, and never set NO_CERTIFICATE_VALIDATION. ALPN evo-m10. Known gap: OpenSSL's built-in verification also loads its default verify paths (OPENSSLDIR, SSL_CERT_FILE, SSL_CERT_DIR), so trust is the pinned CA plus those locations, not the pinned CA alone. 10.2b closes this with exact pinning: the adapter requests the peer chain (INDICATE_CERTIFICATE_RECEIVED/deferred validation) and accepts only a chain to the configured CA with the expected name, checked in owned code. Grants and session admission stay application-level (ADR-0083/10.3).
  • Threading: msquic calls back on its own worker threads. The transport adapter only copies bounded data into bounded queues owned by the region host's thread; callbacks never touch simulation or presentation state and never block.
  • Channels (implemented in 10.2b): realtime intentions and superseding snapshots on QUIC DATAGRAM frames (unreliable, congestion-controlled, payload <= 1,200 bytes, newest-wins, never retransmitted by us); reliable actions and outcomes on bidirectional streams with bounded send/receive windows; liveness from QUIC keep-alive (1 s) and idle timeout (5 s); handshake idle timeout 5 s; congestion control msquic's default (CUBIC) unless 10.2c measurements justify BBR.
  • M9 path: the TLS-over-TCP dev session remains the dev/engineering fallback; M10 features build on QUIC only.
  • Impairment tests: an owned, seeded user-space UDP relay (loss, reorder, duplication, delay, jitter, MTU), no root; packet-level TCP comparison is not run.

Evidence (10.2a, 2026-10-03)

evo-quic-smoke-tests (label quic, preset quic): loopback handshake with the dev CA, one DATAGRAM and one stream message with FIN delivered, an untrusted CA refused (TLS alert 51) and a wrong server name refused (alert 42), on Windows MSVC 19.44 and Linux GCC 14; clang-tidy clean; repeated runs stable. Linux standalone build 280 s; full results in engineering/VALIDATION.md.

Alternatives considered

See the table. ngtcp2 needs more owned networking code; GameNetworkingSockets uses non-TLS cryptography; DTLS plus owned reliability carries the retransmission and congestion risk ADR-0017 warns about; Schannel would split TLS stacks by platform and exclude older Windows; prebuilt msquic binaries are not built from reviewed sources.

Consequences

Two TLS stacks exist while the M9 fallback remains (Mbed TLS in the session library, OpenSSL inside msquic). Clean builds take about 5 minutes longer on Linux (OpenSSL). When the client adopts QUIC, the Windows package (ADR-0086) adds msquic.dll to its allowlist and the msquic (MIT) and OpenSSL (Apache-2.0) notices. OpenSSL and msquic security releases update the pin, checksums and this ADR. UDP-blocked networks are served only by the dev fallback for now.

M10.2b implementation (2026-10-03)

evo_network_quic (include/evo/network/quic_transport.hpp) provides one Endpoint (server or client) owned by one thread that calls poll():

  • Channels: realtime DATAGRAMs (payload <= min(1,200, the peer's current maximum); inbound newest-wins per connection, outbound in-flight cap refuses) and one client-opened bidirectional stream carrying 32-bit length-framed messages (<= 64 KiB, once, in order). Outbound reliable bytes not yet completed and inbound bytes awaiting poll() are bounded; inbound overflow closes the connection (overloaded), an oversized or malformed frame closes it (protocol), a second peer stream is refused. Closures carry a cause: local, peer, idle, untrusted, overloaded, protocol, transport.
  • Exact pinning closes the 10.2a gap: the client defers OpenSSL's verdict, receives the peer leaf as DER and returns pending; poll() verifies it with Mbed TLS against only the configured CA and name and completes validation. The owned check is the sole authority; OpenSSL's verdict is advisory (counted when it disagrees). Reason: dev CAs share a subject name, so with another CA in SSL_CERT_FILE OpenSSL's issuer lookup failed a genuine server; the environment must neither add nor remove trust.
  • Threading rule: callbacks touch state only under one mutex and never call msquic while holding it (msquic may run callbacks inline); handles are closed only by the owning thread. Mbed TLS runs only on the owning thread.
  • Path MTU: msquic assumes a 1,288-byte IP MTU by default (1,260-byte IPv4 UDP payloads), which a compliant 1,280-byte path drops. MinimumMtu is set to 1,280; path discovery only probes upward, and the 1,200-byte datagram budget fits that path.
  • Settings: idle 5 s, keep-alive 1 s, handshake 5 s, flow-control windows equal to the inbound reliable bound, one peer bidirectional stream, no unidirectional streams.
  • Impairment relay (tools/network/impair, tests and measurements only): seeded loss, duplication, reordering, corruption, delay/jitter, payload-size limit and live reconfiguration (outage).

M10.2c measurements (2026-10-03)

evo-quic-hol-measure sent the same 20 Hz, 300-byte state on the DATAGRAM channel and the ordered reliable stream, plus 16 KiB reliable messages, through the relay across RTT {2, 50, 100} ms x loss {0, 1, 5, 10} % with 2 % reordering and 1 % duplication, on Windows and Linux (results).

  • The channel split is confirmed. Under 5-10 % loss the ordered stream's p95 latency is 3-5.5x the datagram channel's (head-of-line blocking). At 100 ms RTT it exceeds the 250 ms responsiveness target (10 % loss: 258 / 310 ms on Linux / Windows), while datagram p95 stays <= 67 ms and newest-state staleness p99 <= 195 ms. Superseding state and intentions must not use the ordered stream.
  • Reliable actions and outcomes fit their budget: the slowest 16 KiB message took 510 ms, inside the 1.5 s final-outcome budget; no reliable message was lost.
  • Congestion control stays CUBIC: nothing measured here motivates BBR. Bandwidth limits, competing flows and 20+ clients are 10.9 work.
  • Limits: user-space relay on one host (no kernel TCP, netem or real paths), development hardware. The relay and tool request 1 ms timer resolution on Windows.