Select and pin the M10.2 QUIC transport (msquic with OpenSSL)
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m28s
Foundation / checks (network) (push) Successful in 8m16s
Foundation / checks (client) (push) Successful in 12m14s
Foundation / checks (tsan) (push) Successful in 7m50s
Foundation / checks (checks) (push) Successful in 31m45s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 25m13s

ADR-0090 records the maintainer's selection of msquic 2.6.2 with OpenSSL 3.5.9 LTS
built from pinned, checksum-verified sources on Windows and Linux (plus the
xdp-for-windows headers msquic needs on Windows), the trust, threading and
channel model, and the known gap that OpenSSL's verifier also trusts its default
paths. Adds the opt-in quic preset, explicit preparation, and a loopback smoke
test (datagram, stream, untrusted CA and wrong name refused) passing on MSVC and
GCC. windows-check picks up newly installed Perl/NASM and pins cl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-03 15:07:16 -04:00
co-authored by Claude Opus 5.5
parent 819137c1df
commit 3c421cd0cf
15 changed files with 604 additions and 4 deletions
+3 -1
View File
@@ -516,7 +516,9 @@ bounded replication and recovery under loss, overload and region transfer.
**Status (2026-10-03):** 10.1 ratified (ADR-0089, schemas/network/m10-budgets-v1.md,
networking/M10_TIMING_TRACES.md); targets are not yet qualified on the reference
hardware (10.9). 10.2-10.9 planned.
hardware (10.9). 10.2 split: 10.2a selection, pins and build of msquic/OpenSSL done
(ADR-0090); 10.2b bounded channels, exact CA pinning and the impairment relay, and 10.2c
loss/head-of-line measurements follow. 10.3-10.9 planned.
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
+3 -1
View File
@@ -9,7 +9,9 @@ budgets on one RKE2 worker and client budgets on the low-spec VM. Development-re
traces (`evo-dev-session trace`, networking/M10_TIMING_TRACES.md) measured admission
p99 <= 5.2 ms, durable p95 about 1.4 s and 50 ms view cadence. Finding: genome v1 and
metabolism use per-tick rates, so the quantum fixes biological rates until M11.1
converts or ratifies them. Next: M10.2 transport selection against these budgets.
converts or ratifies them. M10.2a selects QUIC with msquic 2.6.2 and OpenSSL 3.5.9 LTS
(ADR-0090), pinned and building on Windows and Linux with a loopback smoke test. Next:
M10.2b bounded channels, exact CA pinning and the owned impairment relay.
## Previous gate - 2026-10-02
+94
View File
@@ -0,0 +1,94 @@
# ADR-0090: M10 transport — QUIC with msquic and OpenSSL
## Status
Accepted for M10.2 (2026-10-03). The selection, pins and build integration (10.2a) are
decided here; bounded channels with the impairment relay (10.2b) and loss/head-of-line
measurements (10.2c) implement and test it and may amend the channel parameters.
Resolves the transport half of ADR-0017; its no-custom-cryptography rule stands.
## Context
ADR-0089 fixes the budgets the transport must meet: 1,200-byte realtime payloads,
superseding state separate from reliable actions, 1 s heartbeat, 5 s liveness and
handshake, bounded queues. The M9 dev session is TLS 1.3 over TCP (ADR-0082), where one
lost segment delays everything behind it.
Candidates surveyed 2026-10-03 (GitHub releases and licences):
| Candidate | Latest release | Licence | Fit |
|---|---|---|---|
| msquic (Microsoft) | v2.6.2, 2026-09-30 (v2.6.1 2026-08-28) | MIT | Complete QUIC: TLS 1.3, streams, RFC 9221 DATAGRAM, congestion control, PMTUD, own threads; C API; Windows and Linux |
| ngtcp2 | v1.25.0, 2026-07-26 | MIT | Protocol library; we own UDP I/O, timers, loop and a crypto backend |
| picoquic | date-tagged builds (2026-10-01) | MIT | Research-oriented, picotls |
| lsquic | v4.10.0, 2026-09-20 | MIT | Requires BoringSSL |
| quiche | 0.30.0, 2026-09-17 | BSD-2-Clause | Rust toolchain dependency |
| GameNetworkingSockets | v1.6.1, 2026-10-03 | BSD-3-Clause | Game messages and fragmentation; its own non-TLS encryption and certificates |
| Mbed TLS DTLS + owned reliability | 4.2.0 / 4.1.1 | Apache-2.0 option | DTLS 1.2 only here; we would own reliability, fragmentation and congestion |
| wolfSSL | v5.9.4 | GPL-3.0 | Excluded by licence |
## Decision (maintainer, 2026-10-03)
- **msquic 2.6.2** (commit 819ab74f, MIT) with its **OpenSSL TLS provider on Windows
and Linux**, **OpenSSL 3.5.9 LTS** (Apache-2.0), both from pinned, checksum-verified
release archives (`tools/msquic-pin.cmake`, `tools/prepare-msquic.cmake`). msquic's
OpenSSL submodule points at an untagged openssl-3.5 commit (453eaaa9, 2026-07-29);
the official 3.5.9 release on the same branch replaces it for provenance and later
fixes, and the smoke test proves the combination.
- msquic builds OpenSSL statically into its shared library with TLS 1.3 only and legacy
protocols/algorithms disabled (`no-tls1_2`, `no-dtls`, `no-legacy`, `no-deprecated`
and others in msquic's submodule build). Logging, tests, tools and perf are off; the
library is `msquic.dll` / `libmsquic.so`. Windows builds need Strawberry Perl and NASM
(installed with winget on the development host, 2026-10-03; `tools/windows-check.ps1`
reads them from the registry PATH and pins `CC`/`CXX` to `cl` because Strawberry also
puts MinGW compilers on PATH) plus `nmake`.
- msquic always compiles its Windows XDP datapath (optional at run time), which needs the
headers of its `xdp-for-windows` submodule: pinned at msquic's commit d372b525 (MIT),
archive SHA-256 9a52e1b5..., only those headers are used.
- Upstream builds itself with warnings as errors and MSVC `/analyze`; MSVC 19.44 reports
C28020 in unmodified `crypto_tls.c`. As for Mbed TLS, `cmake/Quic.cmake` appends
`/WX-` (`-Wno-error` elsewhere) to msquic's own `warnings` interface; owned code keeps
warnings as errors. Windows consumers also link `ntdll` for msquic.h's inline address
helpers.
- **Trust:** servers present a certificate chain from environment provisioning (dev: the
ADR-0083 dev CA). Clients load the environment CA with `CaCertificateFile` and use
OpenSSL's verifier on every platform (`USE_TLS_BUILTIN_CERTIFICATE_VALIDATION`; msquic
forces it on Linux, and on Windows its default platform verifier ignores the CA file),
verify the server name, and never set `NO_CERTIFICATE_VALIDATION`. ALPN `evo-m10`.
**Known gap:** OpenSSL's built-in verification also loads its default verify paths
(`OPENSSLDIR`, `SSL_CERT_FILE`, `SSL_CERT_DIR`), so trust is the pinned CA plus those
locations, not the pinned CA alone. 10.2b closes this with exact pinning: the adapter
requests the peer chain (`INDICATE_CERTIFICATE_RECEIVED`/deferred validation) and
accepts only a chain to the configured CA with the expected name, checked in owned
code. Grants and session admission stay application-level (ADR-0083/10.3).
- **Threading:** msquic calls back on its own worker threads. The transport adapter only
copies bounded data into bounded queues owned by the region host's thread; callbacks
never touch simulation or presentation state and never block.
- **Channels (implemented in 10.2b):** realtime intentions and superseding snapshots on
QUIC DATAGRAM frames (unreliable, congestion-controlled, payload <= 1,200 bytes,
newest-wins, never retransmitted by us); reliable actions and outcomes on
bidirectional streams with bounded send/receive windows; liveness from QUIC
keep-alive (1 s) and idle timeout (5 s); handshake idle timeout 5 s; congestion
control msquic's default (CUBIC) unless 10.2c measurements justify BBR.
- **M9 path:** the TLS-over-TCP dev session remains the dev/engineering fallback; M10
features build on QUIC only.
- **Impairment tests:** an owned, seeded user-space UDP relay (loss, reorder,
duplication, delay, jitter, MTU), no root; packet-level TCP comparison is not run.
## Evidence (10.2a, 2026-10-03)
`evo-quic-smoke-tests` (label `quic`, preset `quic`): loopback handshake with the dev CA,
one DATAGRAM and one stream message with FIN delivered, an untrusted CA refused (TLS
alert 51) and a wrong server name refused (alert 42), on Windows MSVC 19.44 and Linux
GCC 14; clang-tidy clean; repeated runs stable. Linux standalone build 280 s; full
results in engineering/VALIDATION.md.
## Alternatives considered
See the table. ngtcp2 needs more owned networking code; GameNetworkingSockets uses
non-TLS cryptography; DTLS plus owned reliability carries the retransmission and
congestion risk ADR-0017 warns about; Schannel would split TLS stacks by platform and
exclude older Windows; prebuilt msquic binaries are not built from reviewed sources.
## Consequences
Two TLS stacks exist while the M9 fallback remains (Mbed TLS in the session library,
OpenSSL inside msquic). Clean builds take about 5 minutes longer on Linux (OpenSSL).
When the client adopts QUIC, the Windows package (ADR-0086) adds `msquic.dll` to its
allowlist and the msquic (MIT) and OpenSSL (Apache-2.0) notices. OpenSSL and msquic
security releases update the pin, checksums and this ADR. UDP-blocked networks are
served only by the dev fallback for now.
+2
View File
@@ -17,6 +17,8 @@ Maintenance statements are shortlist evidence, not measured suitability.
| prometheus-cpp / Otel C++ | Client vs broad SDK; upstream stability docs | Export overhead/cardinality/size/failure measurement | MIT / Apache-2.0; deferred |
| mp-units | Modern dimensional APIs; test compiler diagnostics | Compile time/binary footprint and conversion clarity | MIT; deferred |
| Mbed TLS 4.1 LTS | Supported to March 2029; CMake build without Perl/Python from release tarballs; PSA crypto API | ~2.7 MB static archives (MSVC Release); TLS 1.2/DTLS/PSK/tickets compiled out | Apache-2.0 OR GPL-2.0-or-later (Apache taken); adopted for M9 dev session only (ADR-0082) |
| msquic 2.6.2 + OpenSSL 3.5.9 LTS | Microsoft, monthly releases (2.6.1 2026-08, 2.6.2 2026-09); OpenSSL 3.5 LTS; Windows and Linux from pinned source; Perl+NASM on Windows | ~6.8 MB libmsquic.so with static TLS 1.3-only OpenSSL (Linux GCC Release); +~5 min clean build | MIT / Apache-2.0; adopted for M10 transport (ADR-0090) |
| kustomize 5.8.1 | kubernetes-sigs releases (5.8.0 2025-11, 5.8.1 2026-02, 5.8.2 2026-09); Linux x86_64 binary, checksum-verified | CI-only tool, renders overlays for the policy check | Apache-2.0; adopted for deploy policy (ADR-0088) |
| SplitMix64 algorithm | Author's published fixed transition; explicit state | Small constant work, 8-byte state; noncryptographic | Reference attribution in ADR0007; proof only |
Every experiment records version/hash, compiler/OS, representative workload,
+1
View File
@@ -97,6 +97,7 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0083: Dev region sessions](0083-dev-region-sessions.md) - session server in evo-server, durable no-effect intentions on synthetic carriers, empty organism presentation, reconnect with fresh baselines, loopback-only drill-time dev deployment.
- [ADR-0084: Engineering UI toolkit](0084-engineering-ui-toolkit.md) - Dear ImGui 1.92.9b pinned (M9.8a) and integrated as the engineering client overlay with toolkit-first input routing (M9.8c).
- [ADR-0090: M10 QUIC transport](0090-m10-quic-transport.md) - msquic 2.6.2 with OpenSSL 3.5.9 LTS (TLS 1.3 only) from pinned sources on Windows and Linux; DATAGRAM frames for superseding state, streams for reliable actions; M9 TLS/TCP kept as dev fallback; owned impairment relay; exact CA pinning due in 10.2b.
- [ADR-0089: M10 timing and budgets](0089-m10-timing-and-budgets.md) - ratified 50 ms tick, 1 s flush with final outcomes p95 <= 1.5 s, responsiveness LAN p95 <= 150 ms / 100 ms RTT <= 250 ms, reference hardware; derived scheduling, sequence, baseline, queue/byte/work budgets and the exact/tolerant determinism contract; records the per-tick biology coupling.
- [ADR-0088: Rendered deployment policy](0088-rendered-deployment-policy.md) - the deploy policy check renders every overlay with pinned kustomize v5.8.1 (checksum-verified, explicit preparation) and validates effective objects; unsupported kustomize features refused.
- [ADR-0087: Dev session route](0087-dev-session-route.md) - evolution-dev only: void.aeternalisgames.com:7400 via a dedicated internal MetalLB VIP to the region pod, TLS end to end, CIDR-limited, operator-opened windows enforced by evo-server; committed disabled with fail-closed policy tests.
+2 -1
View File
@@ -2,7 +2,8 @@
Status: locked server authority and untrusted clients; transport/reliability are
proposals for M10. M10 timing and workload budgets: ADR-0089 and
[m10-budgets-v1](../../schemas/network/m10-budgets-v1.md). The scoped M9 dev session (TLS 1.3, protocol v1, dev-only grants)
[m10-budgets-v1](../../schemas/network/m10-budgets-v1.md). M10 transport: QUIC via msquic
with OpenSSL (ADR-0090); channels and impairment tests follow in M10.2b. The scoped M9 dev session (TLS 1.3, protocol v1, dev-only grants)
is specified in [DEV_SESSION_PROTOCOL](../networking/DEV_SESSION_PROTOCOL.md) and
ADR-0082. M0 has no networking. Steam distribution follows later through
an identity-provider adapter; core account IDs are independent of Steam IDs.
+28
View File
@@ -4011,3 +4011,31 @@ network tests and evo-dev-session build; Clang ASan/UBSan network tests; clang-t
the session client, evo-dev-session and network_session tests; format check.
Not covered: emulated RTT/loss, multiple clients, the reference worker and low-spec VM,
movement/prediction; the targets are therefore not yet qualified (10.9).
## 2026-10-03 — M10.2a QUIC transport selection and build (ADR-0090)
Maintainer decisions: msquic; OpenSSL from source on Windows and Linux; M9 TLS/TCP kept as
dev/engineering fallback; owned UDP impairment relay for loss testing. Survey (GitHub
API, 2026-10-03): msquic v2.6.2 MIT, ngtcp2 v1.25.0 MIT, picoquic MIT, lsquic v4.10.0
MIT, quiche 0.30.0 BSD-2, GameNetworkingSockets v1.6.1 BSD-3, wolfSSL GPL-3.0 (excluded).
Pins (`tools/msquic-pin.cmake`, explicit `tools/prepare-msquic.cmake`): msquic 2.6.2
tag archive 206e4604... (two downloads identical); OpenSSL 3.5.9 603f5602... equal to the
release's published .sha256; xdp-for-windows d372b525 headers 9a52e1b5... (two downloads
identical). Strawberry Perl 5.42.3.1 and NASM 3.02 installed on the development host
with winget (installer hashes verified by winget, per the maintainer's go-ahead).
Executed: Linux standalone msquic build (GCC 14, 280 s, libmsquic.so 6.8 MB); preset
`quic` on Linux GCC 14 and Windows MSVC 19.44: `evo-quic-smoke-tests` and network tests
2/2 on both; smoke test repeated 5 times on each without failure; clang-tidy (clang 19)
on the smoke test clean; format check; Windows `network` 1/1 and `client-network` 10/10
after the windows-check PATH/compiler changes.
Found and fixed on Windows: a build-directory clash with the Linux run (separate
directories now), msquic's own `/WX` with MSVC `/analyze` C28020 in crypto_tls.c
(upstream-only `/WX-`), missing XDP headers (pinned), `ntdll` for msquic.h helpers, and
the Windows platform verifier ignoring `CaCertificateFile` (OpenSSL verifier selected
explicitly). Security finding recorded in ADR-0090: OpenSSL's built-in verification also
trusts its default verify paths; exact pinning is 10.2b work.
Not done: bounded channels, impairment relay, loss/HOL measurements, CI integration of
the quic preset, sanitizer builds of msquic, packaging of msquic.dll.