Measure M10.2c loss and head-of-line behaviour of the QUIC channels
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m46s
Foundation / checks (network) (push) Successful in 8m16s
Foundation / checks (client) (push) Successful in 12m6s
Foundation / checks (tsan) (push) Successful in 7m46s
Foundation / checks (checks) (push) Successful in 31m39s
Foundation / checks (postgres) (push) Successful in 33m28s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m15s
Foundation / server-image (push) Successful in 18m31s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m46s
Foundation / checks (network) (push) Successful in 8m16s
Foundation / checks (client) (push) Successful in 12m6s
Foundation / checks (tsan) (push) Successful in 7m46s
Foundation / checks (checks) (push) Successful in 31m39s
Foundation / checks (postgres) (push) Successful in 33m28s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m15s
Foundation / server-image (push) Successful in 18m31s
evo-quic-hol-measure sends the same 20 Hz state on the DATAGRAM channel and the ordered reliable stream plus 16 KiB reliable messages through the impairment relay across RTT x loss. Under 5-10 % loss the stream's p95 is 3-5.5x the datagrams' and exceeds the 250 ms target at 100 ms RTT, confirming the ADR-0090 channel split. The relay and tool request 1 ms timer resolution on Windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+2
-1
@@ -518,7 +518,8 @@ bounded replication and recovery under loss, overload and region transfer.
|
||||
networking/M10_TIMING_TRACES.md); targets are not yet qualified on the reference
|
||||
hardware (10.9). 10.2 split: 10.2a selection, pins and build of msquic/OpenSSL done
|
||||
(ADR-0090); 10.2b bounded channels, exact CA pinning and the impairment relay done;
|
||||
10.2c loss/head-of-line measurements follow. 10.3-10.9 planned.
|
||||
10.2c loss/head-of-line measurements done (networking/M10_TRANSPORT_MEASUREMENTS.md).
|
||||
10.3-10.9 planned.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+4
-1
@@ -13,7 +13,10 @@ converts or ratifies them. M10.2a selects QUIC with msquic 2.6.2 and OpenSSL 3.5
|
||||
(ADR-0090), pinned and building on Windows and Linux with a loopback smoke test. M10.2b adds
|
||||
the bounded QUIC endpoint (datagram and reliable channels, exact CA pinning with the owned
|
||||
check authoritative, minimum-MTU start) and the owned impairment relay, tested on both
|
||||
platforms. Next: M10.2c loss and head-of-line measurements.
|
||||
platforms. M10.2c measures loss and head-of-line blocking through the relay: under 5-10 %
|
||||
loss the ordered stream's p95 latency is 3-5.5x the datagram channel's and exceeds the
|
||||
250 ms target at 100 ms RTT, while datagrams stay <= 67 ms, which confirms the channel split
|
||||
(networking/M10_TRANSPORT_MEASUREMENTS.md). M10.2 is complete. Next: M10.3 session binding.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -120,3 +120,20 @@ served only by the dev fallback for now.
|
||||
- **Impairment relay** (`tools/network/impair`, tests and measurements only): seeded
|
||||
loss, duplication, reordering, corruption, delay/jitter, payload-size limit and live
|
||||
reconfiguration (outage).
|
||||
|
||||
## M10.2c measurements (2026-10-03)
|
||||
`evo-quic-hol-measure` sent the same 20 Hz, 300-byte state on the DATAGRAM channel and
|
||||
the ordered reliable stream, plus 16 KiB reliable messages, through the relay across
|
||||
RTT {2, 50, 100} ms x loss {0, 1, 5, 10} % with 2 % reordering and 1 % duplication, on
|
||||
Windows and Linux ([results](../networking/M10_TRANSPORT_MEASUREMENTS.md)).
|
||||
- **The channel split is confirmed.** Under 5-10 % loss the ordered stream's p95 latency
|
||||
is 3-5.5x the datagram channel's (head-of-line blocking). At 100 ms RTT it exceeds the
|
||||
250 ms responsiveness target (10 % loss: 258 / 310 ms on Linux / Windows), while
|
||||
datagram p95 stays <= 67 ms and newest-state staleness p99 <= 195 ms. Superseding state
|
||||
and intentions must not use the ordered stream.
|
||||
- Reliable actions and outcomes fit their budget: the slowest 16 KiB message took 510 ms,
|
||||
inside the 1.5 s final-outcome budget; no reliable message was lost.
|
||||
- Congestion control stays CUBIC: nothing measured here motivates BBR. Bandwidth limits,
|
||||
competing flows and 20+ clients are 10.9 work.
|
||||
- Limits: user-space relay on one host (no kernel TCP, netem or real paths), development
|
||||
hardware. The relay and tool request 1 ms timer resolution on Windows.
|
||||
|
||||
@@ -4,7 +4,9 @@ Status: locked server authority and untrusted clients; transport/reliability are
|
||||
proposals for M10. M10 timing and workload budgets: ADR-0089 and
|
||||
[m10-budgets-v1](../../schemas/network/m10-budgets-v1.md). M10 transport: QUIC via msquic
|
||||
with OpenSSL (ADR-0090): bounded DATAGRAM and reliable-stream channels with exact CA
|
||||
pinning (`evo_network_quic`, M10.2b); loss/head-of-line measurements follow in M10.2c. The scoped M9 dev session (TLS 1.3, protocol v1, dev-only grants)
|
||||
pinning (`evo_network_quic`, M10.2b); loss/head-of-line measurements (M10.2c,
|
||||
[M10_TRANSPORT_MEASUREMENTS](../networking/M10_TRANSPORT_MEASUREMENTS.md)) confirm
|
||||
superseding state on DATAGRAMs and reliable actions on the ordered stream. The scoped M9 dev session (TLS 1.3, protocol v1, dev-only grants)
|
||||
is specified in [DEV_SESSION_PROTOCOL](../networking/DEV_SESSION_PROTOCOL.md) and
|
||||
ADR-0082. M0 has no networking. Steam distribution follows later through
|
||||
an identity-provider adapter; core account IDs are independent of Steam IDs.
|
||||
|
||||
@@ -4072,3 +4072,22 @@ relay and tests; format check; Windows MSVC 19.44 preset `quic` 3/3 and transpor
|
||||
5/5 repeated runs.
|
||||
Not done: loss/head-of-line measurements and TCP comparison (10.2c), msquic itself under
|
||||
sanitizers, CI integration of the quic preset, integration into the region host/client.
|
||||
|
||||
## 2026-10-03 — M10.2c loss and head-of-line measurements (ADR-0090)
|
||||
|
||||
Added `evo-quic-hol-measure` (`tools/network/quic_hol.cpp`, preset `quic`) and ran it for
|
||||
15 s per configuration (RTT {2, 50, 100} ms x loss {0, 1, 5, 10} %, 2 % reordering, 1 %
|
||||
duplication, 16 KiB reliable messages) on Windows 11 (MSVC 19.44) and WSL2 Ubuntu (GCC 14)
|
||||
on a Ryzen 7 9800X3D. Results: docs/networking/M10_TRANSPORT_MEASUREMENTS.md.
|
||||
|
||||
Found and fixed: Windows numbers first inflated to about 31 ms at 2 ms RTT by the ~15.6 ms
|
||||
default timer tick (the relay and tool now call `timeBeginPeriod(1)`, relay balanced in
|
||||
its destructor; Windows baseline 6 ms p95); datagrams refused before the DATAGRAM
|
||||
extension was negotiated (measurement now waits for it); clang-tidy widening findings.
|
||||
|
||||
Executed: Linux format check, GCC 14 build, CTest `-L quic` 2/2, clang-tidy 19 on the tool
|
||||
and relay; Windows MSVC preset `quic` build and CTest 3/3; both measurement runs.
|
||||
Limits: no kernel TCP/netem comparison (user-space relay by decision), single host and
|
||||
loopback, development hardware, 15 samples of 16 KiB messages per configuration, run
|
||||
length cuts off in-flight stream messages; msquic not under sanitizers; quic preset not
|
||||
in CI.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# M10.2c transport loss and head-of-line measurements
|
||||
|
||||
Supports [ADR-0090](../adr/0090-m10-quic-transport.md) against the
|
||||
[ADR-0089](../adr/0089-m10-timing-and-budgets.md) budgets. Development-reference
|
||||
measurements on one host; they do not qualify the targets on the reference hardware.
|
||||
|
||||
## Method
|
||||
`evo-quic-hol-measure --seconds 15` (`tools/network/quic_hol.cpp`, preset `quic`): a QUIC
|
||||
server sends the same 20 Hz, 300-byte state on both channels of one connection, the
|
||||
DATAGRAM channel and the single ordered reliable stream (the TCP-like case: one lost
|
||||
packet delays everything behind it), plus one 16 KiB reliable message per second
|
||||
(multi-packet, i.e. fragmented, delivery). Traffic crosses the owned impairment relay
|
||||
with delay = RTT/2 each way, jitter RTT/20, 2 % reordering, 1 % duplication and the
|
||||
listed loss, in both directions. Latency is one-way (same process clock); staleness is
|
||||
the age of the newest state the client holds, sampled every 5 ms (20 Hz sending alone
|
||||
gives about 25 ms average, 50 ms worst-case staleness at zero latency).
|
||||
Measurement starts after the datagram channel is negotiated. The relay is user space:
|
||||
this is not a kernel-TCP measurement (the maintainer chose the relay over netem).
|
||||
|
||||
Hosts: AMD Ryzen 7 9800X3D, Windows 11 Pro (MSVC 19.44 build, 1 ms timer resolution
|
||||
requested by the tool and relay) and WSL2 Ubuntu (GCC 14 build), 2026-10-03.
|
||||
|
||||
## Results (ms; Linux / Windows)
|
||||
|
||||
| RTT | Loss | Datagram p95 latency | Stream p95 latency | Datagram p99 staleness | Stream p99 staleness | Datagrams delivered |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 2 | 0 % | 2.5 / 6.0 | 2.5 / 6.0 | 50.2 / 52.0 | 50.2 / 52.0 | 300 / 300 |
|
||||
| 2 | 10 % | 2.5 / 6.0 | 35.5 / 101.1 | 114.9 / 114.8 | 89.9 / 195.4 | 274 / 268 |
|
||||
| 50 | 1 % | 28.3 / 30.1 | 28.4 / 30.6 | 75.3 / 85.3 | 130.0 / 115.1 | 298 / 297 |
|
||||
| 50 | 5 % | 28.4 / 31.0 | 152.0 / 101.6 | 125.0 / 131.3 | 210.0 / 199.2 | 281 / 291 |
|
||||
| 50 | 10 % | 28.5 / 37.7 | 154.9 / 205.6 | 155.0 / 125.8 | 264.9 / 309.4 | 264 / 263 |
|
||||
| 100 | 0 % | 55.9 / 58.6 | 56.0 / 58.6 | 105.2 / 105.8 | 105.2 / 105.8 | 299 / 299 |
|
||||
| 100 | 1 % | 56.0 / 58.2 | 62.4 / 62.6 | 105.2 / 105.9 | 150.0 / 151.0 | 298 / 299 |
|
||||
| 100 | 5 % | 58.2 / 63.6 | 214.5 / 314.2 | 154.9 / 195.0 | 325.2 / 464.0 | 283 / 272 |
|
||||
| 100 | 10 % | 64.2 / 66.7 | 257.9 / 310.4 | 160.0 / 154.7 | 395.2 / 420.3 | 272 / 276 |
|
||||
|
||||
16 KiB reliable messages (15 per run, so p95 is the maximum; Linux / Windows): 12.4 /
|
||||
12.0 ms at 2 ms RTT without loss; 140.8 / 204.8 ms at 50 ms RTT and 5 % loss; 297.4 /
|
||||
510.1 ms at 100 ms RTT and 5 % loss; 412.3 / 438.4 ms at 100 ms RTT and 10 % loss. All
|
||||
were delivered. The reliable stream lost no state message: received counts of 294-299
|
||||
of 300 are messages still in flight when the 15-second run ended.
|
||||
|
||||
## Conclusions
|
||||
- **Head-of-line blocking is the dominant cost of the ordered stream under loss.** At
|
||||
5-10 % loss with 50-100 ms RTT its p95 latency is 3-5.5x the datagram channel's. At
|
||||
100 ms RTT it exceeds ADR-0089's 250 ms responsiveness target at 10 % loss on both
|
||||
hosts and at 5 % loss on Windows (214.5 ms on Linux), while the datagram channel's p95
|
||||
stays within 67 ms.
|
||||
Superseding realtime state and intentions therefore belong on DATAGRAMs (ADR-0090
|
||||
confirmed); losses there are bounded by the next update (staleness p99 <= 195 ms here).
|
||||
- Reliable actions and outcomes tolerate the stream's tail: the slowest 16 KiB message took
|
||||
510 ms (100 ms RTT, 5 % loss), inside the 1.5 s final-outcome budget.
|
||||
- Congestion control stays msquic's default (CUBIC); at this send rate no measurement
|
||||
motivates BBR. Revisit with 20+ clients and bandwidth limits in 10.9.
|
||||
- The Windows build first measured about 31 ms at 2 ms RTT because 1 ms waits round to
|
||||
the ~15.6 ms timer tick (as in M10.1); the relay and tool now request 1 ms resolution.
|
||||
|
||||
## Not covered
|
||||
Kernel TCP/TLS under packet loss (no netem), real networks and NAT, bandwidth caps and
|
||||
competing flows, multiple clients, the reference worker and low-spec VM, the M9 dev path.
|
||||
Reference in New Issue
Block a user