Route the dev session as QUIC over UDP 7400
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 3m16s
Foundation / postgres-tools (push) Successful in 5s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 6s

The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP
7400, so nothing reached it through the VIP. The route container port, Service and
NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check
requires exactly UDP 7400 for the route and counts any protocol on 7400 as session
exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and
internal DNS as separate reviewed steps, end-to-end probe as acceptance);
session protocol v1 recorded as frozen now that it is deployed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-05 17:04:31 -04:00
co-authored by Claude Opus 5.5
parent 5caed552b9
commit d917e8443d
11 changed files with 106 additions and 27 deletions
@@ -2,7 +2,7 @@
# the reviewed absolute session window is active. The network inputs are reserved:
# VIP 172.10.1.202 and client CIDR 172.10.1.0/24.
# Route: void.aeternalisgames.com:7400 -> VIP -> Service -> evo-region pod:7400.
# Raw TCP only: TLS 1.3 terminates in evo-server; no HTTP ingress or nginx TLS.
# QUIC over UDP 7400 only: TLS 1.3 terminates in evo-server; no HTTP ingress, no TCP.
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
resources:
@@ -14,4 +14,5 @@ spec:
- {name: REGION_SESSION_LISTEN, value: "0.0.0.0:7400"}
- {name: REGION_SESSION_WINDOW, value: "dev-20261005-201514:1791233114"}
ports:
- {name: session, containerPort: 7400, protocol: TCP}
# QUIC (ADR-0090): UDP only; nothing listens on TCP 7400.
- {name: session, containerPort: 7400, protocol: UDP}
@@ -1,4 +1,4 @@
# Session ingress to the region pod: TCP 7400 from the permitted development/test
# Session ingress to the region pod: UDP 7400 (QUIC) from the permitted development/test
# client CIDR(s) only, which must equal the Service's loadBalancerSourceRanges.
# Everything else stays denied by the platform default-deny policy.
apiVersion: networking.k8s.io/v1
@@ -15,4 +15,4 @@ spec:
- ipBlock:
cidr: 172.10.1.0/24
ports:
- {protocol: TCP, port: 7400}
- {protocol: UDP, port: 7400}
@@ -8,7 +8,10 @@
# serves on every node; it carries no session traffic and must stay closed at the edge
# firewall. Its allocation and quota accounting are verified before enabling (ADR-0087).
# - loadBalancerSourceRanges duplicates the CIDR at the Service; the edge firewall is
# the third layer. TLS 1.3, the dev CA and single-use grants remain mandatory.
# the third layer. TLS 1.3 (inside QUIC), the dev CA and short-lived signed grants
# remain mandatory.
# - The M10 session is QUIC (ADR-0090): one UDP port, no TCP. A single-protocol Service
# needs no MetalLB mixed-protocol support.
apiVersion: v1
kind: Service
metadata:
@@ -26,4 +29,4 @@ spec:
selector:
app.kubernetes.io/name: evo-region
ports:
- {name: session, port: 7400, targetPort: session, protocol: TCP}
- {name: session, port: 7400, targetPort: session, protocol: UDP}
+5
View File
@@ -37,6 +37,11 @@ evo-server QUIC listener (session control and action records under the host's le
the headless `evo-session-probe`, with real-process tests including SIGKILL before and after
an action's commit. M10.4 is complete; the GUI client still uses the M9 dev path. Next:
M10.5 bounded snapshots and spatial interest.
Deployment (2026-10-05): the maintainer deployed the M10 QUIC listener to evolution-dev
behind the ADR-0087 route; session protocol v1 is therefore frozen. The route admitted
only TCP 7400 while QUIC is UDP, so it is changed to UDP 7400 (ADR-0087 amendment). The
route is not yet verified end to end: the edge firewall rule (UDP 7400 from
172.10.1.0/24), internal DNS to 172.10.1.202 and a probe run through the VIP remain.
## Previous gate - 2026-10-02
+30 -4
View File
@@ -13,8 +13,9 @@ Internal testers need the native client (M9.8e ZIP) to reach the dev region with
`kubectl port-forward`. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
HTTP Ingress for void.aeternalisgames.com in namespace `void`. The session protocol is
TLS 1.3 over raw TCP (ADR-0082), not HTTP. Sessions need synthetic carrier inputs, which
HTTP Ingress for void.aeternalisgames.com in namespace `void`. The session protocol was
TLS 1.3 over raw TCP (ADR-0082) when this was written; since 2026-10-05 the route carries
the M10 session, QUIC over UDP (ADR-0090/0091/0092; amendment below). Never HTTP. Sessions need synthetic carrier inputs, which
ADR-0073 keeps drill-time only so dev history does not grow continuously.
## Decision (maintainer, 2026-10-02)
@@ -35,8 +36,8 @@ reach only a ready pod; after a replacement, clients reconnect with a fresh gran
baseline (ADR-0083).
**Access.** Development workstation/test subnet only, in three layers: the edge
firewall, the Service source ranges and a NetworkPolicy admitting TCP 7400 to the region
pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
firewall, the Service source ranges and a NetworkPolicy admitting UDP 7400 (QUIC; TCP 7400
until 2026-10-05) to the region pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
mandatory; grants remain bearer tokens.
**Listener.** The base keeps the opt-in listener on loopback through
@@ -121,3 +122,28 @@ cannot enable listeners, sessions, inputs or session ports outside the route.
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
fail-closed self-test (`tools/ci/deploy-policy-selftest.py`) encode the route's shape and
the enabling preconditions. World0 and stage are untouched.
## Amendment 2026-10-05: QUIC over UDP 7400 (maintainer)
The deployed listener is the M10 QUIC session (`--m10-listen`, ADR-0092), which is UDP. The
route still admitted only TCP 7400 (container port, Service, NetworkPolicy and the policy
check), so nothing reached the listener through the VIP, and nothing listened on TCP 7400;
the pod-side checks (ready, listener bound) could not show this. Decision:
- The route carries **UDP 7400 only**: container port, `evo-region-session` port and the
session NetworkPolicy. TCP 7400 is removed, not kept beside it: no supported protocol
needs it (the M9 TLS-over-TCP session is no longer wired into the deployment). The
Service has a single protocol, so MetalLB mixed-protocol support is not involved; the
VIP, `Local` traffic policy, no node ports, source ranges and the health-check note are
unchanged.
- `deploy-policy-check.py` requires exactly UDP 7400 for the route and a UDP container
port, and counts any protocol on 7400 (UDP, TCP, SCTP) as exposure of the session port
for every other policy and Service; the self-test refuses stale TCP route objects, TCP
beside UDP, and other policies admitting UDP or SCTP 7400.
- **Edge firewall: separate, explicitly reviewed change**, outside the repository: allow
UDP 7400 to 172.10.1.202 from 172.10.1.0/24 only, never a wider source range. Internal
DNS must resolve `void.aeternalisgames.com` to the VIP 172.10.1.202 (enabling step 4);
on 2026-10-05 it resolved to the control-plane address 172.10.1.160.
- **Acceptance** is end to end, not pod-side: `evo-session-probe` from 172.10.1.0/24
succeeds through `void.aeternalisgames.com:7400 -> edge firewall -> MetalLB VIP ->
Service UDP 7400 -> NetworkPolicy UDP 7400 -> evo-region M10 listener`, with the dev CA
and issuer key read from the authorized cluster Secret without printing or persisting
them. Until that is recorded in VALIDATION.md the route is not declared working.
@@ -50,6 +50,7 @@ duplicates and queries; older ones are resolved from the durable record (10.4b).
**Protocol.** Session protocol v1 gains message types 6-12 and Welcome gains the server
tick (schemas/network/session-v1.md). v1 has never been deployed, so it is amended in
place, as ADR-0083 did for M9 v1; after its first deployment, changes need a new version.
v1 was deployed in evolution-dev on 2026-10-05 and is frozen from then on.
The action kind is a raw u16 on the wire and validated by admission (an enum narrower than
the wire field must never be cast from it).
+19
View File
@@ -4309,3 +4309,22 @@ session and network tests; Windows MSVC 19.44 preset `quic` build (including the
CTest 4/4 (binaries rebuilt, timestamps checked).
Not done: the engineering GUI client on M10 sessions; packaging (the server image does not
enable QUIC); a two-process cross-region move; ASan of the PostgreSQL suite.
## 2026-10-05 — Dev session route moved to UDP 7400 for QUIC (ADR-0087 amendment)
Finding: the deployed M10 QUIC listener (UDP) sat behind a route that admitted only TCP
7400 (container port, Service, NetworkPolicy, policy check). Pod-side checks (ready,
listener bound on 7400) could not detect it. Also found: `void.aeternalisgames.com`
resolves to 172.10.1.160 (control plane) from the workstation, not the VIP 172.10.1.202.
Changed: route container port, Service and NetworkPolicy to UDP 7400 only; policy check
(route exactly UDP 7400, UDP container port, any-protocol session exposure elsewhere) and
six new self-test cases; ADR-0087 amendment; session-v1 frozen note.
Executed: `tools/ci/deploy-policy-check.py` (71 objects in 69 files, 130 rendered objects
in 17 overlays, 0 failures) and `tools/ci/deploy-policy-selftest.py` (44 cases, all
refused) with pinned kustomize v5.8.1, before (38 cases) and after; rendered region/dev
inspected: Service `evo-region-session` UDP 7400 only on 172.10.1.202 with source range
172.10.1.0/24, `Local`, no node ports; NetworkPolicy UDP 7400 from 172.10.1.0/24; pod
port `session` UDP 7400; no other Service exposes 7400.
Not done (needs the maintainer): Argo sync of this change; the edge firewall rule;
internal DNS to 172.10.1.202; retrieving the dev CA and issuer key (no cluster credentials
on the workstation); the end-to-end probe through the VIP. The route is not verified.
+4 -3
View File
@@ -8,9 +8,10 @@ each; types 6-8 travel as one QUIC DATAGRAM each, and only after Welcome.
Integers little-endian. Every message has a fixed length for its type; any other length,
magic, version, nonzero flags, unknown type or out-of-range value is refused, and the
server then sends Close (`unsupported_version` for another version, otherwise
`protocol`) and closes the connection. v1 has not been deployed; it was amended in place
by M10.4 (types 6-12, Welcome server_tick). After its first deployment, changes need a
new version.
`protocol`) and closes the connection. v1 was amended in place once, before deployment, by
M10.4 (types 6-12, Welcome server_tick). **Frozen since 2026-10-05**: v1 is deployed in
evolution-dev (ADR-0087 route); any change to a message, value or rule needs a new
version.
## Header (8 bytes)
| Offset | Size | Field | Rule |
+14 -12
View File
@@ -313,7 +313,8 @@ for path, doc in objects:
if volume.get('secret', {}).get('secretName') == 'evolution-dev-session':
check(volume['secret'].get('optional') is True and volume['secret'].get('defaultMode') == 0o440,
f'{path}: the dev session Secret is optional and 0440')
ROUTE_PORT = [{'name': 'session', 'port': 7400, 'targetPort': 'session', 'protocol': 'TCP'}]
# The M10 session is QUIC (ADR-0090, ADR-0087 amendment 2026-10-05): UDP 7400 only.
ROUTE_PORT = [{'name': 'session', 'port': 7400, 'targetPort': 'session', 'protocol': 'UDP'}]
REGION_SELECTOR = {'app.kubernetes.io/name': 'evo-region'}
@@ -324,7 +325,7 @@ def validate_route_service(where, doc):
check(spec.get('type') == 'LoadBalancer' and spec.get('externalTrafficPolicy') == 'Local'
and spec.get('allocateLoadBalancerNodePorts') is False,
f'{where}: the route is a LoadBalancer with Local traffic policy and no node ports')
check(spec.get('ports') == ROUTE_PORT, f'{where}: the route exposes exactly TCP 7400')
check(spec.get('ports') == ROUTE_PORT, f'{where}: the route exposes exactly UDP 7400 (QUIC)')
check(spec.get('selector') == REGION_SELECTOR, f'{where}: the route selects only the region pod')
annotations = doc['metadata'].get('annotations', {})
check(set(annotations) == {'metallb.io/loadBalancerIPs'},
@@ -351,13 +352,13 @@ def validate_route_service(where, doc):
def validate_route_policy(where, doc, ranges):
"""The route policy admits TCP 7400 to the region pod from exactly the Service ranges."""
"""The route policy admits UDP 7400 to the region pod from exactly the Service ranges."""
spec = doc['spec']
rules = spec.get('ingress', [])
check(spec.get('podSelector') == {'matchLabels': REGION_SELECTOR}
and spec.get('policyTypes') == ['Ingress'] and len(rules) == 1
and rules[0].get('ports') == [{'protocol': 'TCP', 'port': 7400}],
f'{where}: the session policy admits only TCP 7400 to the region pod')
and rules[0].get('ports') == [{'protocol': 'UDP', 'port': 7400}],
f'{where}: the session policy admits only UDP 7400 to the region pod')
peers = rules[0].get('from', []) if rules else []
check(peers and all(set(peer) == {'ipBlock'} and set(peer['ipBlock']) == {'cidr'} for peer in peers)
and sorted(peer['ipBlock']['cidr'] for peer in peers) == sorted(ranges),
@@ -380,7 +381,7 @@ for path, doc in objects:
if doc['kind'] == 'Ingress':
text = yaml.safe_dump(doc)
check('7400' not in text and 'session' not in text,
f'{path}: the dev session is raw TCP, never an Ingress (ADR-0087)')
f'{path}: the dev session is QUIC over UDP, never an Ingress (ADR-0087)')
policies = [(p, d) for p, d in objects if p.startswith(ROUTE) and d['kind'] == 'NetworkPolicy']
check(len(policies) == 1, f'{ROUTE}: exactly one session ingress policy')
for path, doc in policies:
@@ -639,14 +640,13 @@ def selects_region(selector):
def admits_session(rule):
"""True when an ingress rule admits TCP 7400 on the region pod: no or empty ports, a
TCP entry without a port, the 'session' name, 7400 itself or a range spanning it."""
"""True when an ingress rule admits 7400 on the region pod in any protocol (UDP for the
QUIC session, and TCP or SCTP, which nothing may expose there either): no or empty ports,
an entry without a port, the 'session' name, 7400 itself or a range spanning it."""
ports = rule.get('ports')
if not ports:
return True
for port in ports:
if port.get('protocol', 'TCP') != 'TCP':
continue
number = port.get('port')
if number is None or number in (7400, 'session'):
return True
@@ -690,8 +690,10 @@ for overlay, docs in renders.items():
else:
check('--m10-listen' not in text and '--synthetic-inputs' not in text,
f'{where}: only evo-server may listen for sessions')
ports = sorted((p.get('name'), p.get('containerPort')) for p in container.get('ports', []))
allowed = [('ops', 9100), ('session', 7400)] if route_active else [('ops', 9100)]
ports = sorted((p.get('name'), p.get('containerPort'), p.get('protocol', 'TCP'))
for p in container.get('ports', []))
allowed = ([('ops', 9100, 'TCP'), ('session', 7400, 'UDP')] if route_active
else [('ops', 9100, 'TCP')])
if container.get('name') == 'evo-server':
check(ports == sorted(allowed), f'{where}: container ports must be {allowed}')
if kind == 'Service':
+23 -2
View File
@@ -192,7 +192,28 @@ CASES = [
'172.10.1.0/24', '10.99.0.0/24'),
'exactly the Service source ranges'),
('policy admits another port', lambda t: edit(t, ROUTE + 'networkpolicy.yaml', 'port: 7400', 'port: 7401'),
'only TCP 7400'),
'only UDP 7400'),
# ADR-0087 amendment 2026-10-05: the route is QUIC over UDP; a stale TCP route fails.
('route Service still TCP', lambda t: edit(t, ROUTE + 'service.yaml', 'protocol: UDP', 'protocol: TCP'),
'exactly UDP 7400 (QUIC)'),
('route policy still TCP', lambda t: edit(t, ROUTE + 'networkpolicy.yaml', 'protocol: UDP', 'protocol: TCP'),
'only UDP 7400'),
('route Service with TCP beside UDP', lambda t: append(
t, ROUTE + 'service.yaml', ' - {name: session-tcp, port: 7400, targetPort: 7400, protocol: TCP}\n'),
'exactly UDP 7400 (QUIC)'),
('route container port still TCP', lambda t: edit(t, ROUTE + 'listener.yaml',
'containerPort: 7400, protocol: UDP',
'containerPort: 7400, protocol: TCP'),
'container ports must be'),
('region policy admitting UDP 7400', lambda t: add_resource(
t, DEV, 'widened.yaml', region_policy('{matchLabels: {app.kubernetes.io/name: evo-region}}',
'{from: [{ipBlock: {cidr: 10.0.0.0/8}}], '
'ports: [{protocol: UDP, port: 7400}]}')),
'admits the region session port'),
('region policy admitting SCTP 7400', lambda t: add_resource(
t, DEV, 'widened.yaml', region_policy('{matchLabels: {app.kubernetes.io/name: evo-region}}',
'{ports: [{protocol: SCTP, port: 7400}]}')),
'admits the region session port'),
('cluster traffic policy', lambda t: edit(t, ROUTE + 'service.yaml', 'externalTrafficPolicy: Local',
'externalTrafficPolicy: Cluster'), 'Local traffic policy'),
('node ports allocated', lambda t: edit(t, ROUTE + 'service.yaml', 'allocateLoadBalancerNodePorts: false',
@@ -202,7 +223,7 @@ CASES = [
'no PROXY protocol'),
('extra route port', lambda t: append(t, ROUTE + 'service.yaml',
' - {name: ops, port: 9100, targetPort: ops, protocol: TCP}\n'),
'exactly TCP 7400'),
'exactly UDP 7400'),
('another LoadBalancer', lambda t: edit(t, 'deploy/kubernetes/region/base/service.yaml', 'type: ClusterIP',
'type: LoadBalancer'), 'Services are forbidden'),
('session port on another Service', lambda t: append(t, 'deploy/kubernetes/region/base/service.yaml',