The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP 7400, so nothing reached it through the VIP. The route container port, Service and NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check requires exactly UDP 7400 for the route and counts any protocol on 7400 as session exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and internal DNS as separate reviewed steps, end-to-end probe as acceptance); session protocol v1 recorded as frozen now that it is deployed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
10 KiB
ADR-0087: Dev session route and operator-opened session windows
Status
Accepted for M9.8f as design (2026-10-02), evolution-dev only. The route is committed disabled; nothing was applied to the cluster. The M9.8f gate stays open until the route is enabled and connectivity is recorded. Enabling it is blocked by policy until the inputs and server-side window enforcement below exist. Narrowly supersedes ADR-0083 (loopback-only listener, never a Service) and ADR-0069 (no LoadBalancer Services) for evolution-dev, and only once enabled.
Context
Internal testers need the native client (M9.8e ZIP) to reach the dev region without
kubectl port-forward. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
HTTP Ingress for void.aeternalisgames.com in namespace void. The session protocol was
TLS 1.3 over raw TCP (ADR-0082) when this was written; since 2026-10-05 the route carries
the M10 session, QUIC over UDP (ADR-0090/0091/0092; amendment below). Never HTTP. Sessions need synthetic carrier inputs, which
ADR-0073 keeps drill-time only so dev history does not grow continuously.
Decision (maintainer, 2026-10-02)
Route. void.aeternalisgames.com:7400 resolves, in internal DNS, to a dedicated
internal MetalLB VIP (never automatically the control-plane address 172.10.1.160) that
fronts one LoadBalancer Service evo-region-session in evolution-dev, which forwards to
the region pod's port 7400. TLS 1.3 terminates in the region pod; its dev-CA server
certificate carries void.aeternalisgames.com, which the client verifies. No HTTP
Ingress, no TLS termination at nginx, no PROXY-protocol prefix. The Service uses
externalTrafficPolicy: Local (client addresses preserved), no node port for the
session data (allocateLoadBalancerNodePorts: false), a pinned
metallb.io/loadBalancerIPs annotation and loadBalancerSourceRanges.
Health-check exception: with Local, Kubernetes still allocates
spec.healthCheckNodePort, an HTTP endpoint kube-proxy serves on every node reporting
local endpoint health. It carries no session traffic, must stay closed at the edge
firewall, and services.nodeports: "0" is not assumed to prevent or account for it. New connections
reach only a ready pod; after a replacement, clients reconnect with a fresh grant and
baseline (ADR-0083).
Access. Development workstation/test subnet only, in three layers: the edge firewall, the Service source ranges and a NetworkPolicy admitting UDP 7400 (QUIC; TCP 7400 until 2026-10-05) to the region pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay mandatory; grants remain bearer tokens.
Listener. The base keeps the opt-in listener on loopback through
REGION_SESSION_LISTEN=127.0.0.1:7400. Only the route component
(deploy/kubernetes/region/dev-session-route) binds the pod interface and names the
session port. The listener still starts only when sessions are enabled.
Windows. Sessions and synthetic inputs run only during an operator-opened test window with automatic expiry. The maintainer decision on 2026-10-05 sets four hours as both the operator workflow's default and the server's maximum accepted duration; always-on simulation waits for an explicit disposable-world lifecycle and retention. The endpoint (DNS, MetalLB Service) may stay configured permanently. A small dev-only GitOps configuration holds an activation id, the exact world/region identity and an absolute UTC expiry. Operations, via a manually triggered Gitea workflow: Open (default four hours), Extend (new explicit expiry), Close (stop accepting sessions, close existing connections, stop synthetic inputs, drain pending commits) and Status (window, expiry, active sessions, committed tick). Expiry is enforced inside evo-server, not by a delayed cleanup job:
- a restart keeps the original absolute deadline, never a fresh duration;
- an expired configuration cannot reopen the listener;
- expiry stops synthetic generation as well as connections;
- a failed workflow cannot leave simulation running; cleanup may later restore the closed Git configuration. Grant expiry alone does not stop the generator.
Enabling (all required; tools/ci/deploy-policy-check.py refuses otherwise)
ROUTE_ENABLING_PERMITTEDin the policy check isFalseand is changed only together with an amendment here that records the implemented window argument/configuration contract and its runtime expiry tests (expiry during an active session, restart after expiry, delayed application, extend/close races, clock movement, synthetic inputs stopping while pending commits drain). A flag string in a manifest is not evidence.- The VIP (from the MetalLB pool) and permitted client CIDR(s) (no wider than /16)
replace the REPLACE markers; Service ranges equal the policy's. Addresses must be
RFC 1918 or in a range approved here and listed in
APPROVED_INTERNAL(172.10.1.0/24is approved for the selected local-DNS test path; the external firewall remains outside repository control). deploy/kubernetes/platform/devraisesservices.loadbalancersto exactly 1; node ports stay 0; stage and World0 stay 0.- evo-server implements the session window (
--session-window) as above, with tests and an amendment to this ADR, and the base deployment passes it. - Only
region/devincludes the component. The selected internal test network is172.10.1.0/24, explicitly approved in the policy and restricted by the edge firewall, Service source ranges and NetworkPolicy. Then: internal DNS forvoid.aeternalisgames.comto172.10.1.202, the edge firewall rule, a new dev server certificate forvoid.aeternalisgames.com, and live connectivity recorded separately in VALIDATION.md. - Health-check port: a server-side dry-run and the applied object show the allocated
healthCheckNodePortand how the quota accounts for it; the firewall handling is recorded.services.nodeportsis not raised without that evidence. - The rendered-output checks (ADR-0088) pass for every overlay with the route included.
Open points for the maintainer
Review recommendations (M9.8f review, 2026-10-02), pending the maintainer's decision: keep manual Argo sync for the first delivery and report a workflow-created window as pending until applied, with an absolute expiry that delayed sync never shifts (Close also needs propagation; the server deadline is the upper bound); keep the existing void HTTP endpoint unless explicitly retired, possibly choosing a dedicated session hostname instead (DNS cannot select by TCP port), before issuing the certificate; confirm the actual MetalLB pool and client addresses; clock-skew handling remains open.
- Argo delivery. The aevolution-dev Application has no automated sync, and the policy forbids auto-sync outside the platform app (ADR-0069). A workflow commit alone will not deploy; either a manual sync per window or a scoped dev exception must be decided before implementing the workflow. CI holds no cluster credentials (ADR-0020), so Status can report Git state; live fields need a defined read path.
- void namespace. void.aeternalisgames.com already has an HTTP Ingress (namespace
void). Pointing internal DNS at the VIP changes what internal clients reach for that name; confirm or retire the existing route. - Clock skew: the four-hour maximum is decided; define the permitted clock-skew tolerance.
Alternatives considered
- ingress-nginx tcp-services or SNI passthrough on 443: couples game sessions to the shared web ingress (443 likely internet-facing) and needs controller changes.
- Port-forward only (ADR-0083): no tester-friendly endpoint.
- NodePort: opens a port on every node; forbidden by ADR-0069.
- Always-on sessions in dev: continuous synthetic history; rejected for now.
Policy evaluation (ADR-0088): the check renders every overlay with the pinned kustomize and validates the effective objects, so inline or JSON patches and namespace changes cannot enable listeners, sessions, inputs or session ports outside the route.
Consequences
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
fail-closed self-test (tools/ci/deploy-policy-selftest.py) encode the route's shape and
the enabling preconditions. World0 and stage are untouched.
Amendment 2026-10-05: QUIC over UDP 7400 (maintainer)
The deployed listener is the M10 QUIC session (--m10-listen, ADR-0092), which is UDP. The
route still admitted only TCP 7400 (container port, Service, NetworkPolicy and the policy
check), so nothing reached the listener through the VIP, and nothing listened on TCP 7400;
the pod-side checks (ready, listener bound) could not show this. Decision:
- The route carries UDP 7400 only: container port,
evo-region-sessionport and the session NetworkPolicy. TCP 7400 is removed, not kept beside it: no supported protocol needs it (the M9 TLS-over-TCP session is no longer wired into the deployment). The Service has a single protocol, so MetalLB mixed-protocol support is not involved; the VIP,Localtraffic policy, no node ports, source ranges and the health-check note are unchanged. deploy-policy-check.pyrequires exactly UDP 7400 for the route and a UDP container port, and counts any protocol on 7400 (UDP, TCP, SCTP) as exposure of the session port for every other policy and Service; the self-test refuses stale TCP route objects, TCP beside UDP, and other policies admitting UDP or SCTP 7400.- Edge firewall: separate, explicitly reviewed change, outside the repository: allow
UDP 7400 to 172.10.1.202 from 172.10.1.0/24 only, never a wider source range. Internal
DNS must resolve
void.aeternalisgames.comto the VIP 172.10.1.202 (enabling step 4); on 2026-10-05 it resolved to the control-plane address 172.10.1.160. - Acceptance is end to end, not pod-side:
evo-session-probefrom 172.10.1.0/24 succeeds throughvoid.aeternalisgames.com:7400 -> edge firewall -> MetalLB VIP -> Service UDP 7400 -> NetworkPolicy UDP 7400 -> evo-region M10 listener, with the dev CA and issuer key read from the authorized cluster Secret without printing or persisting them. Until that is recorded in VALIDATION.md the route is not declared working.