Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 3m16s
Foundation / postgres-tools (push) Successful in 5s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 6s
The deployed M10 listener is QUIC (UDP) but the ADR-0087 route admitted only TCP 7400, so nothing reached it through the VIP. The route container port, Service and NetworkPolicy now carry UDP 7400 only (no TCP: nothing needs it). The policy check requires exactly UDP 7400 for the route and counts any protocol on 7400 as session exposure elsewhere; six new self-test cases. ADR-0087 amended (edge firewall and internal DNS as separate reviewed steps, end-to-end probe as acceptance); session protocol v1 recorded as frozen now that it is deployed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
150 lines
10 KiB
Markdown
150 lines
10 KiB
Markdown
# ADR-0087: Dev session route and operator-opened session windows
|
|
|
|
## Status
|
|
Accepted for M9.8f as design (2026-10-02), evolution-dev only. The route is committed
|
|
**disabled**; nothing was applied to the cluster. The M9.8f gate stays open until the
|
|
route is enabled and connectivity is recorded. Enabling it is blocked by policy until
|
|
the inputs and server-side window enforcement below exist. Narrowly supersedes
|
|
ADR-0083 (loopback-only listener, never a Service) and ADR-0069 (no LoadBalancer
|
|
Services) for evolution-dev, and only once enabled.
|
|
|
|
## Context
|
|
Internal testers need the native client (M9.8e ZIP) to reach the dev region without
|
|
`kubectl port-forward`. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
|
|
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
|
|
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
|
|
HTTP Ingress for void.aeternalisgames.com in namespace `void`. The session protocol was
|
|
TLS 1.3 over raw TCP (ADR-0082) when this was written; since 2026-10-05 the route carries
|
|
the M10 session, QUIC over UDP (ADR-0090/0091/0092; amendment below). Never HTTP. Sessions need synthetic carrier inputs, which
|
|
ADR-0073 keeps drill-time only so dev history does not grow continuously.
|
|
|
|
## Decision (maintainer, 2026-10-02)
|
|
**Route.** `void.aeternalisgames.com:7400` resolves, in internal DNS, to a dedicated
|
|
internal MetalLB VIP (never automatically the control-plane address 172.10.1.160) that
|
|
fronts one LoadBalancer Service `evo-region-session` in evolution-dev, which forwards to
|
|
the region pod's port 7400. TLS 1.3 terminates in the region pod; its dev-CA server
|
|
certificate carries `void.aeternalisgames.com`, which the client verifies. No HTTP
|
|
Ingress, no TLS termination at nginx, no PROXY-protocol prefix. The Service uses
|
|
`externalTrafficPolicy: Local` (client addresses preserved), no node port for the
|
|
session data (`allocateLoadBalancerNodePorts: false`), a pinned
|
|
`metallb.io/loadBalancerIPs` annotation and `loadBalancerSourceRanges`.
|
|
**Health-check exception:** with `Local`, Kubernetes still allocates
|
|
`spec.healthCheckNodePort`, an HTTP endpoint kube-proxy serves on every node reporting
|
|
local endpoint health. It carries no session traffic, must stay closed at the edge
|
|
firewall, and `services.nodeports: "0"` is not assumed to prevent or account for it. New connections
|
|
reach only a ready pod; after a replacement, clients reconnect with a fresh grant and
|
|
baseline (ADR-0083).
|
|
|
|
**Access.** Development workstation/test subnet only, in three layers: the edge
|
|
firewall, the Service source ranges and a NetworkPolicy admitting UDP 7400 (QUIC; TCP 7400
|
|
until 2026-10-05) to the region pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
|
|
mandatory; grants remain bearer tokens.
|
|
|
|
**Listener.** The base keeps the opt-in listener on loopback through
|
|
`REGION_SESSION_LISTEN=127.0.0.1:7400`. Only the route component
|
|
(`deploy/kubernetes/region/dev-session-route`) binds the pod interface and names the
|
|
session port. The listener still starts only when sessions are enabled.
|
|
|
|
**Windows.** Sessions and synthetic inputs run only during an operator-opened test window
|
|
with automatic expiry. The maintainer decision on 2026-10-05 sets four hours as both the
|
|
operator workflow's default and the server's maximum accepted duration; always-on
|
|
simulation waits for an explicit disposable-world lifecycle and retention. The endpoint
|
|
(DNS, MetalLB Service) may stay configured permanently. A small dev-only GitOps
|
|
configuration holds an activation id,
|
|
the exact world/region identity and an absolute UTC expiry. Operations, via a manually
|
|
triggered Gitea workflow: Open (default four hours), Extend (new explicit expiry), Close
|
|
(stop accepting sessions, close existing connections, stop synthetic inputs, drain
|
|
pending commits) and Status (window, expiry, active sessions, committed tick).
|
|
Expiry is enforced **inside evo-server**, not by a delayed cleanup job:
|
|
- a restart keeps the original absolute deadline, never a fresh duration;
|
|
- an expired configuration cannot reopen the listener;
|
|
- expiry stops synthetic generation as well as connections;
|
|
- a failed workflow cannot leave simulation running; cleanup may later restore the
|
|
closed Git configuration. Grant expiry alone does not stop the generator.
|
|
|
|
## Enabling (all required; `tools/ci/deploy-policy-check.py` refuses otherwise)
|
|
0. `ROUTE_ENABLING_PERMITTED` in the policy check is `False` and is changed only together
|
|
with an amendment here that records the implemented window argument/configuration
|
|
contract and its runtime expiry tests (expiry during an active session, restart after
|
|
expiry, delayed application, extend/close races, clock movement, synthetic inputs
|
|
stopping while pending commits drain). A flag string in a manifest is not evidence.
|
|
1. The VIP (from the MetalLB pool) and permitted client CIDR(s) (no wider than /16)
|
|
replace the REPLACE markers; Service ranges equal the policy's. Addresses must be
|
|
RFC 1918 or in a range approved here and listed in `APPROVED_INTERNAL` (`172.10.1.0/24`
|
|
is approved for the selected local-DNS test path; the external firewall remains
|
|
outside repository control).
|
|
2. `deploy/kubernetes/platform/dev` raises `services.loadbalancers` to exactly 1;
|
|
node ports stay 0; stage and World0 stay 0.
|
|
3. evo-server implements the session window (`--session-window`) as above, with tests
|
|
and an amendment to this ADR, and the base deployment passes it.
|
|
4. Only `region/dev` includes the component. The selected internal test network is
|
|
`172.10.1.0/24`, explicitly approved in the policy and restricted by the edge
|
|
firewall, Service source ranges and NetworkPolicy. Then: internal DNS for
|
|
`void.aeternalisgames.com` to `172.10.1.202`, the edge firewall rule, a new dev
|
|
server certificate for `void.aeternalisgames.com`, and live connectivity recorded
|
|
separately in VALIDATION.md.
|
|
5. Health-check port: a server-side dry-run and the applied object show the allocated
|
|
`healthCheckNodePort` and how the quota accounts for it; the firewall handling is
|
|
recorded. `services.nodeports` is not raised without that evidence.
|
|
6. The rendered-output checks (ADR-0088) pass for every overlay with the route included.
|
|
|
|
## Open points for the maintainer
|
|
Review recommendations (M9.8f review, 2026-10-02), pending the maintainer's decision:
|
|
keep manual Argo sync for the first delivery and report a workflow-created window as
|
|
pending until applied, with an absolute expiry that delayed sync never shifts (Close
|
|
also needs propagation; the server deadline is the upper bound); keep the existing void
|
|
HTTP endpoint unless explicitly retired, possibly choosing a dedicated session hostname
|
|
instead (DNS cannot select by TCP port), before issuing the certificate; confirm the
|
|
actual MetalLB pool and client addresses; clock-skew handling remains open.
|
|
|
|
- **Argo delivery.** The aevolution-dev Application has no automated sync, and the
|
|
policy forbids auto-sync outside the platform app (ADR-0069). A workflow commit alone
|
|
will not deploy; either a manual sync per window or a scoped dev exception must be
|
|
decided before implementing the workflow. CI holds no cluster credentials (ADR-0020),
|
|
so Status can report Git state; live fields need a defined read path.
|
|
- **void namespace.** void.aeternalisgames.com already has an HTTP Ingress (namespace
|
|
`void`). Pointing internal DNS at the VIP changes what internal clients reach for that
|
|
name; confirm or retire the existing route.
|
|
- Clock skew: the four-hour maximum is decided; define the permitted clock-skew tolerance.
|
|
|
|
## Alternatives considered
|
|
- ingress-nginx tcp-services or SNI passthrough on 443: couples game sessions to the
|
|
shared web ingress (443 likely internet-facing) and needs controller changes.
|
|
- Port-forward only (ADR-0083): no tester-friendly endpoint.
|
|
- NodePort: opens a port on every node; forbidden by ADR-0069.
|
|
- Always-on sessions in dev: continuous synthetic history; rejected for now.
|
|
|
|
Policy evaluation (ADR-0088): the check renders every overlay with the pinned kustomize
|
|
and validates the effective objects, so inline or JSON patches and namespace changes
|
|
cannot enable listeners, sessions, inputs or session ports outside the route.
|
|
|
|
## Consequences
|
|
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
|
|
fail-closed self-test (`tools/ci/deploy-policy-selftest.py`) encode the route's shape and
|
|
the enabling preconditions. World0 and stage are untouched.
|
|
|
|
## Amendment 2026-10-05: QUIC over UDP 7400 (maintainer)
|
|
The deployed listener is the M10 QUIC session (`--m10-listen`, ADR-0092), which is UDP. The
|
|
route still admitted only TCP 7400 (container port, Service, NetworkPolicy and the policy
|
|
check), so nothing reached the listener through the VIP, and nothing listened on TCP 7400;
|
|
the pod-side checks (ready, listener bound) could not show this. Decision:
|
|
- The route carries **UDP 7400 only**: container port, `evo-region-session` port and the
|
|
session NetworkPolicy. TCP 7400 is removed, not kept beside it: no supported protocol
|
|
needs it (the M9 TLS-over-TCP session is no longer wired into the deployment). The
|
|
Service has a single protocol, so MetalLB mixed-protocol support is not involved; the
|
|
VIP, `Local` traffic policy, no node ports, source ranges and the health-check note are
|
|
unchanged.
|
|
- `deploy-policy-check.py` requires exactly UDP 7400 for the route and a UDP container
|
|
port, and counts any protocol on 7400 (UDP, TCP, SCTP) as exposure of the session port
|
|
for every other policy and Service; the self-test refuses stale TCP route objects, TCP
|
|
beside UDP, and other policies admitting UDP or SCTP 7400.
|
|
- **Edge firewall: separate, explicitly reviewed change**, outside the repository: allow
|
|
UDP 7400 to 172.10.1.202 from 172.10.1.0/24 only, never a wider source range. Internal
|
|
DNS must resolve `void.aeternalisgames.com` to the VIP 172.10.1.202 (enabling step 4);
|
|
on 2026-10-05 it resolved to the control-plane address 172.10.1.160.
|
|
- **Acceptance** is end to end, not pod-side: `evo-session-probe` from 172.10.1.0/24
|
|
succeeds through `void.aeternalisgames.com:7400 -> edge firewall -> MetalLB VIP ->
|
|
Service UDP 7400 -> NetworkPolicy UDP 7400 -> evo-region M10 listener`, with the dev CA
|
|
and issuer key read from the authorized cluster Secret without printing or persisting
|
|
them. Until that is recorded in VALIDATION.md the route is not declared working.
|