Files
aevolution/docs/adr/0087-dev-session-route.md
T
2026-10-08 12:08:53 -04:00

200 lines
13 KiB
Markdown

[Reviewed]
# ADR-0087: Dev session route and operator-opened session windows
## Status
Accepted for M9.8f as design (2026-10-02), evolution-dev only. Repository status
reconciled 2026-10-08: `region/dev` includes the route component, the policy permits
it, and the server implements an absolute session window. These repository facts
do not establish current live availability or completion of every qualification
check below. The original disabled-route status is historical. Narrowly supersedes
ADR-0083 (loopback-only listener, never a Service) and ADR-0069 (no LoadBalancer
Services) for evolution-dev, and only once enabled.
## Context
Internal testers need the native client (M9.8e ZIP) to reach the dev region without
`kubectl port-forward`. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
HTTP Ingress for void.aeternalisgames.com in namespace `void` at that inventory date.
On 2026-10-08 the maintainer confirmed that the deprecated HTTP service and Ingress
have been removed; there is no remaining HTTP site using this hostname. The session protocol was
TLS 1.3 over raw TCP (ADR-0082) when this was written; since 2026-10-05 the route carries
the M10 session, QUIC over UDP (ADR-0090/0091/0092; amendment below). Never HTTP. Sessions need synthetic carrier inputs, which
ADR-0073 keeps drill-time only so dev history does not grow continuously.
## Decision (maintainer, 2026-10-02)
**Route.** `void.aeternalisgames.com:7400` resolves, in internal DNS, to a dedicated
internal MetalLB VIP (never automatically the control-plane address 172.10.1.160) that
fronts one LoadBalancer Service `evo-region-session` in evolution-dev, which forwards to
the region pod's port 7400. TLS 1.3 terminates in the region pod; its dev-CA server
certificate carries `void.aeternalisgames.com`, which the client verifies. No HTTP
Ingress, no TLS termination at nginx, no PROXY-protocol prefix. The Service uses
`externalTrafficPolicy: Local` (client addresses preserved), no node port for the
session data (`allocateLoadBalancerNodePorts: false`), a pinned
`metallb.io/loadBalancerIPs` annotation and `loadBalancerSourceRanges`.
**Health-check exception:** with `Local`, Kubernetes still allocates
`spec.healthCheckNodePort`, an HTTP endpoint kube-proxy serves on every node reporting
local endpoint health. It carries no session traffic, must stay closed at the edge
firewall, and `services.nodeports: "0"` is not assumed to prevent or account for it. New connections
reach only a ready pod; after a replacement, clients reconnect with a fresh grant and
baseline (ADR-0083).
**Access.** Development workstation/test subnet only, in three layers: the edge
firewall, the Service source ranges and a NetworkPolicy admitting UDP 7400 (QUIC; TCP 7400
until 2026-10-05) to the region pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
mandatory; grants remain bearer tokens.
**Listener.** The base keeps the opt-in listener on loopback through
`REGION_SESSION_LISTEN=127.0.0.1:7400`. Only the route component
(`deploy/kubernetes/region/dev-session-route`) binds the pod interface and names the
session port. The listener still starts only when sessions are enabled.
**Windows.** Sessions and synthetic inputs run only during an operator-opened test window
with automatic expiry. The maintainer decision on 2026-10-05 sets four hours as both the
operator workflow's default and the server's maximum accepted duration; always-on
simulation waits for an explicit disposable-world lifecycle and retention. The endpoint
(DNS, MetalLB Service) may stay configured permanently. A small dev-only GitOps
configuration holds an activation id,
the exact world/region identity and an absolute UTC expiry. Operations, via a manually
triggered Gitea workflow: Open (default four hours), Extend (new explicit expiry), Close
(stop accepting sessions, close existing connections, stop synthetic inputs, drain
pending commits) and Status (window, expiry, active sessions, committed tick).
Expiry is enforced **inside evo-server**, not by a delayed cleanup job:
- a restart keeps the original absolute deadline, never a fresh duration;
- an expired configuration cannot reopen the listener;
- expiry stops synthetic generation as well as connections;
- a failed workflow cannot leave simulation running; cleanup may later restore the
closed Git configuration. Grant expiry alone does not stop the generator.
## Enabling (all required; `tools/ci/deploy-policy-check.py` refuses otherwise)
0. `ROUTE_ENABLING_PERMITTED` in the policy check is now `True`. Its original activation
requirement remains an evidence obligation: record the implemented window argument/configuration
contract and its runtime expiry tests (expiry during an active session, restart after
expiry, delayed application, extend/close races, clock movement, synthetic inputs
stopping while pending commits drain). A flag string in a manifest is not evidence.
1. The VIP (from the MetalLB pool) and permitted client CIDR(s) (no wider than /16)
replace the REPLACE markers; Service ranges equal the policy's. Addresses must be
RFC 1918 or in a range approved here and listed in `APPROVED_INTERNAL` (`172.10.1.0/24`
is approved for the selected local-DNS test path; the external firewall remains
outside repository control).
2. `deploy/kubernetes/platform/dev` raises `services.loadbalancers` to exactly 1;
node ports stay 0; stage and World0 stay 0.
3. evo-server implements the session window (`--session-window`) as above, with tests
and an amendment to this ADR, and the base deployment passes it.
4. Only `region/dev` includes the component. The selected internal test network is
`172.10.1.0/24`, explicitly approved in the policy and restricted by the edge
firewall, Service source ranges and NetworkPolicy. Then: internal DNS for
`void.aeternalisgames.com` to `172.10.1.202`, the edge firewall rule, a new dev
server certificate for `void.aeternalisgames.com`, and live connectivity recorded
separately in VALIDATION.md.
5. Health-check port: a server-side dry-run and the applied object show the allocated
`healthCheckNodePort` and how the quota accounts for it; the firewall handling is
recorded. `services.nodeports` is not raised without that evidence.
6. The rendered-output checks (ADR-0088) pass for every overlay with the route included.
## Open points for the maintainer
Manual Argo delivery was confirmed by the maintainer on 2026-10-08 (amendment below).
The maintainer confirmed retention of the void session hostname and removal of the
deprecated HTTP Ingress on 2026-10-08 (D-003). Actual MetalLB, DNS and client-address
configuration still require operational verification. Running-window clock behavior was decided
on 2026-10-08; numerical host clock-health tolerances remain to be specified.
- **Status observation.** Manual delivery is decided. A workflow commit reports desired
Git state, not applied state. Live status still needs an explicitly defined read path;
until then the operator verifies it through Argo and server observations.
- **Hostname.** Resolved by D-003: retain `void.aeternalisgames.com` for sessions;
the prior HTTP service and Ingress were deprecated and removed, per maintainer.
- **Clock health.** Dual-deadline expiry is decided below. Host synchronization checks
and their numerical tolerance, especially across restarts, remain an engineering
follow-up; a process-local elapsed timer cannot prove UTC accuracy after a restart.
## Alternatives considered
- ingress-nginx tcp-services or SNI passthrough on 443: couples game sessions to the
shared web ingress (443 likely internet-facing) and needs controller changes.
- Port-forward only (ADR-0083): no tester-friendly endpoint.
- NodePort: opens a port on every node; forbidden by ADR-0069.
- Always-on sessions in dev: continuous synthetic history; rejected for now.
## Amendment 2026-10-08: Manual application of session-window changes
Maintainer selected option A during the decision review. The Open/Extend/Close
workflow prepares and commits the desired dev-window configuration. The operator
manually syncs `aevolution-dev` to apply it. This decision grants no automated
Argo sync or deployment credentials to CI.
A successful commit means the change is pending application, not that the server
has opened, extended or closed its window. The committed expiry remains absolute;
a delayed sync never grants a fresh four-hour duration. Close takes effect when
applied; server-enforced expiry remains the automatic upper bound. The existing
four-hour default and maximum remain unchanged. Workflow implementation and live
status verification are follow-up work, not certified by this decision.
## Amendment 2026-10-08: Retain the void session hostname
Maintainer selected option B during the decision review (D-003), confirming that
the old HTTP Ingress was removed and no HTTP site remains for this hostname.
Keep `void.aeternalisgames.com` as the dev-session hostname, using the selected
internal VIP and UDP 7400 route. A separate session hostname or shared web routing
is unnecessary for this use. This records maintainer-provided infrastructure
information; it is not a new live cluster inspection or an instruction to delete
any resources.
## Amendment 2026-10-08: UTC and suspend-aware elapsed expiry
Maintainer selected option A during the decision review (D-002). A running window
closes when either its absolute UTC expiry or its suspend-aware elapsed-time
deadline is reached. At admission, derive the elapsed allowance from the remaining
time until the original UTC expiry, bounded by the existing four-hour maximum.
Capture the elapsed deadline once; ordinary clock checks never move it later.
A backward UTC correction cannot prolong the established elapsed allowance.
A forward correction reaching UTC expiry closes the window immediately. Closure
is latched for that admitted window, including if UTC subsequently moves backward.
The elapsed clock must include host suspend; an arbitrary steady clock is not
sufficient without verifying that property on the platform.
Restarts still use the committed original UTC expiry and only its remaining time,
not a new four-hour duration. Synchronized host clocks remain required; the elapsed
guard alone cannot establish clock accuracy across process or host restarts.
An explicit operator extension follows D-001 and is not an automatic response to
clock correction.
Implementation remains pending. Qualification must cover backward and forward UTC
changes, suspend/resume, latched closure, delayed application, and restart with the
original expiry. Existing expiry consequences (session closure, stopping synthetic
inputs, and draining pending commits) apply to either deadline.
Policy evaluation (ADR-0088): the check renders every overlay with the pinned kustomize
and validates the effective objects, so inline or JSON patches and namespace changes
cannot enable listeners, sessions, inputs or session ports outside the route.
## Consequences
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
fail-closed self-test (`tools/ci/deploy-policy-selftest.py`) encode the route's shape and
the enabling preconditions. World0 and stage are untouched.
## Amendment 2026-10-05: QUIC over UDP 7400 (maintainer)
The deployed listener is the M10 QUIC session (`--m10-listen`, ADR-0092), which is UDP. The
route still admitted only TCP 7400 (container port, Service, NetworkPolicy and the policy
check), so nothing reached the listener through the VIP, and nothing listened on TCP 7400;
the pod-side checks (ready, listener bound) could not show this. Decision:
- The route carries **UDP 7400 only**: container port, `evo-region-session` port and the
session NetworkPolicy. TCP 7400 is removed, not kept beside it: no supported protocol
needs it (the M9 TLS-over-TCP session is no longer wired into the deployment). The
Service has a single protocol, so MetalLB mixed-protocol support is not involved; the
VIP, `Local` traffic policy, no node ports, source ranges and the health-check note are
unchanged.
- `deploy-policy-check.py` requires exactly UDP 7400 for the route and a UDP container
port, and counts any protocol on 7400 (UDP, TCP, SCTP) as exposure of the session port
for every other policy and Service; the self-test refuses stale TCP route objects, TCP
beside UDP, and other policies admitting UDP or SCTP 7400.
- **Edge firewall: separate, explicitly reviewed change**, outside the repository: allow
UDP 7400 to 172.10.1.202 from 172.10.1.0/24 only, never a wider source range. Internal
DNS must resolve `void.aeternalisgames.com` to the VIP 172.10.1.202 (enabling step 4);
on 2026-10-05 it resolved to the control-plane address 172.10.1.160.
- **Acceptance** is end to end, not pod-side: `evo-session-probe` from 172.10.1.0/24
succeeds through `void.aeternalisgames.com:7400 -> edge firewall -> MetalLB VIP ->
Service UDP 7400 -> NetworkPolicy UDP 7400 -> evo-region M10 listener`, with the dev CA
and issuer key read from the authorized cluster Secret without printing or persisting
them. Until that is recorded in VALIDATION.md the route is not declared working.