123 lines
8.1 KiB
Markdown
123 lines
8.1 KiB
Markdown
# ADR-0087: Dev session route and operator-opened session windows
|
|
|
|
## Status
|
|
Accepted for M9.8f as design (2026-10-02), evolution-dev only. The route is committed
|
|
**disabled**; nothing was applied to the cluster. The M9.8f gate stays open until the
|
|
route is enabled and connectivity is recorded. Enabling it is blocked by policy until
|
|
the inputs and server-side window enforcement below exist. Narrowly supersedes
|
|
ADR-0083 (loopback-only listener, never a Service) and ADR-0069 (no LoadBalancer
|
|
Services) for evolution-dev, and only once enabled.
|
|
|
|
## Context
|
|
Internal testers need the native client (M9.8e ZIP) to reach the dev region without
|
|
`kubectl port-forward`. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
|
|
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
|
|
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
|
|
HTTP Ingress for void.aeternalisgames.com in namespace `void`. The session protocol is
|
|
TLS 1.3 over raw TCP (ADR-0082), not HTTP. Sessions need synthetic carrier inputs, which
|
|
ADR-0073 keeps drill-time only so dev history does not grow continuously.
|
|
|
|
## Decision (maintainer, 2026-10-02)
|
|
**Route.** `void.aeternalisgames.com:7400` resolves, in internal DNS, to a dedicated
|
|
internal MetalLB VIP (never automatically the control-plane address 172.10.1.160) that
|
|
fronts one LoadBalancer Service `evo-region-session` in evolution-dev, which forwards to
|
|
the region pod's port 7400. TLS 1.3 terminates in the region pod; its dev-CA server
|
|
certificate carries `void.aeternalisgames.com`, which the client verifies. No HTTP
|
|
Ingress, no TLS termination at nginx, no PROXY-protocol prefix. The Service uses
|
|
`externalTrafficPolicy: Local` (client addresses preserved), no node port for the
|
|
session data (`allocateLoadBalancerNodePorts: false`), a pinned
|
|
`metallb.io/loadBalancerIPs` annotation and `loadBalancerSourceRanges`.
|
|
**Health-check exception:** with `Local`, Kubernetes still allocates
|
|
`spec.healthCheckNodePort`, an HTTP endpoint kube-proxy serves on every node reporting
|
|
local endpoint health. It carries no session traffic, must stay closed at the edge
|
|
firewall, and `services.nodeports: "0"` is not assumed to prevent or account for it. New connections
|
|
reach only a ready pod; after a replacement, clients reconnect with a fresh grant and
|
|
baseline (ADR-0083).
|
|
|
|
**Access.** Development workstation/test subnet only, in three layers: the edge
|
|
firewall, the Service source ranges and a NetworkPolicy admitting TCP 7400 to the region
|
|
pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
|
|
mandatory; grants remain bearer tokens.
|
|
|
|
**Listener.** The base keeps the opt-in listener on loopback through
|
|
`REGION_SESSION_LISTEN=127.0.0.1:7400`. Only the route component
|
|
(`deploy/kubernetes/region/dev-session-route`) binds the pod interface and names the
|
|
session port. The listener still starts only when sessions are enabled.
|
|
|
|
**Windows.** Sessions and synthetic inputs run only during an operator-opened test window
|
|
with automatic expiry (default 30 minutes); always-on simulation waits for an explicit
|
|
disposable-world lifecycle and retention. The endpoint (DNS, MetalLB Service) may stay
|
|
configured permanently. A small dev-only GitOps configuration holds an activation id,
|
|
the exact world/region identity and an absolute UTC expiry. Operations, via a manually
|
|
triggered Gitea workflow: Open (default 30 min), Extend (new explicit expiry), Close
|
|
(stop accepting sessions, close existing connections, stop synthetic inputs, drain
|
|
pending commits) and Status (window, expiry, active sessions, committed tick).
|
|
Expiry is enforced **inside evo-server**, not by a delayed cleanup job:
|
|
- a restart keeps the original absolute deadline, never a fresh duration;
|
|
- an expired configuration cannot reopen the listener;
|
|
- expiry stops synthetic generation as well as connections;
|
|
- a failed workflow cannot leave simulation running; cleanup may later restore the
|
|
closed Git configuration. Grant expiry alone does not stop the generator.
|
|
|
|
## Enabling (all required; `tools/ci/deploy-policy-check.py` refuses otherwise)
|
|
0. `ROUTE_ENABLING_PERMITTED` in the policy check is `False` and is changed only together
|
|
with an amendment here that records the implemented window argument/configuration
|
|
contract and its runtime expiry tests (expiry during an active session, restart after
|
|
expiry, delayed application, extend/close races, clock movement, synthetic inputs
|
|
stopping while pending commits drain). A flag string in a manifest is not evidence.
|
|
1. The VIP (from the MetalLB pool) and permitted client CIDR(s) (no wider than /16)
|
|
replace the REPLACE markers; Service ranges equal the policy's. Addresses must be
|
|
RFC 1918 or in a range approved here and listed in `APPROVED_INTERNAL` (`172.10.1.0/24`
|
|
is approved for the selected local-DNS test path; the external firewall remains
|
|
outside repository control).
|
|
2. `deploy/kubernetes/platform/dev` raises `services.loadbalancers` to exactly 1;
|
|
node ports stay 0; stage and World0 stay 0.
|
|
3. evo-server implements the session window (`--session-window`) as above, with tests
|
|
and an amendment to this ADR, and the base deployment passes it.
|
|
4. Only `region/dev` includes the component. The selected internal test network is
|
|
`172.10.1.0/24`, explicitly approved in the policy and restricted by the edge
|
|
firewall, Service source ranges and NetworkPolicy. Then: internal DNS for
|
|
`void.aeternalisgames.com` to `172.10.1.202`, the edge firewall rule, a new dev
|
|
server certificate for `void.aeternalisgames.com`, and live connectivity recorded
|
|
separately in VALIDATION.md.
|
|
5. Health-check port: a server-side dry-run and the applied object show the allocated
|
|
`healthCheckNodePort` and how the quota accounts for it; the firewall handling is
|
|
recorded. `services.nodeports` is not raised without that evidence.
|
|
6. The rendered-output checks (ADR-0088) pass for every overlay with the route included.
|
|
|
|
## Open points for the maintainer
|
|
Review recommendations (M9.8f review, 2026-10-02), pending the maintainer's decision:
|
|
keep manual Argo sync for the first delivery and report a workflow-created window as
|
|
pending until applied, with an absolute expiry that delayed sync never shifts (Close
|
|
also needs propagation; the server deadline is the upper bound); keep the existing void
|
|
HTTP endpoint unless explicitly retired, possibly choosing a dedicated session hostname
|
|
instead (DNS cannot select by TCP port), before issuing the certificate; confirm the
|
|
actual MetalLB pool and client addresses; set the maximum window length and clock
|
|
handling before implementing windows.
|
|
|
|
- **Argo delivery.** The aevolution-dev Application has no automated sync, and the
|
|
policy forbids auto-sync outside the platform app (ADR-0069). A workflow commit alone
|
|
will not deploy; either a manual sync per window or a scoped dev exception must be
|
|
decided before implementing the workflow. CI holds no cluster credentials (ADR-0020),
|
|
so Status can report Git state; live fields need a defined read path.
|
|
- **void namespace.** void.aeternalisgames.com already has an HTTP Ingress (namespace
|
|
`void`). Pointing internal DNS at the VIP changes what internal clients reach for that
|
|
name; confirm or retire the existing route.
|
|
- Window limits: maximum window length and clock-skew tolerance.
|
|
|
|
## Alternatives considered
|
|
- ingress-nginx tcp-services or SNI passthrough on 443: couples game sessions to the
|
|
shared web ingress (443 likely internet-facing) and needs controller changes.
|
|
- Port-forward only (ADR-0083): no tester-friendly endpoint.
|
|
- NodePort: opens a port on every node; forbidden by ADR-0069.
|
|
- Always-on sessions in dev: continuous synthetic history; rejected for now.
|
|
|
|
Policy evaluation (ADR-0088): the check renders every overlay with the pinned kustomize
|
|
and validates the effective objects, so inline or JSON patches and namespace changes
|
|
cannot enable listeners, sessions, inputs or session ports outside the route.
|
|
|
|
## Consequences
|
|
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
|
|
fail-closed self-test (`tools/ci/deploy-policy-selftest.py`) encode the route's shape and
|
|
the enabling preconditions. World0 and stage are untouched.
|