Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 4s
Foundation / checks (network) (push) Successful in 4s
Foundation / checks (postgres) (push) Successful in 4s
Foundation / checks (tsan) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 4m33s
Foundation / postgres-tools (push) Successful in 2m14s
Foundation / server-image (push) Successful in 5m21s
Foundation / image (push) Successful in 6m18s
The dev session window moves from Git to SQL (ADR-0087 amendment): the admin portal opens, extends (30 min to 4 h from now) and closes it, and the cellular region re-reads it every 5 s with --session-window-source db. No push, sync or restart per window; a closed window keeps the region running instead of crash-looping. - Schema 62: append-only session_windows (4 h cap, no rewrites), load_session_window_v1 for the runtime group, admin_set_session_window_v1 and admin_session_windows_v1 for a dedicated admin group role, granted and audited each migration run when EVO_PG_ADMIN_ROLE is set (aevolution_admin in DEV). Schema 61's views are revoked from the runtime group. - RegionSessions admission switch: while closed every Hello is refused before signature work and closing closes open sessions (new sessions were still admitted after expiry before). - Admin API: GET/POST /api/regions/<n>/session-window, window state in the regions list; the portal's session-window card with history. - Git: REGION_SESSION_WINDOW and the pre-push refresh rule are removed; the policy check requires the database source and refuses a committed window. Needs the new image and the aevolution_admin role (with the portal's login in aevolution-admin-pg) before a sync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
231 lines
15 KiB
Markdown
231 lines
15 KiB
Markdown
[Reviewed]
|
|
# ADR-0087: Dev session route and operator-opened session windows
|
|
|
|
## Status
|
|
Accepted for M9.8f as design (2026-10-02), evolution-dev only. Repository status
|
|
reconciled 2026-10-08: `region/dev` includes the route component, the policy permits
|
|
it, and the server implements an absolute session window. These repository facts
|
|
do not establish current live availability or completion of every qualification
|
|
check below. The original disabled-route status is historical. Narrowly supersedes
|
|
ADR-0083 (loopback-only listener, never a Service) and ADR-0069 (no LoadBalancer
|
|
Services) for evolution-dev, and only once enabled.
|
|
|
|
## Context
|
|
Internal testers need the native client (M9.8e ZIP) to reach the dev region without
|
|
`kubectl port-forward`. The cluster inventory (INVENTORY.md, 2026-09-29) shows RKE2's
|
|
ingress-nginx 1.12.1 DaemonSet serving 80/443 on all four node addresses (public sites
|
|
included), MetalLB 0.15.2 in FRR mode with one LoadBalancer (Gitea SSH), and an existing
|
|
HTTP Ingress for void.aeternalisgames.com in namespace `void` at that inventory date.
|
|
On 2026-10-08 the maintainer confirmed that the deprecated HTTP service and Ingress
|
|
have been removed; there is no remaining HTTP site using this hostname. The session protocol was
|
|
TLS 1.3 over raw TCP (ADR-0082) when this was written; since 2026-10-05 the route carries
|
|
the M10 session, QUIC over UDP (ADR-0090/0091/0092; amendment below). Never HTTP. Sessions need synthetic carrier inputs, which
|
|
ADR-0073 keeps drill-time only so dev history does not grow continuously.
|
|
|
|
## Decision (maintainer, 2026-10-02)
|
|
**Route.** `void.aeternalisgames.com:7400` resolves, in internal DNS, to a dedicated
|
|
internal MetalLB VIP (never automatically the control-plane address 172.10.1.160) that
|
|
fronts one LoadBalancer Service `evo-region-session` in evolution-dev, which forwards to
|
|
the region pod's port 7400. TLS 1.3 terminates in the region pod; its dev-CA server
|
|
certificate carries `void.aeternalisgames.com`, which the client verifies. No HTTP
|
|
Ingress, no TLS termination at nginx, no PROXY-protocol prefix. The Service uses
|
|
`externalTrafficPolicy: Local` (client addresses preserved), no node port for the
|
|
session data (`allocateLoadBalancerNodePorts: false`), a pinned
|
|
`metallb.io/loadBalancerIPs` annotation and `loadBalancerSourceRanges`.
|
|
**Health-check exception:** with `Local`, Kubernetes still allocates
|
|
`spec.healthCheckNodePort`, an HTTP endpoint kube-proxy serves on every node reporting
|
|
local endpoint health. It carries no session traffic, must stay closed at the edge
|
|
firewall, and `services.nodeports: "0"` is not assumed to prevent or account for it. New connections
|
|
reach only a ready pod; after a replacement, clients reconnect with a fresh grant and
|
|
baseline (ADR-0083).
|
|
|
|
**Access.** Development workstation/test subnet only, in three layers: the edge
|
|
firewall, the Service source ranges and a NetworkPolicy admitting UDP 7400 (QUIC; TCP 7400
|
|
until 2026-10-05) to the region pod from exactly those CIDRs. TLS with the dev CA and single-use, short-lived grants stay
|
|
mandatory; grants remain bearer tokens.
|
|
|
|
**Listener.** The base keeps the opt-in listener on loopback through
|
|
`REGION_SESSION_LISTEN=127.0.0.1:7400`. Only the route component
|
|
(`deploy/kubernetes/region/dev-session-route`) binds the pod interface and names the
|
|
session port. The listener still starts only when sessions are enabled.
|
|
|
|
**Windows.** Sessions and synthetic inputs run only during an operator-opened test window
|
|
with automatic expiry. The maintainer decision on 2026-10-05 sets four hours as both the
|
|
operator workflow's default and the server's maximum accepted duration; always-on
|
|
simulation waits for an explicit disposable-world lifecycle and retention. The endpoint
|
|
(DNS, MetalLB Service) may stay configured permanently. A small dev-only GitOps
|
|
configuration holds an activation id,
|
|
the exact world/region identity and an absolute UTC expiry. Operations, via a manually
|
|
triggered Gitea workflow: Open (default four hours), Extend (new explicit expiry), Close
|
|
(stop accepting sessions, close existing connections, stop synthetic inputs, drain
|
|
pending commits) and Status (window, expiry, active sessions, committed tick).
|
|
Expiry is enforced **inside evo-server**, not by a delayed cleanup job:
|
|
- a restart keeps the original absolute deadline, never a fresh duration;
|
|
- an expired configuration cannot reopen the listener;
|
|
- expiry stops synthetic generation as well as connections;
|
|
- a failed workflow cannot leave simulation running; cleanup may later restore the
|
|
closed Git configuration. Grant expiry alone does not stop the generator.
|
|
|
|
## Enabling (all required; `tools/ci/deploy-policy-check.py` refuses otherwise)
|
|
0. `ROUTE_ENABLING_PERMITTED` in the policy check is now `True`. Its original activation
|
|
requirement remains an evidence obligation: record the implemented window argument/configuration
|
|
contract and its runtime expiry tests (expiry during an active session, restart after
|
|
expiry, delayed application, extend/close races, clock movement, synthetic inputs
|
|
stopping while pending commits drain). A flag string in a manifest is not evidence.
|
|
1. The VIP (from the MetalLB pool) and permitted client CIDR(s) (no wider than /16)
|
|
replace the REPLACE markers; Service ranges equal the policy's. Addresses must be
|
|
RFC 1918 or in a range approved here and listed in `APPROVED_INTERNAL` (`172.10.1.0/24`
|
|
is approved for the selected local-DNS test path; the external firewall remains
|
|
outside repository control).
|
|
2. `deploy/kubernetes/platform/dev` raises `services.loadbalancers` to exactly 1;
|
|
node ports stay 0; stage and World0 stay 0.
|
|
3. evo-server implements the session window (`--session-window`) as above, with tests
|
|
and an amendment to this ADR, and the base deployment passes it.
|
|
4. Only `region/dev` includes the component. The selected internal test network is
|
|
`172.10.1.0/24`, explicitly approved in the policy and restricted by the edge
|
|
firewall, Service source ranges and NetworkPolicy. Then: internal DNS for
|
|
`void.aeternalisgames.com` to `172.10.1.202`, the edge firewall rule, a new dev
|
|
server certificate for `void.aeternalisgames.com`, and live connectivity recorded
|
|
separately in VALIDATION.md.
|
|
5. Health-check port: a server-side dry-run and the applied object show the allocated
|
|
`healthCheckNodePort` and how the quota accounts for it; the firewall handling is
|
|
recorded. `services.nodeports` is not raised without that evidence.
|
|
6. The rendered-output checks (ADR-0088) pass for every overlay with the route included.
|
|
|
|
## Open points for the maintainer
|
|
Manual Argo delivery was confirmed by the maintainer on 2026-10-08 (amendment below).
|
|
The maintainer confirmed retention of the void session hostname and removal of the
|
|
deprecated HTTP Ingress on 2026-10-08 (D-003). Actual MetalLB, DNS and client-address
|
|
configuration still require operational verification. Running-window clock behavior was decided
|
|
on 2026-10-08; numerical host clock-health tolerances remain to be specified.
|
|
|
|
- **Status observation.** Manual delivery is decided. A workflow commit reports desired
|
|
Git state, not applied state. Live status still needs an explicitly defined read path;
|
|
until then the operator verifies it through Argo and server observations.
|
|
- **Hostname.** Resolved by D-003: retain `void.aeternalisgames.com` for sessions;
|
|
the prior HTTP service and Ingress were deprecated and removed, per maintainer.
|
|
- **Clock health.** Dual-deadline expiry is decided below. Host synchronization checks
|
|
and their numerical tolerance, especially across restarts, remain an engineering
|
|
follow-up; a process-local elapsed timer cannot prove UTC accuracy after a restart.
|
|
|
|
## Alternatives considered
|
|
- ingress-nginx tcp-services or SNI passthrough on 443: couples game sessions to the
|
|
shared web ingress (443 likely internet-facing) and needs controller changes.
|
|
- Port-forward only (ADR-0083): no tester-friendly endpoint.
|
|
- NodePort: opens a port on every node; forbidden by ADR-0069.
|
|
- Always-on sessions in dev: continuous synthetic history; rejected for now.
|
|
|
|
## Amendment 2026-10-08: Manual application of session-window changes
|
|
|
|
Maintainer selected option A during the decision review. The Open/Extend/Close
|
|
workflow prepares and commits the desired dev-window configuration. The operator
|
|
manually syncs `aevolution-dev` to apply it. This decision grants no automated
|
|
Argo sync or deployment credentials to CI.
|
|
|
|
A successful commit means the change is pending application, not that the server
|
|
has opened, extended or closed its window. The committed expiry remains absolute;
|
|
a delayed sync never grants a fresh four-hour duration. Close takes effect when
|
|
applied; server-enforced expiry remains the automatic upper bound. The existing
|
|
four-hour default and maximum remain unchanged. Workflow implementation and live
|
|
status verification are follow-up work, not certified by this decision.
|
|
|
|
## Amendment 2026-10-08: Retain the void session hostname
|
|
|
|
Maintainer selected option B during the decision review (D-003), confirming that
|
|
the old HTTP Ingress was removed and no HTTP site remains for this hostname.
|
|
Keep `void.aeternalisgames.com` as the dev-session hostname, using the selected
|
|
internal VIP and UDP 7400 route. A separate session hostname or shared web routing
|
|
is unnecessary for this use. This records maintainer-provided infrastructure
|
|
information; it is not a new live cluster inspection or an instruction to delete
|
|
any resources.
|
|
|
|
## Amendment 2026-10-08: UTC and suspend-aware elapsed expiry
|
|
|
|
Maintainer selected option A during the decision review (D-002). A running window
|
|
closes when either its absolute UTC expiry or its suspend-aware elapsed-time
|
|
deadline is reached. At admission, derive the elapsed allowance from the remaining
|
|
time until the original UTC expiry, bounded by the existing four-hour maximum.
|
|
Capture the elapsed deadline once; ordinary clock checks never move it later.
|
|
|
|
A backward UTC correction cannot prolong the established elapsed allowance.
|
|
A forward correction reaching UTC expiry closes the window immediately. Closure
|
|
is latched for that admitted window, including if UTC subsequently moves backward.
|
|
The elapsed clock must include host suspend; an arbitrary steady clock is not
|
|
sufficient without verifying that property on the platform.
|
|
|
|
Restarts still use the committed original UTC expiry and only its remaining time,
|
|
not a new four-hour duration. Synchronized host clocks remain required; the elapsed
|
|
guard alone cannot establish clock accuracy across process or host restarts.
|
|
An explicit operator extension follows D-001 and is not an automatic response to
|
|
clock correction.
|
|
|
|
Implementation remains pending. Qualification must cover backward and forward UTC
|
|
changes, suspend/resume, latched closure, delayed application, and restart with the
|
|
original expiry. Existing expiry consequences (session closure, stopping synthetic
|
|
inputs, and draining pending commits) apply to either deadline.
|
|
|
|
Policy evaluation (ADR-0088): the check renders every overlay with the pinned kustomize
|
|
and validates the effective objects, so inline or JSON patches and namespace changes
|
|
cannot enable listeners, sessions, inputs or session ports outside the route.
|
|
|
|
## Consequences
|
|
One reviewed exception to "no LoadBalancer" for evolution-dev. The policy check and its
|
|
fail-closed self-test (`tools/ci/deploy-policy-selftest.py`) encode the route's shape and
|
|
the enabling preconditions. World0 and stage are untouched.
|
|
|
|
## Amendment 2026-10-05: QUIC over UDP 7400 (maintainer)
|
|
The deployed listener is the M10 QUIC session (`--m10-listen`, ADR-0092), which is UDP. The
|
|
route still admitted only TCP 7400 (container port, Service, NetworkPolicy and the policy
|
|
check), so nothing reached the listener through the VIP, and nothing listened on TCP 7400;
|
|
the pod-side checks (ready, listener bound) could not show this. Decision:
|
|
- The route carries **UDP 7400 only**: container port, `evo-region-session` port and the
|
|
session NetworkPolicy. TCP 7400 is removed, not kept beside it: no supported protocol
|
|
needs it (the M9 TLS-over-TCP session is no longer wired into the deployment). The
|
|
Service has a single protocol, so MetalLB mixed-protocol support is not involved; the
|
|
VIP, `Local` traffic policy, no node ports, source ranges and the health-check note are
|
|
unchanged.
|
|
- `deploy-policy-check.py` requires exactly UDP 7400 for the route and a UDP container
|
|
port, and counts any protocol on 7400 (UDP, TCP, SCTP) as exposure of the session port
|
|
for every other policy and Service; the self-test refuses stale TCP route objects, TCP
|
|
beside UDP, and other policies admitting UDP or SCTP 7400.
|
|
- **Edge firewall: separate, explicitly reviewed change**, outside the repository: allow
|
|
UDP 7400 to 172.10.1.202 from 172.10.1.0/24 only, never a wider source range. Internal
|
|
DNS must resolve `void.aeternalisgames.com` to the VIP 172.10.1.202 (enabling step 4);
|
|
on 2026-10-05 it resolved to the control-plane address 172.10.1.160.
|
|
- **Acceptance** is end to end, not pod-side: `evo-session-probe` from 172.10.1.0/24
|
|
succeeds through `void.aeternalisgames.com:7400 -> edge firewall -> MetalLB VIP ->
|
|
Service UDP 7400 -> NetworkPolicy UDP 7400 -> evo-region M10 listener`, with the dev CA
|
|
and issuer key read from the authorized cluster Secret without printing or persisting
|
|
them. Until that is recorded in VALIDATION.md the route is not declared working.
|
|
|
|
## Amendment 2026-10-27: Session windows in SQL, opened from the admin portal
|
|
|
|
Maintainer decision (ADR-0182 discussion). Committing the window to Git meant a push and a
|
|
full Argo sync for every opening or extension, and an expired window made the region exit
|
|
and crash-loop until the next sync. The window moves to the database:
|
|
|
|
- **Record.** Schema 62 adds the append-only `evolution.session_windows` (world, region,
|
|
activation id, absolute UTC expiry or none, author, reason, time written). A row with an
|
|
expiry opens or replaces the window until then; a row without one closes it. The table
|
|
refuses updates, deletes and truncation, and a window may end at most **four hours** after
|
|
it is written (the maximum is unchanged).
|
|
- **Writing.** Only the admin group role, through `admin_set_session_window_v1`, from the
|
|
admin portal (Open or Extend for 30 min to 4 h, Close). The author is the signed-in
|
|
admin's email and Keycloak subject. The runtime group may only read the newest row
|
|
(`load_session_window_v1`).
|
|
- **Reading.** The cellular region runs with `--session-window-source db` and reads the newest
|
|
row every 5 s on its session-ledger connection. A newer activation replaces the window
|
|
without disturbing open sessions; a closing, expired, over-long or missing row closes it.
|
|
Each admitted window keeps its own absolute deadline, a window once closed never reopens
|
|
(only a newer activation opens sessions again), and an unreadable database leaves the
|
|
current window to close at its deadline.
|
|
- **Closed means closed.** While no window is open, every Hello is refused with `shutdown`
|
|
before signature work, and closing closes the open sessions. Previously the cellular
|
|
bridge closed the sessions once at expiry but still admitted new ones. The region keeps
|
|
running while closed instead of exiting.
|
|
- **Git.** `REGION_SESSION_WINDOW` and the pre-push refresh are removed; the policy check
|
|
requires `--session-window-source db` and refuses a committed window. A fixed
|
|
`--session-window` remains for the lineage/primitive bridge and process tests.
|
|
|
|
The elapsed-time guard of the 2026-10-08 clock amendment is still pending.
|