Files
sor/SOR.Gateway.12FactorPlan.md
2026-07-02 13:09:05 -04:00

17 KiB

SOR.Gateway 12-Factor Refactor Plan

Date: 2026-06-08

Scope reviewed:

  • Server/src/SOR.Gateway
  • Server/src/SOR.Networking.WebSocket
  • Server/src/SOR.Networking.Nats
  • Server/src/SOR.Networking.Redis
  • Gateway wiring in Server/src/SOR.Host

Reference: https://12factor.net/

Executive Summary

SOR.Gateway is already shaped like a 12-Factor edge process in several important ways. It is a service module, it depends mostly on abstractions for transport and messaging, it accepts WebSocket connections through host-provided port binding, it uses Redis Pub/Sub as an attached service for push delivery, and it routes worker-owned messages through NATS request/reply.

The main gaps are around operational hardening and structure:

  • ConnectionLoop.cs owns auth handshake parsing, authentication dispatch, connection registration, Redis push subscription, message receive, deserialization, dispatch, error mapping, send, and cleanup.
  • GatewayHostedService accepts unlimited connections and starts one task per connection without an explicit connection limit or backpressure policy.
  • WebSocketTransport uses an unbounded pending-connection channel.
  • GatewayOptions.MaxMessagesPerSecond exists, but the comment says enforcement is not implemented.
  • Redis push subscriber forwards messages using fire-and-forget tasks and CancellationToken.None, which makes shutdown and overload behavior harder to reason about.
  • Gateway readiness is not dependency-aware; it needs Auth gRPC, Redis, NATS, and WebSocket transport checks when the gateway process is enabled.
  • Service names and NATS RPC subjects are string-derived and should be centralized with validation.
  • There is no visible gateway test project under Server/tests.

The plan is to keep the gateway stateless from a process-correctness perspective, but split the connection lifecycle into focused components and add explicit controls for config validation, limits, readiness, and shutdown.

Current Structure Observations

Strong Foundations

  • SOR.Gateway is a library module registered by AddGatewayService.
  • SOR.Host wires concrete implementations and only enables gateway when ENABLED_SERVICES includes gateway.
  • Gateway depends on ITransport, IMessageDispatcher, IMessageSerializer, and IGatewayAuthValidator.
  • Auth calls are hidden behind IGatewayAuthValidator.
  • Message handlers are resolved locally when co-located and remotely through NATS when not co-located.
  • Redis Pub/Sub is used as an attached service for cross-instance push delivery.
  • ConnectionManager is explicitly documented as per-instance only, which is the correct model for a horizontally scaled gateway.
  • WebSocket send path is serialized by a per-connection lock.
  • Shutdown attempts to cancel the accept loop and wait for active connection tasks.

Main Problems

  • Gateway connection lifecycle is too concentrated in ConnectionLoop.
  • Auth payload records are private to ConnectionLoop, which makes auth handshake behavior harder to test independently.
  • Rate limiting is configured but not enforced.
  • There is no max active connection limit, max unauthenticated connection limit, or bounded accept queue.
  • Redis push subscription is created directly by ConnectionLoop instead of through an injected abstraction.
  • Gateway has direct StackExchange.Redis dependency in connection handling because RedisPushSubscriber is directly constructed.
  • Remote dispatch subject naming is duplicated with NatsMessageRpcResponder.
  • Gateway does not expose operational metrics such as active connections, auth failures, dispatch timeouts, push failures, or pending accept queue length.
  • Readiness is not separated from liveness.

Target Gateway Shape

Keep SOR.Gateway as the edge service module, but split responsibilities:

Server/src/SOR.Gateway/
  ServiceRegistration.cs
  Configuration/
    GatewayOptions.cs
    GatewayOptionsValidator.cs
    GatewayLimitsOptions.cs
  Connections/
    GatewayHostedService.cs
    ConnectionLoop.cs
    ConnectionManager.cs
    ConnectionLimiter.cs
    ConnectionMetrics.cs
  Auth/
    IGatewayAuthValidator.cs
    GatewayAuthValidator.cs
    AuthHandshakeService.cs
    AuthHandshakeMessages.cs
  Dispatch/
    MessageDispatcher.cs
    RpcSubjectNames.cs
    RemoteDispatchOptions.cs
  Push/
    IConnectionPushSubscriber.cs
    RedisConnectionPushSubscriber.cs
    ConnectionPushSubscription.cs
  RateLimiting/
    IMessageRateLimiter.cs
    TokenBucketMessageRateLimiter.cs
  Health/
    GatewayHealthChecks.cs

ConnectionLoop should become orchestration only:

  1. Wait for auth handshake.
  2. Register authenticated connection.
  3. Subscribe to push delivery.
  4. Receive client messages.
  5. Apply rate limit and validation.
  6. Dispatch.
  7. Send reply.
  8. Dispose resources.

12-Factor Gap Map

I. Codebase

Status: Mostly aligned.

Current state:

  • Gateway lives in the same server codebase and is enabled through host process formation.
  • Concrete transports are wired in SOR.Host.

Plan:

  • Keep one codebase and one host binary.
  • Document gateway as a process type: ENABLED_SERVICES=gateway.
  • Keep gateway code free of game/chat/collection domain logic.

II. Dependencies

Status: Partially aligned.

Current state:

  • Gateway uses abstractions for transport and messaging.
  • Gateway directly references Redis and NATS packages for push subscription and remote dispatch.
  • Concrete WebSocket transport is outside the gateway project.

Plan:

  • Keep ITransport abstraction for WebSocket replacement.
  • Introduce IConnectionPushSubscriber so ConnectionLoop no longer directly constructs Redis subscribers.
  • Centralize RPC subject naming in shared networking code.
  • Keep NATS-specific code contained in dispatch or networking infrastructure, not spread through connection lifecycle code.

III. Config

Status: Partially aligned.

Current state:

  • GatewayOptions has auth timeout, max messages per second, and remote dispatch timeout.
  • WebSocket options are configured separately.
  • Options are bound but not validated.

Plan:

  • Add typed validation for:
    • Gateway:AuthTimeoutMs
    • Gateway:MaxMessagesPerSecond
    • Gateway:RemoteDispatchTimeoutMs
    • Gateway:MaxConnections
    • Gateway:MaxPendingConnections
    • Gateway:MaxUnauthenticatedConnections
    • Gateway:PushSendTimeoutMs
    • WebSocket:Path
    • WebSocket:PingIntervalMs
    • WebSocket:MaxMessageSizeBytes
    • Services:Auth
  • Fail fast when gateway is enabled and required config is missing or invalid.

IV. Backing Services

Status: Mostly aligned, with readiness gaps.

Current state:

  • Auth is an attached service via gRPC.
  • Redis is an attached service for push subscription.
  • NATS is an attached service for remote dispatch.
  • Gateway does not persist critical application state locally.

Plan:

  • Treat Auth, Redis, and NATS as required gateway backing services.
  • Add readiness checks for all required attached services.
  • Keep connection registry in memory because it is per-instance socket state.
  • Keep durable application state in worker services, not the gateway.
  • Document Redis Pub/Sub push as best-effort delivery and require client sync for durable state.

V. Build, Release, Run

Status: Mostly aligned.

Current state:

  • Gateway is built as part of SOR.Host.
  • Runtime behavior is selected by ENABLED_SERVICES.

Plan:

  • Keep gateway deploy as host image plus gateway formation config.
  • Add release checklist for gateway:
    • build host image
    • validate gateway config
    • verify Auth, Redis, and NATS endpoints
    • deploy ENABLED_SERVICES=gateway
  • Keep gateway startup free of migrations or admin repair work.

VI. Processes

Status: Mostly aligned conceptually, needs limits.

Current state:

  • Gateway process is stateless across requests except live WebSocket connections.
  • Each connection gets a task.
  • Connection manager is process-local by design.

Plan:

  • Add explicit process limits:
    • max active connections
    • max pending upgrades
    • max unauthenticated connections
    • per-connection message rate
    • per-connection send queue or timeout
  • Make limits configurable and validated.
  • Ensure multiple gateway processes can run behind a load balancer without shared memory assumptions.

VII. Port Binding

Status: Aligned through host.

Current state:

  • SOR.Host binds HTTP/WebSocket port.
  • Gateway uses ITransport and WebSocket middleware when enabled.

Plan:

  • Keep port binding in SOR.Host.
  • Keep gateway unaware of public port numbers.
  • Document WebSocket path and load balancer requirements.
  • Ensure /health and /ready remain available on gateway process deployments.

VIII. Concurrency

Status: Partially aligned.

Current state:

  • Accept loop runs continuously.
  • Each connection runs independently.
  • WebSocket transport queues pending connections without a bound.
  • Send path is serialized per connection.
  • Remote dispatch uses per-request NATS timeouts.

Plan:

  • Replace unbounded pending connection channel with bounded capacity.
  • Add backpressure behavior for pending WebSocket upgrades.
  • Enforce MaxMessagesPerSecond.
  • Add remote dispatch concurrency limits if needed.
  • Track push forwarding tasks or move push forwarding through a bounded per-connection send queue.
  • Add metrics for active tasks, pending connections, dispatch latency, and timeout rates.

IX. Disposability

Status: Partially aligned.

Current state:

  • Gateway cancels accept loop on shutdown and waits for tracked connection tasks.
  • Connection cleanup unregisters, unsubscribes push, disposes socket.
  • Push forwarding ignores cancellation.

Plan:

  • Ensure connection loops respond quickly to shutdown.
  • Add configurable graceful shutdown timeout and connection drain behavior.
  • Avoid CancellationToken.None in push forwarding except where explicitly justified.
  • Track or bound fire-and-forget push sends.
  • Ensure Redis unsubscribe is attempted but does not block shutdown indefinitely.
  • Add load balancer drain guidance for gateway deployments.

X. Dev/Prod Parity

Status: Partially aligned.

Current state:

  • Local config supports Redis, NATS, and Auth service URLs.
  • Gateway can run co-located with auth or other services in all mode.

Plan:

  • Support two local modes:
    • ENABLED_SERVICES=all for convenience
    • ENABLED_SERVICES=gateway plus separate Auth/worker processes for production-like testing
  • Keep config keys identical across environments.
  • Add local test instructions for Redis, NATS, Auth, and WebSocket client behavior.

XI. Logs

Status: Mostly aligned.

Current state:

  • Gateway uses structured logging.
  • Important events are logged: accept loop start, auth success, dispatch route, remote dispatch errors, push failures.

Plan:

  • Keep logs as stdout event streams.
  • Add structured fields consistently:
    • ConnectionId
    • UserId
    • MessageType
    • Seq
    • RemoteAddress
    • DispatchMode
    • CloseReason
  • Avoid logging credentials, session tokens, passwords, or Keycloak tokens.
  • Add startup summary for gateway limits and dependencies.

XII. Admin Processes

Status: Mostly aligned.

Current state:

  • Gateway has no migrations or persistent state to repair.

Plan:

  • Keep gateway free of admin tasks.
  • Add operational commands elsewhere if needed:
    • inspect connected users through metrics
    • validate gateway config
    • smoke test Auth/Redis/NATS connectivity
  • Do not embed maintenance behavior into gateway startup.

Refactor Phases

Phase 0: Baseline and Tests

Deliverables:

  • Add Server/tests/SOR.Gateway.Tests.
  • Add tests for:
    • auth handshake success and failure paths
    • malformed first message
    • unauthenticated non-auth first message
    • local dispatch
    • remote dispatch timeout
    • remote dispatch no responder
    • connection registration and unregister behavior
    • push subscription disposal

Acceptance:

  • Current behavior is covered before extracting lifecycle components.

Phase 1: Extract Auth Handshake

Deliverables:

  • Add AuthHandshakeService.
  • Move auth payload records out of ConnectionLoop.
  • Return a structured result containing user ID, display name, session token, ack type, and starter-deck flag.
  • Keep error response mapping explicit.

Acceptance:

  • ConnectionLoop no longer switches directly over auth message types.
  • Auth handshake can be unit tested without a WebSocket.

Phase 2: Extract Push Subscription Abstraction

Deliverables:

  • Add IConnectionPushSubscriber.
  • Add RedisConnectionPushSubscriber.
  • Inject the abstraction into ConnectionLoop.
  • Add configurable send timeout and cancellation behavior.

Acceptance:

  • ConnectionLoop no longer directly depends on IConnectionMultiplexer.
  • Push behavior can be tested without Redis.

Phase 3: Enforce Limits and Backpressure

Deliverables:

  • Implement per-connection message rate limiting.
  • Add ConnectionLimiter.
  • Add bounded pending WebSocket connection channel.
  • Add config for max active and pending connections.
  • Define close/error reasons for limit failures.

Acceptance:

  • Gateway can reject or close overload traffic predictably.
  • MaxMessagesPerSecond is enforced or removed if intentionally unsupported.

Phase 4: Centralize Dispatch Contracts

Deliverables:

  • Move sor.rpc.{messageType} subject generation to one shared helper.
  • Ensure MessageDispatcher and NatsMessageRpcResponder use the same helper.
  • Add validation for message type to subject conversion.
  • Add tests for local and remote dispatch routes.

Acceptance:

  • RPC subject naming cannot drift between gateway and worker responders.

Phase 5: Health, Readiness, and Metrics

Deliverables:

  • Add gateway readiness checks for:
    • Auth gRPC endpoint
    • Redis
    • NATS
    • WebSocket transport registration
  • Add metrics or health details for:
    • active authenticated connections
    • pending WebSocket upgrades
    • auth failure counts
    • dispatch timeout counts
    • push forwarding failures

Acceptance:

  • Gateway can be kept out of load balancer rotation until dependencies are reachable.

Phase 6: Graceful Shutdown and Drain

Deliverables:

  • Add shutdown/drain options.
  • Stop accepting new WebSocket upgrades during shutdown.
  • Close existing connections with a clear server shutdown reason after drain timeout.
  • Bound Redis unsubscribe and push send waits.

Acceptance:

  • Gateway shutdown is predictable under active client load.
  • Load balancer drain behavior is documented.

Phase 7: Documentation and Deployment Runbook

Deliverables:

  • Document gateway process formation.
  • Document required attached services.
  • Document load balancer and WebSocket upgrade requirements.
  • Document scaling guidance and capacity knobs.
  • Document expected best-effort push semantics.

Acceptance:

  • Gateway can be deployed and scaled independently with clear operational expectations.

Suggested Milestone Order

  1. Add gateway tests around current connection lifecycle.
  2. Extract auth handshake service.
  3. Extract push subscription abstraction.
  4. Implement rate limiting and connection limits.
  5. Bound WebSocket pending connection queue.
  6. Centralize RPC subject names.
  7. Add readiness checks and basic metrics.
  8. Harden graceful shutdown and drain behavior.
  9. Write gateway deployment runbook.

Risk Register

Risk Why It Matters Mitigation
Connection lifecycle regressions Gateway is the client entry point. Add tests before splitting ConnectionLoop.
Overload behavior changes Bounding queues can reject clients that currently wait. Roll out with conservative limits and clear close reasons.
Rate limiting false positives Bursty legitimate game traffic may trip limits. Use token bucket with burst capacity and tune in staging.
Push delivery assumptions Redis Pub/Sub is best-effort. Document semantics and keep durable state sync in worker services.
Shutdown disconnects active games Gateway shutdown can drop sockets. Add drain mode and client reconnect expectations.
RPC subject drift Gateway and workers must agree on NATS subjects. Centralize subject generation and test it.

Definition of Done

The gateway refactor is complete when:

  • ConnectionLoop is small lifecycle orchestration, not auth/push/dispatch implementation.
  • Auth handshake behavior is isolated and tested.
  • Redis push subscription is behind an injected abstraction.
  • Message rate limiting is enforced or deliberately removed from config.
  • Active connections and pending upgrades have explicit limits.
  • WebSocket pending connection queue is bounded.
  • Gateway readiness checks cover Auth, Redis, NATS, and transport registration.
  • Gateway logs are structured and never include credentials or tokens.
  • Graceful shutdown drains or closes connections predictably.
  • Gateway remains horizontally scalable with no cross-instance in-memory correctness dependency.