17 KiB
SOR.Gateway 12-Factor Refactor Plan
Date: 2026-06-08
Scope reviewed:
Server/src/SOR.GatewayServer/src/SOR.Networking.WebSocketServer/src/SOR.Networking.NatsServer/src/SOR.Networking.Redis- Gateway wiring in
Server/src/SOR.Host
Reference: https://12factor.net/
Executive Summary
SOR.Gateway is already shaped like a 12-Factor edge process in several important ways. It is a service module, it depends mostly on abstractions for transport and messaging, it accepts WebSocket connections through host-provided port binding, it uses Redis Pub/Sub as an attached service for push delivery, and it routes worker-owned messages through NATS request/reply.
The main gaps are around operational hardening and structure:
ConnectionLoop.csowns auth handshake parsing, authentication dispatch, connection registration, Redis push subscription, message receive, deserialization, dispatch, error mapping, send, and cleanup.GatewayHostedServiceaccepts unlimited connections and starts one task per connection without an explicit connection limit or backpressure policy.WebSocketTransportuses an unbounded pending-connection channel.GatewayOptions.MaxMessagesPerSecondexists, but the comment says enforcement is not implemented.- Redis push subscriber forwards messages using fire-and-forget tasks and
CancellationToken.None, which makes shutdown and overload behavior harder to reason about. - Gateway readiness is not dependency-aware; it needs Auth gRPC, Redis, NATS, and WebSocket transport checks when the gateway process is enabled.
- Service names and NATS RPC subjects are string-derived and should be centralized with validation.
- There is no visible gateway test project under
Server/tests.
The plan is to keep the gateway stateless from a process-correctness perspective, but split the connection lifecycle into focused components and add explicit controls for config validation, limits, readiness, and shutdown.
Current Structure Observations
Strong Foundations
SOR.Gatewayis a library module registered byAddGatewayService.SOR.Hostwires concrete implementations and only enables gateway whenENABLED_SERVICESincludesgateway.- Gateway depends on
ITransport,IMessageDispatcher,IMessageSerializer, andIGatewayAuthValidator. - Auth calls are hidden behind
IGatewayAuthValidator. - Message handlers are resolved locally when co-located and remotely through NATS when not co-located.
- Redis Pub/Sub is used as an attached service for cross-instance push delivery.
ConnectionManageris explicitly documented as per-instance only, which is the correct model for a horizontally scaled gateway.- WebSocket send path is serialized by a per-connection lock.
- Shutdown attempts to cancel the accept loop and wait for active connection tasks.
Main Problems
- Gateway connection lifecycle is too concentrated in
ConnectionLoop. - Auth payload records are private to
ConnectionLoop, which makes auth handshake behavior harder to test independently. - Rate limiting is configured but not enforced.
- There is no max active connection limit, max unauthenticated connection limit, or bounded accept queue.
- Redis push subscription is created directly by
ConnectionLoopinstead of through an injected abstraction. - Gateway has direct
StackExchange.Redisdependency in connection handling becauseRedisPushSubscriberis directly constructed. - Remote dispatch subject naming is duplicated with
NatsMessageRpcResponder. - Gateway does not expose operational metrics such as active connections, auth failures, dispatch timeouts, push failures, or pending accept queue length.
- Readiness is not separated from liveness.
Target Gateway Shape
Keep SOR.Gateway as the edge service module, but split responsibilities:
Server/src/SOR.Gateway/
ServiceRegistration.cs
Configuration/
GatewayOptions.cs
GatewayOptionsValidator.cs
GatewayLimitsOptions.cs
Connections/
GatewayHostedService.cs
ConnectionLoop.cs
ConnectionManager.cs
ConnectionLimiter.cs
ConnectionMetrics.cs
Auth/
IGatewayAuthValidator.cs
GatewayAuthValidator.cs
AuthHandshakeService.cs
AuthHandshakeMessages.cs
Dispatch/
MessageDispatcher.cs
RpcSubjectNames.cs
RemoteDispatchOptions.cs
Push/
IConnectionPushSubscriber.cs
RedisConnectionPushSubscriber.cs
ConnectionPushSubscription.cs
RateLimiting/
IMessageRateLimiter.cs
TokenBucketMessageRateLimiter.cs
Health/
GatewayHealthChecks.cs
ConnectionLoop should become orchestration only:
- Wait for auth handshake.
- Register authenticated connection.
- Subscribe to push delivery.
- Receive client messages.
- Apply rate limit and validation.
- Dispatch.
- Send reply.
- Dispose resources.
12-Factor Gap Map
I. Codebase
Status: Mostly aligned.
Current state:
- Gateway lives in the same server codebase and is enabled through host process formation.
- Concrete transports are wired in
SOR.Host.
Plan:
- Keep one codebase and one host binary.
- Document gateway as a process type:
ENABLED_SERVICES=gateway. - Keep gateway code free of game/chat/collection domain logic.
II. Dependencies
Status: Partially aligned.
Current state:
- Gateway uses abstractions for transport and messaging.
- Gateway directly references Redis and NATS packages for push subscription and remote dispatch.
- Concrete WebSocket transport is outside the gateway project.
Plan:
- Keep
ITransportabstraction for WebSocket replacement. - Introduce
IConnectionPushSubscribersoConnectionLoopno longer directly constructs Redis subscribers. - Centralize RPC subject naming in shared networking code.
- Keep NATS-specific code contained in dispatch or networking infrastructure, not spread through connection lifecycle code.
III. Config
Status: Partially aligned.
Current state:
GatewayOptionshas auth timeout, max messages per second, and remote dispatch timeout.- WebSocket options are configured separately.
- Options are bound but not validated.
Plan:
- Add typed validation for:
Gateway:AuthTimeoutMsGateway:MaxMessagesPerSecondGateway:RemoteDispatchTimeoutMsGateway:MaxConnectionsGateway:MaxPendingConnectionsGateway:MaxUnauthenticatedConnectionsGateway:PushSendTimeoutMsWebSocket:PathWebSocket:PingIntervalMsWebSocket:MaxMessageSizeBytesServices:Auth
- Fail fast when gateway is enabled and required config is missing or invalid.
IV. Backing Services
Status: Mostly aligned, with readiness gaps.
Current state:
- Auth is an attached service via gRPC.
- Redis is an attached service for push subscription.
- NATS is an attached service for remote dispatch.
- Gateway does not persist critical application state locally.
Plan:
- Treat Auth, Redis, and NATS as required gateway backing services.
- Add readiness checks for all required attached services.
- Keep connection registry in memory because it is per-instance socket state.
- Keep durable application state in worker services, not the gateway.
- Document Redis Pub/Sub push as best-effort delivery and require client sync for durable state.
V. Build, Release, Run
Status: Mostly aligned.
Current state:
- Gateway is built as part of
SOR.Host. - Runtime behavior is selected by
ENABLED_SERVICES.
Plan:
- Keep gateway deploy as host image plus gateway formation config.
- Add release checklist for gateway:
- build host image
- validate gateway config
- verify Auth, Redis, and NATS endpoints
- deploy
ENABLED_SERVICES=gateway
- Keep gateway startup free of migrations or admin repair work.
VI. Processes
Status: Mostly aligned conceptually, needs limits.
Current state:
- Gateway process is stateless across requests except live WebSocket connections.
- Each connection gets a task.
- Connection manager is process-local by design.
Plan:
- Add explicit process limits:
- max active connections
- max pending upgrades
- max unauthenticated connections
- per-connection message rate
- per-connection send queue or timeout
- Make limits configurable and validated.
- Ensure multiple gateway processes can run behind a load balancer without shared memory assumptions.
VII. Port Binding
Status: Aligned through host.
Current state:
SOR.Hostbinds HTTP/WebSocket port.- Gateway uses
ITransportand WebSocket middleware when enabled.
Plan:
- Keep port binding in
SOR.Host. - Keep gateway unaware of public port numbers.
- Document WebSocket path and load balancer requirements.
- Ensure
/healthand/readyremain available on gateway process deployments.
VIII. Concurrency
Status: Partially aligned.
Current state:
- Accept loop runs continuously.
- Each connection runs independently.
- WebSocket transport queues pending connections without a bound.
- Send path is serialized per connection.
- Remote dispatch uses per-request NATS timeouts.
Plan:
- Replace unbounded pending connection channel with bounded capacity.
- Add backpressure behavior for pending WebSocket upgrades.
- Enforce
MaxMessagesPerSecond. - Add remote dispatch concurrency limits if needed.
- Track push forwarding tasks or move push forwarding through a bounded per-connection send queue.
- Add metrics for active tasks, pending connections, dispatch latency, and timeout rates.
IX. Disposability
Status: Partially aligned.
Current state:
- Gateway cancels accept loop on shutdown and waits for tracked connection tasks.
- Connection cleanup unregisters, unsubscribes push, disposes socket.
- Push forwarding ignores cancellation.
Plan:
- Ensure connection loops respond quickly to shutdown.
- Add configurable graceful shutdown timeout and connection drain behavior.
- Avoid
CancellationToken.Nonein push forwarding except where explicitly justified. - Track or bound fire-and-forget push sends.
- Ensure Redis unsubscribe is attempted but does not block shutdown indefinitely.
- Add load balancer drain guidance for gateway deployments.
X. Dev/Prod Parity
Status: Partially aligned.
Current state:
- Local config supports Redis, NATS, and Auth service URLs.
- Gateway can run co-located with auth or other services in
allmode.
Plan:
- Support two local modes:
ENABLED_SERVICES=allfor convenienceENABLED_SERVICES=gatewayplus separate Auth/worker processes for production-like testing
- Keep config keys identical across environments.
- Add local test instructions for Redis, NATS, Auth, and WebSocket client behavior.
XI. Logs
Status: Mostly aligned.
Current state:
- Gateway uses structured logging.
- Important events are logged: accept loop start, auth success, dispatch route, remote dispatch errors, push failures.
Plan:
- Keep logs as stdout event streams.
- Add structured fields consistently:
ConnectionIdUserIdMessageTypeSeqRemoteAddressDispatchModeCloseReason
- Avoid logging credentials, session tokens, passwords, or Keycloak tokens.
- Add startup summary for gateway limits and dependencies.
XII. Admin Processes
Status: Mostly aligned.
Current state:
- Gateway has no migrations or persistent state to repair.
Plan:
- Keep gateway free of admin tasks.
- Add operational commands elsewhere if needed:
- inspect connected users through metrics
- validate gateway config
- smoke test Auth/Redis/NATS connectivity
- Do not embed maintenance behavior into gateway startup.
Refactor Phases
Phase 0: Baseline and Tests
Deliverables:
- Add
Server/tests/SOR.Gateway.Tests. - Add tests for:
- auth handshake success and failure paths
- malformed first message
- unauthenticated non-auth first message
- local dispatch
- remote dispatch timeout
- remote dispatch no responder
- connection registration and unregister behavior
- push subscription disposal
Acceptance:
- Current behavior is covered before extracting lifecycle components.
Phase 1: Extract Auth Handshake
Deliverables:
- Add
AuthHandshakeService. - Move auth payload records out of
ConnectionLoop. - Return a structured result containing user ID, display name, session token, ack type, and starter-deck flag.
- Keep error response mapping explicit.
Acceptance:
ConnectionLoopno longer switches directly over auth message types.- Auth handshake can be unit tested without a WebSocket.
Phase 2: Extract Push Subscription Abstraction
Deliverables:
- Add
IConnectionPushSubscriber. - Add
RedisConnectionPushSubscriber. - Inject the abstraction into
ConnectionLoop. - Add configurable send timeout and cancellation behavior.
Acceptance:
ConnectionLoopno longer directly depends onIConnectionMultiplexer.- Push behavior can be tested without Redis.
Phase 3: Enforce Limits and Backpressure
Deliverables:
- Implement per-connection message rate limiting.
- Add
ConnectionLimiter. - Add bounded pending WebSocket connection channel.
- Add config for max active and pending connections.
- Define close/error reasons for limit failures.
Acceptance:
- Gateway can reject or close overload traffic predictably.
MaxMessagesPerSecondis enforced or removed if intentionally unsupported.
Phase 4: Centralize Dispatch Contracts
Deliverables:
- Move
sor.rpc.{messageType}subject generation to one shared helper. - Ensure
MessageDispatcherandNatsMessageRpcResponderuse the same helper. - Add validation for message type to subject conversion.
- Add tests for local and remote dispatch routes.
Acceptance:
- RPC subject naming cannot drift between gateway and worker responders.
Phase 5: Health, Readiness, and Metrics
Deliverables:
- Add gateway readiness checks for:
- Auth gRPC endpoint
- Redis
- NATS
- WebSocket transport registration
- Add metrics or health details for:
- active authenticated connections
- pending WebSocket upgrades
- auth failure counts
- dispatch timeout counts
- push forwarding failures
Acceptance:
- Gateway can be kept out of load balancer rotation until dependencies are reachable.
Phase 6: Graceful Shutdown and Drain
Deliverables:
- Add shutdown/drain options.
- Stop accepting new WebSocket upgrades during shutdown.
- Close existing connections with a clear server shutdown reason after drain timeout.
- Bound Redis unsubscribe and push send waits.
Acceptance:
- Gateway shutdown is predictable under active client load.
- Load balancer drain behavior is documented.
Phase 7: Documentation and Deployment Runbook
Deliverables:
- Document gateway process formation.
- Document required attached services.
- Document load balancer and WebSocket upgrade requirements.
- Document scaling guidance and capacity knobs.
- Document expected best-effort push semantics.
Acceptance:
- Gateway can be deployed and scaled independently with clear operational expectations.
Suggested Milestone Order
- Add gateway tests around current connection lifecycle.
- Extract auth handshake service.
- Extract push subscription abstraction.
- Implement rate limiting and connection limits.
- Bound WebSocket pending connection queue.
- Centralize RPC subject names.
- Add readiness checks and basic metrics.
- Harden graceful shutdown and drain behavior.
- Write gateway deployment runbook.
Risk Register
| Risk | Why It Matters | Mitigation |
|---|---|---|
| Connection lifecycle regressions | Gateway is the client entry point. | Add tests before splitting ConnectionLoop. |
| Overload behavior changes | Bounding queues can reject clients that currently wait. | Roll out with conservative limits and clear close reasons. |
| Rate limiting false positives | Bursty legitimate game traffic may trip limits. | Use token bucket with burst capacity and tune in staging. |
| Push delivery assumptions | Redis Pub/Sub is best-effort. | Document semantics and keep durable state sync in worker services. |
| Shutdown disconnects active games | Gateway shutdown can drop sockets. | Add drain mode and client reconnect expectations. |
| RPC subject drift | Gateway and workers must agree on NATS subjects. | Centralize subject generation and test it. |
Definition of Done
The gateway refactor is complete when:
ConnectionLoopis small lifecycle orchestration, not auth/push/dispatch implementation.- Auth handshake behavior is isolated and tested.
- Redis push subscription is behind an injected abstraction.
- Message rate limiting is enforced or deliberately removed from config.
- Active connections and pending upgrades have explicit limits.
- WebSocket pending connection queue is bounded.
- Gateway readiness checks cover Auth, Redis, NATS, and transport registration.
- Gateway logs are structured and never include credentials or tokens.
- Graceful shutdown drains or closes connections predictably.
- Gateway remains horizontally scalable with no cross-instance in-memory correctness dependency.