# SOR.Networking 12-Factor Refactor Plan Date: 2026-06-10 Scope reviewed: - `Server/src/SOR.Networking.Abstractions` - `Server/src/SOR.Networking.WebSocket` - `Server/src/SOR.Networking.Nats` - `Server/src/SOR.Networking.Redis` - Networking wiring in `Server/src/SOR.Host` - Gateway-facing usage in `Server/src/SOR.Gateway` - Shared subject/channel contracts in `Server/src/SOR.Shared` Reference: https://12factor.net/ ## Executive Summary `SOR.Networking` is the shared transport and messaging layer for the server. It defines the abstractions used by service modules and supplies concrete implementations for WebSocket client transport, NATS event/RPC messaging, and Redis Pub/Sub client push. The module already has strong 12-factor foundations: - Application code depends on abstractions: `ITransport`, `IClientConnection`, `IEventBus`, and `IClientPush`. - Concrete providers are isolated in separate projects: WebSocket, NATS, and Redis. - `SOR.Host` is the composition root that wires concrete infrastructure to abstractions. - Redis and NATS are treated as attached backing services. - WebSocket, NATS, and Redis lifetimes are mostly singleton/process-scoped. - Event bus delivery semantics are documented as core NATS at-most-once. - Redis push delivery semantics are documented as best-effort. - NATS RPC responders use queue groups for handler request distribution. The main 12-factor gaps are operational rather than architectural: - WebSocket transport uses an unbounded accept queue, so overload can become memory pressure. - WebSocket options are bound but not validated at startup. - There is no explicit max connection, max pending accept, max send size, or outbound backpressure policy. - NATS URL and Redis connection are configured in `SOR.Host`, but dependency health/readiness is still host-level and minimal. - NATS fan-out subscriptions use no queue group by default, which is correct for broadcast but risky if consumers accidentally scale a singleton-style worker. - NATS RPC subject and queue naming is duplicated with Gateway dispatch logic instead of centralized in shared contracts. - Redis push ignores subscriber count by design, but metrics do not currently expose dropped/offline delivery volume. - `RedisClientPush.PushAsync` accepts a cancellation token but does not pass it to `PublishAsync`, because StackExchange.Redis Pub/Sub does not use that token directly. - `NatsMessageRpcResponder` starts listener tasks directly and logs unexpected listener termination, but does not restart failed listeners. - Readiness, metrics, and integration smoke tests are not packaged with the networking modules. The plan is to preserve the current abstraction split, then add typed provider options, validation, bounded queues, health checks, delivery-semantic documentation in interfaces, central subject/channel naming helpers, and contract/integration tests around the transport and broker boundaries. ## Current Structure Observations ### Strong Foundations - `SOR.Networking.Abstractions` keeps service code decoupled from WebSocket, NATS, and Redis. - `SOR.Networking.WebSocket` implements `ITransport` and `IClientConnection` without leaking ASP.NET Core WebSocket types through the abstraction boundary. - `WebSocketClientConnection` serializes concurrent sends with `SemaphoreSlim`. - `WebSocketClientConnection` enforces `MaxMessageSizeBytes` while assembling frames. - `WebSocketTransport` uses a channel to hand accepted ASP.NET Core WebSocket connections to the Gateway accept loop. - `SOR.Networking.Nats` wraps NATS behind `IEventBus`. - `NatsEventBus` explicitly documents core NATS at-most-once semantics and reserves JetStream for durable delivery needs. - `NatsMessageRpcResponder` exposes local `IMessageHandler` implementations over NATS request/reply so Gateway can relay client requests to worker processes. - `NatsMessageRpcResponder` uses queue groups per message type for competing consumers. - `SOR.Networking.Redis` keeps push publishing separate from Gateway's Redis subscription/read side. - `IClientPush` and `RedisClientPush` both document best-effort/offline-drop behavior. - `SOR.Shared.Nats.NatsSubjects` and `SOR.Shared.Redis.RedisKeys` already provide shared naming contracts for event subjects and Redis push channels. ### Main Problems - `WebSocketTransport` uses `Channel.CreateUnbounded`, so the process has no configured accept backlog limit. - `WebSocketTransportOptions` contains path, ping interval, and max message size, but no validator prevents invalid values. - `appsettings.json` includes `WebSocket:AuthTimeoutMs` and `WebSocket:MaxMessagesPerSecond`, but `WebSocketTransportOptions` only defines path, ping interval, and max message size. That creates config drift between host settings and transport settings. - WebSocket keep-alive is configured in `SOR.Host` separately from the application-level ping loop in `WebSocketClientConnection`. - The ping loop is fire-and-forget; it catches/logs exceptions, but it is not represented as a hosted lifecycle primitive. - There is no explicit per-connection outbound queue limit or send timeout policy. - `NatsEventBus.SubscribeAsync` always uses `queueGroup: null`, which is good for true fan-out but should be hard to misuse for singleton worker workflows. - NATS RPC subject/queue naming exists in both Gateway dispatch and `NatsMessageRpcResponder` as local helper methods. - `NatsMessageRpcResponder` creates one listener task per handler type. If a listener exits unexpectedly, it is logged but not restarted. - `NatsMessageRpcResponder` discovers duplicate handlers at startup, but there is no diagnostics endpoint or test that verifies the registered RPC surface. - `RedisClientPush` publishes bytes and intentionally ignores subscriber count; this should be observable as `subscriber_count=0`, not invisible. - `IClientPush` does not expose or record delivery hints such as subscriber count, offline drop, or publish latency. - `SOR.Host` has only a minimal `/health` check and a TODO for Redis, NATS, and Postgres dependency checks. - There are no visible contract tests proving message serialization, RPC relay behavior, WebSocket frame limits, cancellation, or shutdown semantics. ## Target Networking Shape Keep the provider split, but make operational policy explicit: ```text Server/src/SOR.Networking.Abstractions/ EventBus/ IEventBus.cs EventSubscriptionOptions.cs DeliverySemantics.cs Messaging/ IMessageDispatcher.cs IMessageHandler.cs IMessageSerializer.cs Push/ IClientPush.cs ClientPushResult.cs Rpc/ RpcSubjects.cs RemoteDispatchOptions.cs Transport/ ITransport.cs IClientConnection.cs TransportLimits.cs Server/src/SOR.Networking.WebSocket/ Configuration/ WebSocketTransportOptions.cs WebSocketTransportOptionsValidator.cs Transport/ WebSocketTransport.cs WebSocketClientConnection.cs Health/ WebSocketTransportHealthCheck.cs Metrics/ WebSocketMetrics.cs Server/src/SOR.Networking.Nats/ Configuration/ NatsMessagingOptions.cs NatsMessagingOptionsValidator.cs EventBus/ NatsEventBus.cs Rpc/ NatsMessageRpcResponder.cs Health/ NatsHealthCheck.cs Metrics/ NatsMetrics.cs Server/src/SOR.Networking.Redis/ Configuration/ RedisPushOptions.cs RedisPushOptionsValidator.cs Push/ RedisClientPush.cs Health/ RedisHealthCheck.cs Metrics/ RedisPushMetrics.cs ``` ## 12-Factor Assessment ### 1. Codebase Status: mostly aligned. Keep `SOR.Networking` as shared infrastructure code, not a deployable process. The process boundary remains `SOR.Host` and role selection remains driven by `ENABLED_SERVICES`. Plan: - Keep abstractions and provider implementations in separate projects. - Move duplicated NATS RPC subject/queue naming into one shared helper. - Keep Gateway-specific Redis subscription behavior in `SOR.Gateway`, but document the full push contract in Networking. - Add networking contract tests that can run without service modules. ### 2. Dependencies Status: mostly aligned. Provider dependencies are isolated: NATS packages stay in `SOR.Networking.Nats`, Redis packages stay in `SOR.Networking.Redis`, and ASP.NET Core WebSocket types stay in `SOR.Networking.WebSocket`. Plan: - Keep provider packages out of `SOR.Networking.Abstractions`. - Add test projects per provider where integration dependencies are needed. - Avoid introducing direct references from service modules to provider-specific networking projects. ### 3. Config Status: partially aligned. Config is environment-compatible through `SOR.Host`, but provider options are incomplete and not validated. Plan: - Add validators for `WebSocketTransportOptions`, `NatsMessagingOptions`, and `RedisPushOptions`. - Reconcile config drift around `AuthTimeoutMs` and `MaxMessagesPerSecond`; either move them to Gateway options or define them in transport options if transport owns them. - Promote NATS RPC timeout, queue group prefix, subject prefix, and listener restart policy into typed options. - Promote WebSocket accept backlog, max active connections, send timeout, ping interval, and max payload size into typed options. - Promote Redis push channel prefix and metrics behavior into typed options. - Fail fast on invalid or missing production config. ### 4. Backing Services Status: aligned in concept, incomplete operationally. Redis and NATS are attached backing services. They should be replaceable by changing config and provider registration, not changing business services. Plan: - Add Redis and NATS health checks that can be included in readiness. - Separate liveness from readiness: liveness checks process health, readiness checks broker connectivity. - Document Redis Pub/Sub as best-effort push, not durable state. - Document core NATS as at-most-once eventing, and define when JetStream is required. ### 5. Build, Release, Run Status: mostly aligned. Networking projects are library code loaded by the host at runtime. Plan: - Keep build artifacts environment-neutral. - Keep provider selection and provider URLs in config. - Add startup validation so bad broker URLs, invalid WebSocket limits, or missing options fail during release/run startup rather than at first message. ### 6. Processes Status: mostly aligned. The networking modules keep connection/process state but do not own durable application state. WebSocket connections are inherently process-local; Redis and NATS externalize cross-process communication. Plan: - Treat WebSocket connections as disposable process state. - Treat Redis/NATS as the only cross-process communication boundary. - Keep user/game/session state outside Networking. - Make offline reconnect behavior rely on persistent service state and client sync, not push replay. ### 7. Port Binding Status: aligned through `SOR.Host`. Networking libraries do not bind ports directly. `SOR.Host` owns Kestrel, HTTP, gRPC, and WebSocket upgrade configuration. Plan: - Keep WebSocket path and Kestrel behavior in host-level config. - Keep transport registration behind `AddWebSocketTransport`. - Ensure gateway-only WebSocket registration remains role-gated by `ENABLED_SERVICES`. ### 8. Concurrency Status: partially aligned. The code uses async APIs, channels, and cancellation, but overload behavior is not explicit. Plan: - Replace the unbounded WebSocket accept channel with a bounded channel and a configured full-mode policy. - Add metrics for pending accepts, active connections, rejected upgrades, receive payload size, send latency, and disconnect reason. - Add per-connection send timeout and max outbound message size. - Make NATS subscription type explicit: fan-out subscription vs queue-group subscription. - Add listener restart policy or process-fatal behavior for unexpected NATS RPC listener exits. - Add tests for duplicate handler registration and RPC queue group distribution. ### 9. Disposability Status: partially aligned. Cancellation is present, but shutdown and drain behavior should be more explicit. Plan: - On shutdown, stop accepting new WebSocket upgrades before waiting for active connections. - Add a configured graceful connection drain timeout. - Ensure pending accepts are completed/cancelled cleanly. - Ensure NATS RPC subscriptions stop cleanly and report listener shutdown. - Ensure Redis push publishes fail fast when Redis is unavailable rather than blocking request paths indefinitely. - Add integration tests for shutdown cancellation in WebSocket, NATS responder, and Redis push. ### 10. Dev/Prod Parity Status: partially aligned. The host uses local default Redis/NATS config, but parity depends on developers running equivalent backing services. Plan: - Provide a local compose profile for Redis and NATS. - Keep local broker behavior close to production defaults. - Add a smoke-test command that verifies WebSocket upgrade, NATS publish/subscribe, NATS RPC, and Redis push publish. - Avoid hidden in-memory fallbacks in production service registration. ### 11. Logs Status: partially aligned. Logging exists, but operational signals need to be structured and measurable. Plan: - Add structured fields for transport name, connection ID, user ID, subject, queue group, channel, message type, subscriber count, payload size, latency, and failure reason. - Add counters for WebSocket accepted/rejected/closed, NATS publish/subscribe failures, RPC requests/errors/timeouts, and Redis push subscriber counts. - Keep logs on stdout/stderr through host logging. - Avoid logging payload bodies unless a redaction policy exists. ### 12. Admin Processes Status: early. Networking has no explicit one-off diagnostics or repair commands. Plan: - Add broker diagnostics that can run as admin commands or smoke-test jobs. - Add a handler registry dump for NATS RPC responder startup. - Add an optional subject/channel inventory command for known `NatsSubjects`, RPC subjects, and Redis keys. - If durable messaging is introduced later, add replay/dead-letter admin tooling at that time. ## Refactor Phases ### Phase 0: Baseline Contracts - Add tests for `IMessageSerializer` wire compatibility across WebSocket, Redis push, and NATS RPC payloads. - Add tests for WebSocket max message size handling. - Add tests for duplicate `IMessageHandler` detection in `NatsMessageRpcResponder`. - Add tests for Redis push channel naming. - Add tests that Gateway and responder compute the same RPC subject/queue names. ### Phase 1: Configuration Cleanup - Add typed option classes for NATS messaging and Redis push. - Add `IValidateOptions` validators for WebSocket, NATS, and Redis options. - Resolve `WebSocket` config drift by moving auth/rate-limit options to Gateway or implementing them in transport. - Add startup validation for required broker URLs and limits. ### Phase 2: Backpressure and Limits - Replace unbounded pending WebSocket accepts with bounded channel config. - Add `MaxActiveConnections`. - Add explicit reject behavior for overload. - Add send timeout and optional max outbound payload size. - Add metrics for pending accepts and rejected upgrades. ### Phase 3: Subject and Channel Contracts - Centralize NATS RPC subject and queue naming in shared code. - Keep event subjects in `NatsSubjects`. - Keep Redis push channel names in `RedisKeys`. - Add tests that prevent accidental naming drift across Gateway, responder, and publishers. ### Phase 4: Broker Hardening - Add NATS health check. - Add Redis health check. - Add NATS RPC listener restart/fatal policy. - Add explicit subscription options for fan-out vs queue-group consumption. - Define when core NATS is sufficient and when JetStream is required. ### Phase 5: Observability - Add metrics wrappers around WebSocket accepts/sends/receives. - Add NATS publish, subscribe, and RPC metrics. - Add Redis push metrics including subscriber count. - Add structured logging fields across transport, push, event bus, and RPC. ### Phase 6: Disposability - Add graceful WebSocket drain behavior. - Add tests for cancellation and shutdown paths. - Ensure NATS and Redis dependencies fail quickly when unavailable. - Add readiness checks that remove the process from traffic before shutdown drains active work. ### Phase 7: Dev/Prod Parity and Runbooks - Add local broker startup instructions. - Add smoke tests for WebSocket, Redis, NATS event bus, and NATS RPC. - Document expected production env vars. - Document operational delivery semantics for events, RPC, and push. ## Immediate Next Steps 1. Add `WebSocketTransportOptionsValidator` and reconcile unused WebSocket config keys. 2. Centralize RPC subject/queue naming used by Gateway and `NatsMessageRpcResponder`. 3. Add Redis/NATS readiness checks and wire them into host readiness. 4. Add metrics for Redis push subscriber count and NATS RPC handler latency. 5. Replace the WebSocket accept queue with a bounded, configurable channel. ## Definition of Done - All networking provider options are typed, validated, and environment-overridable. - WebSocket overload behavior is bounded and observable. - Redis, NATS, and WebSocket readiness/health behavior is explicit. - Event, push, and RPC delivery semantics are documented in interfaces and runbooks. - NATS RPC naming cannot drift between Gateway and responder. - Integration tests cover WebSocket frame limits, NATS event publish/subscribe, NATS RPC, Redis push publish, and shutdown cancellation. - No service module directly depends on concrete provider implementations.