Files
sor/SOR.Networking.12FactorPlan.md
T
2026-07-02 13:09:05 -04:00

17 KiB

SOR.Networking 12-Factor Refactor Plan

Date: 2026-06-10

Scope reviewed:

  • Server/src/SOR.Networking.Abstractions
  • Server/src/SOR.Networking.WebSocket
  • Server/src/SOR.Networking.Nats
  • Server/src/SOR.Networking.Redis
  • Networking wiring in Server/src/SOR.Host
  • Gateway-facing usage in Server/src/SOR.Gateway
  • Shared subject/channel contracts in Server/src/SOR.Shared

Reference: https://12factor.net/

Executive Summary

SOR.Networking is the shared transport and messaging layer for the server. It defines the abstractions used by service modules and supplies concrete implementations for WebSocket client transport, NATS event/RPC messaging, and Redis Pub/Sub client push.

The module already has strong 12-factor foundations:

  • Application code depends on abstractions: ITransport, IClientConnection, IEventBus, and IClientPush.
  • Concrete providers are isolated in separate projects: WebSocket, NATS, and Redis.
  • SOR.Host is the composition root that wires concrete infrastructure to abstractions.
  • Redis and NATS are treated as attached backing services.
  • WebSocket, NATS, and Redis lifetimes are mostly singleton/process-scoped.
  • Event bus delivery semantics are documented as core NATS at-most-once.
  • Redis push delivery semantics are documented as best-effort.
  • NATS RPC responders use queue groups for handler request distribution.

The main 12-factor gaps are operational rather than architectural:

  • WebSocket transport uses an unbounded accept queue, so overload can become memory pressure.
  • WebSocket options are bound but not validated at startup.
  • There is no explicit max connection, max pending accept, max send size, or outbound backpressure policy.
  • NATS URL and Redis connection are configured in SOR.Host, but dependency health/readiness is still host-level and minimal.
  • NATS fan-out subscriptions use no queue group by default, which is correct for broadcast but risky if consumers accidentally scale a singleton-style worker.
  • NATS RPC subject and queue naming is duplicated with Gateway dispatch logic instead of centralized in shared contracts.
  • Redis push ignores subscriber count by design, but metrics do not currently expose dropped/offline delivery volume.
  • RedisClientPush.PushAsync accepts a cancellation token but does not pass it to PublishAsync, because StackExchange.Redis Pub/Sub does not use that token directly.
  • NatsMessageRpcResponder starts listener tasks directly and logs unexpected listener termination, but does not restart failed listeners.
  • Readiness, metrics, and integration smoke tests are not packaged with the networking modules.

The plan is to preserve the current abstraction split, then add typed provider options, validation, bounded queues, health checks, delivery-semantic documentation in interfaces, central subject/channel naming helpers, and contract/integration tests around the transport and broker boundaries.

Current Structure Observations

Strong Foundations

  • SOR.Networking.Abstractions keeps service code decoupled from WebSocket, NATS, and Redis.
  • SOR.Networking.WebSocket implements ITransport and IClientConnection without leaking ASP.NET Core WebSocket types through the abstraction boundary.
  • WebSocketClientConnection serializes concurrent sends with SemaphoreSlim.
  • WebSocketClientConnection enforces MaxMessageSizeBytes while assembling frames.
  • WebSocketTransport uses a channel to hand accepted ASP.NET Core WebSocket connections to the Gateway accept loop.
  • SOR.Networking.Nats wraps NATS behind IEventBus.
  • NatsEventBus explicitly documents core NATS at-most-once semantics and reserves JetStream for durable delivery needs.
  • NatsMessageRpcResponder exposes local IMessageHandler implementations over NATS request/reply so Gateway can relay client requests to worker processes.
  • NatsMessageRpcResponder uses queue groups per message type for competing consumers.
  • SOR.Networking.Redis keeps push publishing separate from Gateway's Redis subscription/read side.
  • IClientPush and RedisClientPush both document best-effort/offline-drop behavior.
  • SOR.Shared.Nats.NatsSubjects and SOR.Shared.Redis.RedisKeys already provide shared naming contracts for event subjects and Redis push channels.

Main Problems

  • WebSocketTransport uses Channel.CreateUnbounded<IClientConnection>, so the process has no configured accept backlog limit.
  • WebSocketTransportOptions contains path, ping interval, and max message size, but no validator prevents invalid values.
  • appsettings.json includes WebSocket:AuthTimeoutMs and WebSocket:MaxMessagesPerSecond, but WebSocketTransportOptions only defines path, ping interval, and max message size. That creates config drift between host settings and transport settings.
  • WebSocket keep-alive is configured in SOR.Host separately from the application-level ping loop in WebSocketClientConnection.
  • The ping loop is fire-and-forget; it catches/logs exceptions, but it is not represented as a hosted lifecycle primitive.
  • There is no explicit per-connection outbound queue limit or send timeout policy.
  • NatsEventBus.SubscribeAsync always uses queueGroup: null, which is good for true fan-out but should be hard to misuse for singleton worker workflows.
  • NATS RPC subject/queue naming exists in both Gateway dispatch and NatsMessageRpcResponder as local helper methods.
  • NatsMessageRpcResponder creates one listener task per handler type. If a listener exits unexpectedly, it is logged but not restarted.
  • NatsMessageRpcResponder discovers duplicate handlers at startup, but there is no diagnostics endpoint or test that verifies the registered RPC surface.
  • RedisClientPush publishes bytes and intentionally ignores subscriber count; this should be observable as subscriber_count=0, not invisible.
  • IClientPush does not expose or record delivery hints such as subscriber count, offline drop, or publish latency.
  • SOR.Host has only a minimal /health check and a TODO for Redis, NATS, and Postgres dependency checks.
  • There are no visible contract tests proving message serialization, RPC relay behavior, WebSocket frame limits, cancellation, or shutdown semantics.

Target Networking Shape

Keep the provider split, but make operational policy explicit:

Server/src/SOR.Networking.Abstractions/
  EventBus/
    IEventBus.cs
    EventSubscriptionOptions.cs
    DeliverySemantics.cs
  Messaging/
    IMessageDispatcher.cs
    IMessageHandler.cs
    IMessageSerializer.cs
  Push/
    IClientPush.cs
    ClientPushResult.cs
  Rpc/
    RpcSubjects.cs
    RemoteDispatchOptions.cs
  Transport/
    ITransport.cs
    IClientConnection.cs
    TransportLimits.cs

Server/src/SOR.Networking.WebSocket/
  Configuration/
    WebSocketTransportOptions.cs
    WebSocketTransportOptionsValidator.cs
  Transport/
    WebSocketTransport.cs
    WebSocketClientConnection.cs
  Health/
    WebSocketTransportHealthCheck.cs
  Metrics/
    WebSocketMetrics.cs

Server/src/SOR.Networking.Nats/
  Configuration/
    NatsMessagingOptions.cs
    NatsMessagingOptionsValidator.cs
  EventBus/
    NatsEventBus.cs
  Rpc/
    NatsMessageRpcResponder.cs
  Health/
    NatsHealthCheck.cs
  Metrics/
    NatsMetrics.cs

Server/src/SOR.Networking.Redis/
  Configuration/
    RedisPushOptions.cs
    RedisPushOptionsValidator.cs
  Push/
    RedisClientPush.cs
  Health/
    RedisHealthCheck.cs
  Metrics/
    RedisPushMetrics.cs

12-Factor Assessment

1. Codebase

Status: mostly aligned.

Keep SOR.Networking as shared infrastructure code, not a deployable process. The process boundary remains SOR.Host and role selection remains driven by ENABLED_SERVICES.

Plan:

  • Keep abstractions and provider implementations in separate projects.
  • Move duplicated NATS RPC subject/queue naming into one shared helper.
  • Keep Gateway-specific Redis subscription behavior in SOR.Gateway, but document the full push contract in Networking.
  • Add networking contract tests that can run without service modules.

2. Dependencies

Status: mostly aligned.

Provider dependencies are isolated: NATS packages stay in SOR.Networking.Nats, Redis packages stay in SOR.Networking.Redis, and ASP.NET Core WebSocket types stay in SOR.Networking.WebSocket.

Plan:

  • Keep provider packages out of SOR.Networking.Abstractions.
  • Add test projects per provider where integration dependencies are needed.
  • Avoid introducing direct references from service modules to provider-specific networking projects.

3. Config

Status: partially aligned.

Config is environment-compatible through SOR.Host, but provider options are incomplete and not validated.

Plan:

  • Add validators for WebSocketTransportOptions, NatsMessagingOptions, and RedisPushOptions.
  • Reconcile config drift around AuthTimeoutMs and MaxMessagesPerSecond; either move them to Gateway options or define them in transport options if transport owns them.
  • Promote NATS RPC timeout, queue group prefix, subject prefix, and listener restart policy into typed options.
  • Promote WebSocket accept backlog, max active connections, send timeout, ping interval, and max payload size into typed options.
  • Promote Redis push channel prefix and metrics behavior into typed options.
  • Fail fast on invalid or missing production config.

4. Backing Services

Status: aligned in concept, incomplete operationally.

Redis and NATS are attached backing services. They should be replaceable by changing config and provider registration, not changing business services.

Plan:

  • Add Redis and NATS health checks that can be included in readiness.
  • Separate liveness from readiness: liveness checks process health, readiness checks broker connectivity.
  • Document Redis Pub/Sub as best-effort push, not durable state.
  • Document core NATS as at-most-once eventing, and define when JetStream is required.

5. Build, Release, Run

Status: mostly aligned.

Networking projects are library code loaded by the host at runtime.

Plan:

  • Keep build artifacts environment-neutral.
  • Keep provider selection and provider URLs in config.
  • Add startup validation so bad broker URLs, invalid WebSocket limits, or missing options fail during release/run startup rather than at first message.

6. Processes

Status: mostly aligned.

The networking modules keep connection/process state but do not own durable application state. WebSocket connections are inherently process-local; Redis and NATS externalize cross-process communication.

Plan:

  • Treat WebSocket connections as disposable process state.
  • Treat Redis/NATS as the only cross-process communication boundary.
  • Keep user/game/session state outside Networking.
  • Make offline reconnect behavior rely on persistent service state and client sync, not push replay.

7. Port Binding

Status: aligned through SOR.Host.

Networking libraries do not bind ports directly. SOR.Host owns Kestrel, HTTP, gRPC, and WebSocket upgrade configuration.

Plan:

  • Keep WebSocket path and Kestrel behavior in host-level config.
  • Keep transport registration behind AddWebSocketTransport.
  • Ensure gateway-only WebSocket registration remains role-gated by ENABLED_SERVICES.

8. Concurrency

Status: partially aligned.

The code uses async APIs, channels, and cancellation, but overload behavior is not explicit.

Plan:

  • Replace the unbounded WebSocket accept channel with a bounded channel and a configured full-mode policy.
  • Add metrics for pending accepts, active connections, rejected upgrades, receive payload size, send latency, and disconnect reason.
  • Add per-connection send timeout and max outbound message size.
  • Make NATS subscription type explicit: fan-out subscription vs queue-group subscription.
  • Add listener restart policy or process-fatal behavior for unexpected NATS RPC listener exits.
  • Add tests for duplicate handler registration and RPC queue group distribution.

9. Disposability

Status: partially aligned.

Cancellation is present, but shutdown and drain behavior should be more explicit.

Plan:

  • On shutdown, stop accepting new WebSocket upgrades before waiting for active connections.
  • Add a configured graceful connection drain timeout.
  • Ensure pending accepts are completed/cancelled cleanly.
  • Ensure NATS RPC subscriptions stop cleanly and report listener shutdown.
  • Ensure Redis push publishes fail fast when Redis is unavailable rather than blocking request paths indefinitely.
  • Add integration tests for shutdown cancellation in WebSocket, NATS responder, and Redis push.

10. Dev/Prod Parity

Status: partially aligned.

The host uses local default Redis/NATS config, but parity depends on developers running equivalent backing services.

Plan:

  • Provide a local compose profile for Redis and NATS.
  • Keep local broker behavior close to production defaults.
  • Add a smoke-test command that verifies WebSocket upgrade, NATS publish/subscribe, NATS RPC, and Redis push publish.
  • Avoid hidden in-memory fallbacks in production service registration.

11. Logs

Status: partially aligned.

Logging exists, but operational signals need to be structured and measurable.

Plan:

  • Add structured fields for transport name, connection ID, user ID, subject, queue group, channel, message type, subscriber count, payload size, latency, and failure reason.
  • Add counters for WebSocket accepted/rejected/closed, NATS publish/subscribe failures, RPC requests/errors/timeouts, and Redis push subscriber counts.
  • Keep logs on stdout/stderr through host logging.
  • Avoid logging payload bodies unless a redaction policy exists.

12. Admin Processes

Status: early.

Networking has no explicit one-off diagnostics or repair commands.

Plan:

  • Add broker diagnostics that can run as admin commands or smoke-test jobs.
  • Add a handler registry dump for NATS RPC responder startup.
  • Add an optional subject/channel inventory command for known NatsSubjects, RPC subjects, and Redis keys.
  • If durable messaging is introduced later, add replay/dead-letter admin tooling at that time.

Refactor Phases

Phase 0: Baseline Contracts

  • Add tests for IMessageSerializer wire compatibility across WebSocket, Redis push, and NATS RPC payloads.
  • Add tests for WebSocket max message size handling.
  • Add tests for duplicate IMessageHandler detection in NatsMessageRpcResponder.
  • Add tests for Redis push channel naming.
  • Add tests that Gateway and responder compute the same RPC subject/queue names.

Phase 1: Configuration Cleanup

  • Add typed option classes for NATS messaging and Redis push.
  • Add IValidateOptions<T> validators for WebSocket, NATS, and Redis options.
  • Resolve WebSocket config drift by moving auth/rate-limit options to Gateway or implementing them in transport.
  • Add startup validation for required broker URLs and limits.

Phase 2: Backpressure and Limits

  • Replace unbounded pending WebSocket accepts with bounded channel config.
  • Add MaxActiveConnections.
  • Add explicit reject behavior for overload.
  • Add send timeout and optional max outbound payload size.
  • Add metrics for pending accepts and rejected upgrades.

Phase 3: Subject and Channel Contracts

  • Centralize NATS RPC subject and queue naming in shared code.
  • Keep event subjects in NatsSubjects.
  • Keep Redis push channel names in RedisKeys.
  • Add tests that prevent accidental naming drift across Gateway, responder, and publishers.

Phase 4: Broker Hardening

  • Add NATS health check.
  • Add Redis health check.
  • Add NATS RPC listener restart/fatal policy.
  • Add explicit subscription options for fan-out vs queue-group consumption.
  • Define when core NATS is sufficient and when JetStream is required.

Phase 5: Observability

  • Add metrics wrappers around WebSocket accepts/sends/receives.
  • Add NATS publish, subscribe, and RPC metrics.
  • Add Redis push metrics including subscriber count.
  • Add structured logging fields across transport, push, event bus, and RPC.

Phase 6: Disposability

  • Add graceful WebSocket drain behavior.
  • Add tests for cancellation and shutdown paths.
  • Ensure NATS and Redis dependencies fail quickly when unavailable.
  • Add readiness checks that remove the process from traffic before shutdown drains active work.

Phase 7: Dev/Prod Parity and Runbooks

  • Add local broker startup instructions.
  • Add smoke tests for WebSocket, Redis, NATS event bus, and NATS RPC.
  • Document expected production env vars.
  • Document operational delivery semantics for events, RPC, and push.

Immediate Next Steps

  1. Add WebSocketTransportOptionsValidator and reconcile unused WebSocket config keys.
  2. Centralize RPC subject/queue naming used by Gateway and NatsMessageRpcResponder.
  3. Add Redis/NATS readiness checks and wire them into host readiness.
  4. Add metrics for Redis push subscriber count and NATS RPC handler latency.
  5. Replace the WebSocket accept queue with a bounded, configurable channel.

Definition of Done

  • All networking provider options are typed, validated, and environment-overridable.
  • WebSocket overload behavior is bounded and observable.
  • Redis, NATS, and WebSocket readiness/health behavior is explicit.
  • Event, push, and RPC delivery semantics are documented in interfaces and runbooks.
  • NATS RPC naming cannot drift between Gateway and responder.
  • Integration tests cover WebSocket frame limits, NATS event publish/subscribe, NATS RPC, Redis push publish, and shutdown cancellation.
  • No service module directly depends on concrete provider implementations.