369 lines
17 KiB
Markdown
369 lines
17 KiB
Markdown
# SOR.Networking 12-Factor Refactor Plan
|
|
|
|
Date: 2026-06-10
|
|
|
|
Scope reviewed:
|
|
- `Server/src/SOR.Networking.Abstractions`
|
|
- `Server/src/SOR.Networking.WebSocket`
|
|
- `Server/src/SOR.Networking.Nats`
|
|
- `Server/src/SOR.Networking.Redis`
|
|
- Networking wiring in `Server/src/SOR.Host`
|
|
- Gateway-facing usage in `Server/src/SOR.Gateway`
|
|
- Shared subject/channel contracts in `Server/src/SOR.Shared`
|
|
|
|
Reference: https://12factor.net/
|
|
|
|
## Executive Summary
|
|
|
|
`SOR.Networking` is the shared transport and messaging layer for the server. It defines the abstractions used by service modules and supplies concrete implementations for WebSocket client transport, NATS event/RPC messaging, and Redis Pub/Sub client push.
|
|
|
|
The module already has strong 12-factor foundations:
|
|
|
|
- Application code depends on abstractions: `ITransport`, `IClientConnection`, `IEventBus`, and `IClientPush`.
|
|
- Concrete providers are isolated in separate projects: WebSocket, NATS, and Redis.
|
|
- `SOR.Host` is the composition root that wires concrete infrastructure to abstractions.
|
|
- Redis and NATS are treated as attached backing services.
|
|
- WebSocket, NATS, and Redis lifetimes are mostly singleton/process-scoped.
|
|
- Event bus delivery semantics are documented as core NATS at-most-once.
|
|
- Redis push delivery semantics are documented as best-effort.
|
|
- NATS RPC responders use queue groups for handler request distribution.
|
|
|
|
The main 12-factor gaps are operational rather than architectural:
|
|
|
|
- WebSocket transport uses an unbounded accept queue, so overload can become memory pressure.
|
|
- WebSocket options are bound but not validated at startup.
|
|
- There is no explicit max connection, max pending accept, max send size, or outbound backpressure policy.
|
|
- NATS URL and Redis connection are configured in `SOR.Host`, but dependency health/readiness is still host-level and minimal.
|
|
- NATS fan-out subscriptions use no queue group by default, which is correct for broadcast but risky if consumers accidentally scale a singleton-style worker.
|
|
- NATS RPC subject and queue naming is duplicated with Gateway dispatch logic instead of centralized in shared contracts.
|
|
- Redis push ignores subscriber count by design, but metrics do not currently expose dropped/offline delivery volume.
|
|
- `RedisClientPush.PushAsync` accepts a cancellation token but does not pass it to `PublishAsync`, because StackExchange.Redis Pub/Sub does not use that token directly.
|
|
- `NatsMessageRpcResponder` starts listener tasks directly and logs unexpected listener termination, but does not restart failed listeners.
|
|
- Readiness, metrics, and integration smoke tests are not packaged with the networking modules.
|
|
|
|
The plan is to preserve the current abstraction split, then add typed provider options, validation, bounded queues, health checks, delivery-semantic documentation in interfaces, central subject/channel naming helpers, and contract/integration tests around the transport and broker boundaries.
|
|
|
|
## Current Structure Observations
|
|
|
|
### Strong Foundations
|
|
|
|
- `SOR.Networking.Abstractions` keeps service code decoupled from WebSocket, NATS, and Redis.
|
|
- `SOR.Networking.WebSocket` implements `ITransport` and `IClientConnection` without leaking ASP.NET Core WebSocket types through the abstraction boundary.
|
|
- `WebSocketClientConnection` serializes concurrent sends with `SemaphoreSlim`.
|
|
- `WebSocketClientConnection` enforces `MaxMessageSizeBytes` while assembling frames.
|
|
- `WebSocketTransport` uses a channel to hand accepted ASP.NET Core WebSocket connections to the Gateway accept loop.
|
|
- `SOR.Networking.Nats` wraps NATS behind `IEventBus`.
|
|
- `NatsEventBus` explicitly documents core NATS at-most-once semantics and reserves JetStream for durable delivery needs.
|
|
- `NatsMessageRpcResponder` exposes local `IMessageHandler` implementations over NATS request/reply so Gateway can relay client requests to worker processes.
|
|
- `NatsMessageRpcResponder` uses queue groups per message type for competing consumers.
|
|
- `SOR.Networking.Redis` keeps push publishing separate from Gateway's Redis subscription/read side.
|
|
- `IClientPush` and `RedisClientPush` both document best-effort/offline-drop behavior.
|
|
- `SOR.Shared.Nats.NatsSubjects` and `SOR.Shared.Redis.RedisKeys` already provide shared naming contracts for event subjects and Redis push channels.
|
|
|
|
### Main Problems
|
|
|
|
- `WebSocketTransport` uses `Channel.CreateUnbounded<IClientConnection>`, so the process has no configured accept backlog limit.
|
|
- `WebSocketTransportOptions` contains path, ping interval, and max message size, but no validator prevents invalid values.
|
|
- `appsettings.json` includes `WebSocket:AuthTimeoutMs` and `WebSocket:MaxMessagesPerSecond`, but `WebSocketTransportOptions` only defines path, ping interval, and max message size. That creates config drift between host settings and transport settings.
|
|
- WebSocket keep-alive is configured in `SOR.Host` separately from the application-level ping loop in `WebSocketClientConnection`.
|
|
- The ping loop is fire-and-forget; it catches/logs exceptions, but it is not represented as a hosted lifecycle primitive.
|
|
- There is no explicit per-connection outbound queue limit or send timeout policy.
|
|
- `NatsEventBus.SubscribeAsync` always uses `queueGroup: null`, which is good for true fan-out but should be hard to misuse for singleton worker workflows.
|
|
- NATS RPC subject/queue naming exists in both Gateway dispatch and `NatsMessageRpcResponder` as local helper methods.
|
|
- `NatsMessageRpcResponder` creates one listener task per handler type. If a listener exits unexpectedly, it is logged but not restarted.
|
|
- `NatsMessageRpcResponder` discovers duplicate handlers at startup, but there is no diagnostics endpoint or test that verifies the registered RPC surface.
|
|
- `RedisClientPush` publishes bytes and intentionally ignores subscriber count; this should be observable as `subscriber_count=0`, not invisible.
|
|
- `IClientPush` does not expose or record delivery hints such as subscriber count, offline drop, or publish latency.
|
|
- `SOR.Host` has only a minimal `/health` check and a TODO for Redis, NATS, and Postgres dependency checks.
|
|
- There are no visible contract tests proving message serialization, RPC relay behavior, WebSocket frame limits, cancellation, or shutdown semantics.
|
|
|
|
## Target Networking Shape
|
|
|
|
Keep the provider split, but make operational policy explicit:
|
|
|
|
```text
|
|
Server/src/SOR.Networking.Abstractions/
|
|
EventBus/
|
|
IEventBus.cs
|
|
EventSubscriptionOptions.cs
|
|
DeliverySemantics.cs
|
|
Messaging/
|
|
IMessageDispatcher.cs
|
|
IMessageHandler.cs
|
|
IMessageSerializer.cs
|
|
Push/
|
|
IClientPush.cs
|
|
ClientPushResult.cs
|
|
Rpc/
|
|
RpcSubjects.cs
|
|
RemoteDispatchOptions.cs
|
|
Transport/
|
|
ITransport.cs
|
|
IClientConnection.cs
|
|
TransportLimits.cs
|
|
|
|
Server/src/SOR.Networking.WebSocket/
|
|
Configuration/
|
|
WebSocketTransportOptions.cs
|
|
WebSocketTransportOptionsValidator.cs
|
|
Transport/
|
|
WebSocketTransport.cs
|
|
WebSocketClientConnection.cs
|
|
Health/
|
|
WebSocketTransportHealthCheck.cs
|
|
Metrics/
|
|
WebSocketMetrics.cs
|
|
|
|
Server/src/SOR.Networking.Nats/
|
|
Configuration/
|
|
NatsMessagingOptions.cs
|
|
NatsMessagingOptionsValidator.cs
|
|
EventBus/
|
|
NatsEventBus.cs
|
|
Rpc/
|
|
NatsMessageRpcResponder.cs
|
|
Health/
|
|
NatsHealthCheck.cs
|
|
Metrics/
|
|
NatsMetrics.cs
|
|
|
|
Server/src/SOR.Networking.Redis/
|
|
Configuration/
|
|
RedisPushOptions.cs
|
|
RedisPushOptionsValidator.cs
|
|
Push/
|
|
RedisClientPush.cs
|
|
Health/
|
|
RedisHealthCheck.cs
|
|
Metrics/
|
|
RedisPushMetrics.cs
|
|
```
|
|
|
|
## 12-Factor Assessment
|
|
|
|
### 1. Codebase
|
|
|
|
Status: mostly aligned.
|
|
|
|
Keep `SOR.Networking` as shared infrastructure code, not a deployable process. The process boundary remains `SOR.Host` and role selection remains driven by `ENABLED_SERVICES`.
|
|
|
|
Plan:
|
|
- Keep abstractions and provider implementations in separate projects.
|
|
- Move duplicated NATS RPC subject/queue naming into one shared helper.
|
|
- Keep Gateway-specific Redis subscription behavior in `SOR.Gateway`, but document the full push contract in Networking.
|
|
- Add networking contract tests that can run without service modules.
|
|
|
|
### 2. Dependencies
|
|
|
|
Status: mostly aligned.
|
|
|
|
Provider dependencies are isolated: NATS packages stay in `SOR.Networking.Nats`, Redis packages stay in `SOR.Networking.Redis`, and ASP.NET Core WebSocket types stay in `SOR.Networking.WebSocket`.
|
|
|
|
Plan:
|
|
- Keep provider packages out of `SOR.Networking.Abstractions`.
|
|
- Add test projects per provider where integration dependencies are needed.
|
|
- Avoid introducing direct references from service modules to provider-specific networking projects.
|
|
|
|
### 3. Config
|
|
|
|
Status: partially aligned.
|
|
|
|
Config is environment-compatible through `SOR.Host`, but provider options are incomplete and not validated.
|
|
|
|
Plan:
|
|
- Add validators for `WebSocketTransportOptions`, `NatsMessagingOptions`, and `RedisPushOptions`.
|
|
- Reconcile config drift around `AuthTimeoutMs` and `MaxMessagesPerSecond`; either move them to Gateway options or define them in transport options if transport owns them.
|
|
- Promote NATS RPC timeout, queue group prefix, subject prefix, and listener restart policy into typed options.
|
|
- Promote WebSocket accept backlog, max active connections, send timeout, ping interval, and max payload size into typed options.
|
|
- Promote Redis push channel prefix and metrics behavior into typed options.
|
|
- Fail fast on invalid or missing production config.
|
|
|
|
### 4. Backing Services
|
|
|
|
Status: aligned in concept, incomplete operationally.
|
|
|
|
Redis and NATS are attached backing services. They should be replaceable by changing config and provider registration, not changing business services.
|
|
|
|
Plan:
|
|
- Add Redis and NATS health checks that can be included in readiness.
|
|
- Separate liveness from readiness: liveness checks process health, readiness checks broker connectivity.
|
|
- Document Redis Pub/Sub as best-effort push, not durable state.
|
|
- Document core NATS as at-most-once eventing, and define when JetStream is required.
|
|
|
|
### 5. Build, Release, Run
|
|
|
|
Status: mostly aligned.
|
|
|
|
Networking projects are library code loaded by the host at runtime.
|
|
|
|
Plan:
|
|
- Keep build artifacts environment-neutral.
|
|
- Keep provider selection and provider URLs in config.
|
|
- Add startup validation so bad broker URLs, invalid WebSocket limits, or missing options fail during release/run startup rather than at first message.
|
|
|
|
### 6. Processes
|
|
|
|
Status: mostly aligned.
|
|
|
|
The networking modules keep connection/process state but do not own durable application state. WebSocket connections are inherently process-local; Redis and NATS externalize cross-process communication.
|
|
|
|
Plan:
|
|
- Treat WebSocket connections as disposable process state.
|
|
- Treat Redis/NATS as the only cross-process communication boundary.
|
|
- Keep user/game/session state outside Networking.
|
|
- Make offline reconnect behavior rely on persistent service state and client sync, not push replay.
|
|
|
|
### 7. Port Binding
|
|
|
|
Status: aligned through `SOR.Host`.
|
|
|
|
Networking libraries do not bind ports directly. `SOR.Host` owns Kestrel, HTTP, gRPC, and WebSocket upgrade configuration.
|
|
|
|
Plan:
|
|
- Keep WebSocket path and Kestrel behavior in host-level config.
|
|
- Keep transport registration behind `AddWebSocketTransport`.
|
|
- Ensure gateway-only WebSocket registration remains role-gated by `ENABLED_SERVICES`.
|
|
|
|
### 8. Concurrency
|
|
|
|
Status: partially aligned.
|
|
|
|
The code uses async APIs, channels, and cancellation, but overload behavior is not explicit.
|
|
|
|
Plan:
|
|
- Replace the unbounded WebSocket accept channel with a bounded channel and a configured full-mode policy.
|
|
- Add metrics for pending accepts, active connections, rejected upgrades, receive payload size, send latency, and disconnect reason.
|
|
- Add per-connection send timeout and max outbound message size.
|
|
- Make NATS subscription type explicit: fan-out subscription vs queue-group subscription.
|
|
- Add listener restart policy or process-fatal behavior for unexpected NATS RPC listener exits.
|
|
- Add tests for duplicate handler registration and RPC queue group distribution.
|
|
|
|
### 9. Disposability
|
|
|
|
Status: partially aligned.
|
|
|
|
Cancellation is present, but shutdown and drain behavior should be more explicit.
|
|
|
|
Plan:
|
|
- On shutdown, stop accepting new WebSocket upgrades before waiting for active connections.
|
|
- Add a configured graceful connection drain timeout.
|
|
- Ensure pending accepts are completed/cancelled cleanly.
|
|
- Ensure NATS RPC subscriptions stop cleanly and report listener shutdown.
|
|
- Ensure Redis push publishes fail fast when Redis is unavailable rather than blocking request paths indefinitely.
|
|
- Add integration tests for shutdown cancellation in WebSocket, NATS responder, and Redis push.
|
|
|
|
### 10. Dev/Prod Parity
|
|
|
|
Status: partially aligned.
|
|
|
|
The host uses local default Redis/NATS config, but parity depends on developers running equivalent backing services.
|
|
|
|
Plan:
|
|
- Provide a local compose profile for Redis and NATS.
|
|
- Keep local broker behavior close to production defaults.
|
|
- Add a smoke-test command that verifies WebSocket upgrade, NATS publish/subscribe, NATS RPC, and Redis push publish.
|
|
- Avoid hidden in-memory fallbacks in production service registration.
|
|
|
|
### 11. Logs
|
|
|
|
Status: partially aligned.
|
|
|
|
Logging exists, but operational signals need to be structured and measurable.
|
|
|
|
Plan:
|
|
- Add structured fields for transport name, connection ID, user ID, subject, queue group, channel, message type, subscriber count, payload size, latency, and failure reason.
|
|
- Add counters for WebSocket accepted/rejected/closed, NATS publish/subscribe failures, RPC requests/errors/timeouts, and Redis push subscriber counts.
|
|
- Keep logs on stdout/stderr through host logging.
|
|
- Avoid logging payload bodies unless a redaction policy exists.
|
|
|
|
### 12. Admin Processes
|
|
|
|
Status: early.
|
|
|
|
Networking has no explicit one-off diagnostics or repair commands.
|
|
|
|
Plan:
|
|
- Add broker diagnostics that can run as admin commands or smoke-test jobs.
|
|
- Add a handler registry dump for NATS RPC responder startup.
|
|
- Add an optional subject/channel inventory command for known `NatsSubjects`, RPC subjects, and Redis keys.
|
|
- If durable messaging is introduced later, add replay/dead-letter admin tooling at that time.
|
|
|
|
## Refactor Phases
|
|
|
|
### Phase 0: Baseline Contracts
|
|
|
|
- Add tests for `IMessageSerializer` wire compatibility across WebSocket, Redis push, and NATS RPC payloads.
|
|
- Add tests for WebSocket max message size handling.
|
|
- Add tests for duplicate `IMessageHandler` detection in `NatsMessageRpcResponder`.
|
|
- Add tests for Redis push channel naming.
|
|
- Add tests that Gateway and responder compute the same RPC subject/queue names.
|
|
|
|
### Phase 1: Configuration Cleanup
|
|
|
|
- Add typed option classes for NATS messaging and Redis push.
|
|
- Add `IValidateOptions<T>` validators for WebSocket, NATS, and Redis options.
|
|
- Resolve `WebSocket` config drift by moving auth/rate-limit options to Gateway or implementing them in transport.
|
|
- Add startup validation for required broker URLs and limits.
|
|
|
|
### Phase 2: Backpressure and Limits
|
|
|
|
- Replace unbounded pending WebSocket accepts with bounded channel config.
|
|
- Add `MaxActiveConnections`.
|
|
- Add explicit reject behavior for overload.
|
|
- Add send timeout and optional max outbound payload size.
|
|
- Add metrics for pending accepts and rejected upgrades.
|
|
|
|
### Phase 3: Subject and Channel Contracts
|
|
|
|
- Centralize NATS RPC subject and queue naming in shared code.
|
|
- Keep event subjects in `NatsSubjects`.
|
|
- Keep Redis push channel names in `RedisKeys`.
|
|
- Add tests that prevent accidental naming drift across Gateway, responder, and publishers.
|
|
|
|
### Phase 4: Broker Hardening
|
|
|
|
- Add NATS health check.
|
|
- Add Redis health check.
|
|
- Add NATS RPC listener restart/fatal policy.
|
|
- Add explicit subscription options for fan-out vs queue-group consumption.
|
|
- Define when core NATS is sufficient and when JetStream is required.
|
|
|
|
### Phase 5: Observability
|
|
|
|
- Add metrics wrappers around WebSocket accepts/sends/receives.
|
|
- Add NATS publish, subscribe, and RPC metrics.
|
|
- Add Redis push metrics including subscriber count.
|
|
- Add structured logging fields across transport, push, event bus, and RPC.
|
|
|
|
### Phase 6: Disposability
|
|
|
|
- Add graceful WebSocket drain behavior.
|
|
- Add tests for cancellation and shutdown paths.
|
|
- Ensure NATS and Redis dependencies fail quickly when unavailable.
|
|
- Add readiness checks that remove the process from traffic before shutdown drains active work.
|
|
|
|
### Phase 7: Dev/Prod Parity and Runbooks
|
|
|
|
- Add local broker startup instructions.
|
|
- Add smoke tests for WebSocket, Redis, NATS event bus, and NATS RPC.
|
|
- Document expected production env vars.
|
|
- Document operational delivery semantics for events, RPC, and push.
|
|
|
|
## Immediate Next Steps
|
|
|
|
1. Add `WebSocketTransportOptionsValidator` and reconcile unused WebSocket config keys.
|
|
2. Centralize RPC subject/queue naming used by Gateway and `NatsMessageRpcResponder`.
|
|
3. Add Redis/NATS readiness checks and wire them into host readiness.
|
|
4. Add metrics for Redis push subscriber count and NATS RPC handler latency.
|
|
5. Replace the WebSocket accept queue with a bounded, configurable channel.
|
|
|
|
## Definition of Done
|
|
|
|
- All networking provider options are typed, validated, and environment-overridable.
|
|
- WebSocket overload behavior is bounded and observable.
|
|
- Redis, NATS, and WebSocket readiness/health behavior is explicit.
|
|
- Event, push, and RPC delivery semantics are documented in interfaces and runbooks.
|
|
- NATS RPC naming cannot drift between Gateway and responder.
|
|
- Integration tests cover WebSocket frame limits, NATS event publish/subscribe, NATS RPC, Redis push publish, and shutdown cancellation.
|
|
- No service module directly depends on concrete provider implementations.
|