# SOR.Matchmaking 12-Factor Refactor Plan Date: 2026-06-10 Scope reviewed: - `Server/src/SOR.Matchmaking` - Matchmaking persistence in `Server/src/SOR.Database/Contexts/GameDbContext.cs` - Identity rank reads in `Server/src/SOR.Database/Contexts/SorIdentityDbContext.cs` - Matchmaking wiring in `Server/src/SOR.Host` - Push and event dependencies through `IClientPush` and `IEventBus` Reference: https://12factor.net/ ## Executive Summary `SOR.Matchmaking` is a service module registered by `SOR.Host` when `ENABLED_SERVICES` includes `matchmaking`. It exposes handlers for joining, leaving, and checking matchmaking queue status, and it starts `MatchmakerService`, a polling background worker that pairs players by rank range, creates `GameSession` rows, publishes `GameStartedEvent`, and pushes `matchmaking.match_found` notifications to both players. The module already has useful foundations: - Queue state is durable in the `game.matchmaking_entries` table. - Queue join is idempotent in the common case where an active searching entry already exists. - Match creation snapshots ranked rules and board template data into the game session. - Match found delivery uses existing `IClientPush`. - Game lifecycle notification uses existing `IEventBus`. - The worker is cancellation-aware and logs tick failures. The main 12-Factor gaps are around process formation, config, and concurrency: - Request handlers and the matchmaking worker are always registered together. - Poll interval, rank range, range expansion, max range, and expansion cadence are hardcoded constants. - Multiple matchmaking worker processes can load and match the same queue entries unless there is only one instance operationally. - Active queue uniqueness is application-level only; concurrent joins can create duplicate searching entries. - Match creation, queue-entry updates, event publish, and push delivery are not modeled with an outbox/idempotent delivery boundary. - Join does not visibly validate that the selected deck exists, belongs to the account, and is valid for ranked play. - Queue cleanup, stale entry expiration, abandoned initializing sessions, and retry/reconciliation jobs are not explicit. - Readiness is host-level only and does not account for Game DB, Identity DB, push, or event bus dependencies. The plan is to keep `SOR.Matchmaking` as the ranked queue module, but split request handlers from worker registration, add typed config, harden queue invariants with database/lease semantics, introduce a transaction/outbox boundary for match creation, and add observability and operational jobs. ## Current Structure Observations ### Strong Foundations - `AddMatchmakingService` registers `GameDbContext`, `SorIdentityDbContext`, all handlers, and `MatchmakerService`. - `MatchmakingJoinHandler` returns the existing searching entry when found. - `MatchmakingLeaveHandler` marks searching entries cancelled instead of deleting them. - `MatchmakerService` orders queue entries by wait time. - Rank range expands over time to reduce indefinite waits. - Game sessions snapshot ranked rules, board dimensions, turn timer, rewards, and board template JSON. - `GameStartedEvent` is published after game session creation. - Both matched players receive a push payload with session ID and opponent info. ### Main Problems - Matchmaking algorithm parameters are hardcoded in `MatchmakerService`. - `MatchmakerService` starts whenever the matchmaking module is enabled, which couples API handlers and worker role. - The worker loads all searching entries into memory each tick. - There is no distributed lock, row claim, or queue-group style ownership for matching work. - There is no database constraint guaranteeing one active searching entry per account. - `JoinQueueRequest` accepts a deck ID but does not visibly validate deck ownership or deck validity. - Publishing `GameStartedEvent` can fail after the game session and queue entries are committed. - Push delivery can fail after the match is committed. - There is no durable delivery state for match-found notifications. - Stale searching entries and old cancelled/matched entries do not appear to have retention cleanup. - No visible tests cover concurrent joins or multiple workers. ## Target Matchmaking Shape Keep `SOR.Matchmaking` focused on ranked queue orchestration, but split responsibilities: ```text Server/src/SOR.Matchmaking/ Configuration/ MatchmakingOptions.cs MatchmakingWorkerOptions.cs MatchmakingRankOptions.cs MatchmakingOptionsValidator.cs Bootstrap/ MatchmakingRegistration.cs MatchmakingHandlersRegistration.cs MatchmakingWorkerRegistration.cs MatchmakingHealthChecks.cs Contracts/ MatchmakingMessages.cs MatchmakingErrors.cs Application/ Queue/ MatchmakingQueueService.cs QueueEntryValidator.cs QueueStatusService.cs Matching/ MatchmakerWorker.cs MatchCandidateSelector.cs MatchCreationService.cs RankedRuleSnapshotBuilder.cs Delivery/ MatchFoundNotificationService.cs MatchmakingOutboxService.cs Persistence/ IMatchmakingRepository.cs IDeckEligibilityReader.cs IRankReader.cs IMatchmakingLeaseStore.cs IMatchmakingUnitOfWork.cs Jobs/ ExpireStaleQueueEntriesJob.cs ReconcileUndeliveredMatchesJob.cs CleanupOldMatchmakingEntriesJob.cs ``` Desired process roles: - `matchmaking-api`: handles join, leave, and status requests. - `matchmaking-worker`: runs matching ticks and owns match creation. - `matchmaking-jobs`: runs cleanup and reconciliation jobs. Small deployments can co-locate these roles, but registration should make the role boundary explicit. ## 12-Factor Gap Map ### 1. Codebase Mostly aligned. Matchmaking is one small module, but handler and worker concerns are co-registered. Plan: - Keep one module. - Split API handlers, worker, and maintenance jobs into explicit registration methods. - Extract matching algorithm and match creation into testable services. ### 2. Dependencies Mostly aligned. Dependencies are explicit: Game DB, Identity DB, event bus, and client push. Plan: - Add abstractions for rank reads, deck eligibility, queue repository, match creation, and notification delivery. - Keep `IEventBus` and `IClientPush` behind a match delivery/outbox service. - Avoid direct Identity queries in the worker where a rank reader service is sufficient. ### 3. Config Weakly aligned. All matchmaking behavior is hardcoded. Plan: - Add typed `MatchmakingOptions`. - Move poll interval, batch size, rank range, range expansion, max range, stale queue timeout, and worker enablement into config. - Validate options at startup. Candidate config shape: ```json { "Matchmaking": { "ApiEnabled": true, "WorkerEnabled": true, "PollIntervalMs": 5000, "MaxEntriesPerTick": 200, "InitialRankRange": 100, "RangeExpandStep": 100, "MaxRankRange": 500, "ExpandEverySeconds": 30, "StaleQueueEntryMinutes": 15, "RequireValidDeck": true } } ``` ### 4. Backing Services Partially aligned. Matchmaking uses PostgreSQL, NATS, Redis/client push, and Identity rank data, but readiness does not reflect those dependencies. Plan: - Treat Game DB, Identity DB, event bus, and push delivery as attached resources. - Add role-specific readiness checks. - Define degraded mode: API may accept queue operations only when worker/delivery dependencies are healthy, or explicitly reject joins if worker is unavailable. ### 5. Build, Release, Run Mostly aligned. There are no build-time assets and no migrations run here. Plan: - Keep schema migrations outside matchmaking startup. - Include deployment checks for worker replica count and queue ownership mode. - Version algorithm changes through config/release notes because matching behavior is player-visible. ### 6. Processes Partially aligned. API request handling and worker polling are one process role today. Plan: - Split registration into: - `AddMatchmakingCore` - `AddMatchmakingHandlers` - `AddMatchmakingWorker` - `AddMatchmakingJobs` - Let `SOR.Host` decide which roles start. - Validate unsafe combinations, such as multiple workers without leases. ### 7. Port Binding Aligned through `SOR.Host`. Matchmaking does not bind ports directly. Plan: - Keep port binding in Host/Gateway. - Expose matchmaking readiness through host health checks. ### 8. Concurrency Weakly aligned. This is the main risk area. Plan: - Add a database-backed active queue uniqueness rule for one searching entry per account. - Add row claiming with `FOR UPDATE SKIP LOCKED`, optimistic status transitions, or a worker lease model. - Ensure game session creation and queue entry resolution happen in one transaction. - Make match creation idempotent per entry pair. - Add tests for two workers attempting to match the same entries. ### 9. Disposability Partially aligned. The worker accepts cancellation, but tick duration and shutdown boundaries are not explicit. Plan: - Make ticks bounded by max entries and command timeout. - Stop starting new match attempts on shutdown. - Complete or rollback the current transaction before exit. - Move push/event retry to outbox/reconciliation so shutdown does not lose delivery. ### 10. Dev/Prod Parity Partially aligned. Same code runs locally and in production, but worker count and timing behavior are not config-driven. Plan: - Use the same worker code locally and in production. - Configure local shorter poll intervals if useful. - Document single-worker versus multi-worker deployment behavior. - Add a local test profile for queue scenarios. ### 11. Logs Partially aligned. Worker starts/stops, tick errors, and found matches are logged. Plan: - Add structured logs and metrics for joins, leaves, matches, candidate search, stale entries, event publish, push delivery, and worker claim conflicts. - Avoid logging sensitive deck contents. Suggested fields: - `entryId` - `accountId` - `opponentAccountId` - `gameSessionId` - `rankScore` - `opponentRankScore` - `rankRange` - `waitSeconds` - `workerId` - `deliveryState` ### 12. Admin Processes Partially aligned. Cleanup/reconciliation are operational jobs but not explicit. Plan: - Add admin/job processes for stale queue expiration, delivery reconciliation, old entry cleanup, and queue health reports. - Add dry-run reports for cleanup. - Add Admin visibility for current queue depth, age, rank distribution, and worker status. ## Refactor Phases ### Phase 0: Baseline Tests Goals: - Preserve current behavior before restructuring. Work: - Add tests for join idempotency, leave idempotency, and status. - Add tests for rank range expansion. - Add tests for closest-opponent selection. - Add tests for session rule snapshot generation. - Add tests for event/push failure behavior as it works today. Done when: - Current behavior is covered before refactors. ### Phase 1: Typed Options Goals: - Make matchmaking behavior configurable. Work: - Add `MatchmakingOptions`. - Replace hardcoded poll/range constants with options. - Add startup validation. - Preserve current defaults. Done when: - Operators can tune queue behavior per environment without code changes. ### Phase 2: Split API and Worker Registration Goals: - Make process formation explicit. Work: - Split `AddMatchmakingService` into core, handlers, worker, and jobs. - Keep current host behavior initially by calling both handlers and worker registration. - Add host-level role support later if desired. Done when: - A deployment can run API-only, worker-only, or combined matchmaking roles intentionally. ### Phase 3: Extract Queue and Matching Services Goals: - Make matching logic testable and smaller. Work: - Extract `MatchmakingQueueService` for join/leave/status. - Extract `MatchCandidateSelector` for rank-window selection. - Extract `MatchCreationService` for session creation and queue resolution. - Extract `RankedRuleSnapshotBuilder`. Done when: - Handlers and worker are orchestration shells over tested services. ### Phase 4: Deck Eligibility Validation Goals: - Prevent invalid ranked queue entries. Work: - Add `IDeckEligibilityReader`. - Validate deck ownership, deck validity, and ranked eligibility on join. - Decide whether already queued entries should be revalidated before match creation. Done when: - Players cannot queue ranked matchmaking with invalid or unowned decks. ### Phase 5: Concurrency Hardening Goals: - Make matchmaking safe under concurrent joins and multiple workers. Work: - Add database uniqueness or conditional insert for one searching entry per account. - Add queue row claiming or worker leases. - Use transactions around claiming, session creation, and entry resolution. - Add multi-worker concurrency tests. Done when: - Multiple API instances and worker instances cannot duplicate queue entries or matches. ### Phase 6: Outbox and Delivery Reconciliation Goals: - Avoid losing game-started events or match-found pushes after DB commit. Work: - Add an outbox record for `GameStartedEvent` and match-found notifications. - Publish/push from outbox with retry. - Add reconciliation job for undelivered match notifications. - Make push delivery idempotent from the client perspective. Done when: - A committed match cannot silently miss lifecycle event or player notification delivery. ### Phase 7: Queue Maintenance Jobs Goals: - Keep queue data healthy. Work: - Add stale searching entry expiration. - Add cleanup for old cancelled/matched entries. - Add abandoned initializing session detection handoff or integration with game inactivity logic. - Add admin queue reports. Done when: - Queue state does not grow unbounded or remain stuck indefinitely. ### Phase 8: Readiness and Observability Goals: - Make matchmaking operations diagnosable. Work: - Add readiness checks for Game DB, Identity DB, event bus, push, and worker lease. - Add metrics for queue depth, oldest wait time, matches per minute, average rank delta, tick latency, delivery failures, and claim conflicts. - Add runbooks for stuck queues, delivery failures, and worker scaling. Done when: - Operators can see whether players can join, be matched, and receive match notifications. ## Suggested Work Order 1. Add baseline tests for join/leave/status and candidate selection. 2. Add typed `MatchmakingOptions` with current defaults. 3. Split handler and worker registration. 4. Extract queue, selector, snapshot, and match creation services. 5. Add deck eligibility validation. 6. Add active-entry uniqueness and worker row claiming. 7. Add outbox/reconciliation for event and push delivery. 8. Add queue cleanup jobs. 9. Add readiness checks, metrics, dashboards, and runbooks. ## Completion Criteria The refactor is complete when: - Matchmaking API and worker roles are registered explicitly. - Matchmaking behavior is controlled by typed config. - Queue joins are concurrency-safe and idempotent. - Multiple workers are either prevented or coordinated safely. - Match creation is transactional and idempotent. - Game-started events and match-found pushes are delivered through a retryable outbox. - Deck eligibility is validated before ranked matchmaking. - Stale queue and undelivered match states have maintenance jobs. - Readiness and metrics show queue health, worker health, and delivery health. ## Immediate Low-Risk Next Steps - Add `MatchmakingOptions` with defaults matching current constants. - Extract `MatchCandidateSelector` and cover rank-window behavior with tests. - Split `AddMatchmakingService` internally into handler and worker registration methods without changing host behavior. - Add a database/model plan for one active searching entry per account. - Document the current single-worker assumption until row claiming or leases are implemented.