Files
sor/SOR.Matchmaking.12FactorPlan.md
2026-07-02 13:09:05 -04:00

16 KiB

SOR.Matchmaking 12-Factor Refactor Plan

Date: 2026-06-10

Scope reviewed:

  • Server/src/SOR.Matchmaking
  • Matchmaking persistence in Server/src/SOR.Database/Contexts/GameDbContext.cs
  • Identity rank reads in Server/src/SOR.Database/Contexts/SorIdentityDbContext.cs
  • Matchmaking wiring in Server/src/SOR.Host
  • Push and event dependencies through IClientPush and IEventBus

Reference: https://12factor.net/

Executive Summary

SOR.Matchmaking is a service module registered by SOR.Host when ENABLED_SERVICES includes matchmaking. It exposes handlers for joining, leaving, and checking matchmaking queue status, and it starts MatchmakerService, a polling background worker that pairs players by rank range, creates GameSession rows, publishes GameStartedEvent, and pushes matchmaking.match_found notifications to both players.

The module already has useful foundations:

  • Queue state is durable in the game.matchmaking_entries table.
  • Queue join is idempotent in the common case where an active searching entry already exists.
  • Match creation snapshots ranked rules and board template data into the game session.
  • Match found delivery uses existing IClientPush.
  • Game lifecycle notification uses existing IEventBus.
  • The worker is cancellation-aware and logs tick failures.

The main 12-Factor gaps are around process formation, config, and concurrency:

  • Request handlers and the matchmaking worker are always registered together.
  • Poll interval, rank range, range expansion, max range, and expansion cadence are hardcoded constants.
  • Multiple matchmaking worker processes can load and match the same queue entries unless there is only one instance operationally.
  • Active queue uniqueness is application-level only; concurrent joins can create duplicate searching entries.
  • Match creation, queue-entry updates, event publish, and push delivery are not modeled with an outbox/idempotent delivery boundary.
  • Join does not visibly validate that the selected deck exists, belongs to the account, and is valid for ranked play.
  • Queue cleanup, stale entry expiration, abandoned initializing sessions, and retry/reconciliation jobs are not explicit.
  • Readiness is host-level only and does not account for Game DB, Identity DB, push, or event bus dependencies.

The plan is to keep SOR.Matchmaking as the ranked queue module, but split request handlers from worker registration, add typed config, harden queue invariants with database/lease semantics, introduce a transaction/outbox boundary for match creation, and add observability and operational jobs.

Current Structure Observations

Strong Foundations

  • AddMatchmakingService registers GameDbContext, SorIdentityDbContext, all handlers, and MatchmakerService.
  • MatchmakingJoinHandler returns the existing searching entry when found.
  • MatchmakingLeaveHandler marks searching entries cancelled instead of deleting them.
  • MatchmakerService orders queue entries by wait time.
  • Rank range expands over time to reduce indefinite waits.
  • Game sessions snapshot ranked rules, board dimensions, turn timer, rewards, and board template JSON.
  • GameStartedEvent is published after game session creation.
  • Both matched players receive a push payload with session ID and opponent info.

Main Problems

  • Matchmaking algorithm parameters are hardcoded in MatchmakerService.
  • MatchmakerService starts whenever the matchmaking module is enabled, which couples API handlers and worker role.
  • The worker loads all searching entries into memory each tick.
  • There is no distributed lock, row claim, or queue-group style ownership for matching work.
  • There is no database constraint guaranteeing one active searching entry per account.
  • JoinQueueRequest accepts a deck ID but does not visibly validate deck ownership or deck validity.
  • Publishing GameStartedEvent can fail after the game session and queue entries are committed.
  • Push delivery can fail after the match is committed.
  • There is no durable delivery state for match-found notifications.
  • Stale searching entries and old cancelled/matched entries do not appear to have retention cleanup.
  • No visible tests cover concurrent joins or multiple workers.

Target Matchmaking Shape

Keep SOR.Matchmaking focused on ranked queue orchestration, but split responsibilities:

Server/src/SOR.Matchmaking/
  Configuration/
    MatchmakingOptions.cs
    MatchmakingWorkerOptions.cs
    MatchmakingRankOptions.cs
    MatchmakingOptionsValidator.cs
  Bootstrap/
    MatchmakingRegistration.cs
    MatchmakingHandlersRegistration.cs
    MatchmakingWorkerRegistration.cs
    MatchmakingHealthChecks.cs
  Contracts/
    MatchmakingMessages.cs
    MatchmakingErrors.cs
  Application/
    Queue/
      MatchmakingQueueService.cs
      QueueEntryValidator.cs
      QueueStatusService.cs
    Matching/
      MatchmakerWorker.cs
      MatchCandidateSelector.cs
      MatchCreationService.cs
      RankedRuleSnapshotBuilder.cs
    Delivery/
      MatchFoundNotificationService.cs
      MatchmakingOutboxService.cs
  Persistence/
    IMatchmakingRepository.cs
    IDeckEligibilityReader.cs
    IRankReader.cs
    IMatchmakingLeaseStore.cs
    IMatchmakingUnitOfWork.cs
  Jobs/
    ExpireStaleQueueEntriesJob.cs
    ReconcileUndeliveredMatchesJob.cs
    CleanupOldMatchmakingEntriesJob.cs

Desired process roles:

  • matchmaking-api: handles join, leave, and status requests.
  • matchmaking-worker: runs matching ticks and owns match creation.
  • matchmaking-jobs: runs cleanup and reconciliation jobs.

Small deployments can co-locate these roles, but registration should make the role boundary explicit.

12-Factor Gap Map

1. Codebase

Mostly aligned. Matchmaking is one small module, but handler and worker concerns are co-registered.

Plan:

  • Keep one module.
  • Split API handlers, worker, and maintenance jobs into explicit registration methods.
  • Extract matching algorithm and match creation into testable services.

2. Dependencies

Mostly aligned. Dependencies are explicit: Game DB, Identity DB, event bus, and client push.

Plan:

  • Add abstractions for rank reads, deck eligibility, queue repository, match creation, and notification delivery.
  • Keep IEventBus and IClientPush behind a match delivery/outbox service.
  • Avoid direct Identity queries in the worker where a rank reader service is sufficient.

3. Config

Weakly aligned. All matchmaking behavior is hardcoded.

Plan:

  • Add typed MatchmakingOptions.
  • Move poll interval, batch size, rank range, range expansion, max range, stale queue timeout, and worker enablement into config.
  • Validate options at startup.

Candidate config shape:

{
  "Matchmaking": {
    "ApiEnabled": true,
    "WorkerEnabled": true,
    "PollIntervalMs": 5000,
    "MaxEntriesPerTick": 200,
    "InitialRankRange": 100,
    "RangeExpandStep": 100,
    "MaxRankRange": 500,
    "ExpandEverySeconds": 30,
    "StaleQueueEntryMinutes": 15,
    "RequireValidDeck": true
  }
}

4. Backing Services

Partially aligned. Matchmaking uses PostgreSQL, NATS, Redis/client push, and Identity rank data, but readiness does not reflect those dependencies.

Plan:

  • Treat Game DB, Identity DB, event bus, and push delivery as attached resources.
  • Add role-specific readiness checks.
  • Define degraded mode: API may accept queue operations only when worker/delivery dependencies are healthy, or explicitly reject joins if worker is unavailable.

5. Build, Release, Run

Mostly aligned. There are no build-time assets and no migrations run here.

Plan:

  • Keep schema migrations outside matchmaking startup.
  • Include deployment checks for worker replica count and queue ownership mode.
  • Version algorithm changes through config/release notes because matching behavior is player-visible.

6. Processes

Partially aligned. API request handling and worker polling are one process role today.

Plan:

  • Split registration into:
    • AddMatchmakingCore
    • AddMatchmakingHandlers
    • AddMatchmakingWorker
    • AddMatchmakingJobs
  • Let SOR.Host decide which roles start.
  • Validate unsafe combinations, such as multiple workers without leases.

7. Port Binding

Aligned through SOR.Host. Matchmaking does not bind ports directly.

Plan:

  • Keep port binding in Host/Gateway.
  • Expose matchmaking readiness through host health checks.

8. Concurrency

Weakly aligned. This is the main risk area.

Plan:

  • Add a database-backed active queue uniqueness rule for one searching entry per account.
  • Add row claiming with FOR UPDATE SKIP LOCKED, optimistic status transitions, or a worker lease model.
  • Ensure game session creation and queue entry resolution happen in one transaction.
  • Make match creation idempotent per entry pair.
  • Add tests for two workers attempting to match the same entries.

9. Disposability

Partially aligned. The worker accepts cancellation, but tick duration and shutdown boundaries are not explicit.

Plan:

  • Make ticks bounded by max entries and command timeout.
  • Stop starting new match attempts on shutdown.
  • Complete or rollback the current transaction before exit.
  • Move push/event retry to outbox/reconciliation so shutdown does not lose delivery.

10. Dev/Prod Parity

Partially aligned. Same code runs locally and in production, but worker count and timing behavior are not config-driven.

Plan:

  • Use the same worker code locally and in production.
  • Configure local shorter poll intervals if useful.
  • Document single-worker versus multi-worker deployment behavior.
  • Add a local test profile for queue scenarios.

11. Logs

Partially aligned. Worker starts/stops, tick errors, and found matches are logged.

Plan:

  • Add structured logs and metrics for joins, leaves, matches, candidate search, stale entries, event publish, push delivery, and worker claim conflicts.
  • Avoid logging sensitive deck contents.

Suggested fields:

  • entryId
  • accountId
  • opponentAccountId
  • gameSessionId
  • rankScore
  • opponentRankScore
  • rankRange
  • waitSeconds
  • workerId
  • deliveryState

12. Admin Processes

Partially aligned. Cleanup/reconciliation are operational jobs but not explicit.

Plan:

  • Add admin/job processes for stale queue expiration, delivery reconciliation, old entry cleanup, and queue health reports.
  • Add dry-run reports for cleanup.
  • Add Admin visibility for current queue depth, age, rank distribution, and worker status.

Refactor Phases

Phase 0: Baseline Tests

Goals:

  • Preserve current behavior before restructuring.

Work:

  • Add tests for join idempotency, leave idempotency, and status.
  • Add tests for rank range expansion.
  • Add tests for closest-opponent selection.
  • Add tests for session rule snapshot generation.
  • Add tests for event/push failure behavior as it works today.

Done when:

  • Current behavior is covered before refactors.

Phase 1: Typed Options

Goals:

  • Make matchmaking behavior configurable.

Work:

  • Add MatchmakingOptions.
  • Replace hardcoded poll/range constants with options.
  • Add startup validation.
  • Preserve current defaults.

Done when:

  • Operators can tune queue behavior per environment without code changes.

Phase 2: Split API and Worker Registration

Goals:

  • Make process formation explicit.

Work:

  • Split AddMatchmakingService into core, handlers, worker, and jobs.
  • Keep current host behavior initially by calling both handlers and worker registration.
  • Add host-level role support later if desired.

Done when:

  • A deployment can run API-only, worker-only, or combined matchmaking roles intentionally.

Phase 3: Extract Queue and Matching Services

Goals:

  • Make matching logic testable and smaller.

Work:

  • Extract MatchmakingQueueService for join/leave/status.
  • Extract MatchCandidateSelector for rank-window selection.
  • Extract MatchCreationService for session creation and queue resolution.
  • Extract RankedRuleSnapshotBuilder.

Done when:

  • Handlers and worker are orchestration shells over tested services.

Phase 4: Deck Eligibility Validation

Goals:

  • Prevent invalid ranked queue entries.

Work:

  • Add IDeckEligibilityReader.
  • Validate deck ownership, deck validity, and ranked eligibility on join.
  • Decide whether already queued entries should be revalidated before match creation.

Done when:

  • Players cannot queue ranked matchmaking with invalid or unowned decks.

Phase 5: Concurrency Hardening

Goals:

  • Make matchmaking safe under concurrent joins and multiple workers.

Work:

  • Add database uniqueness or conditional insert for one searching entry per account.
  • Add queue row claiming or worker leases.
  • Use transactions around claiming, session creation, and entry resolution.
  • Add multi-worker concurrency tests.

Done when:

  • Multiple API instances and worker instances cannot duplicate queue entries or matches.

Phase 6: Outbox and Delivery Reconciliation

Goals:

  • Avoid losing game-started events or match-found pushes after DB commit.

Work:

  • Add an outbox record for GameStartedEvent and match-found notifications.
  • Publish/push from outbox with retry.
  • Add reconciliation job for undelivered match notifications.
  • Make push delivery idempotent from the client perspective.

Done when:

  • A committed match cannot silently miss lifecycle event or player notification delivery.

Phase 7: Queue Maintenance Jobs

Goals:

  • Keep queue data healthy.

Work:

  • Add stale searching entry expiration.
  • Add cleanup for old cancelled/matched entries.
  • Add abandoned initializing session detection handoff or integration with game inactivity logic.
  • Add admin queue reports.

Done when:

  • Queue state does not grow unbounded or remain stuck indefinitely.

Phase 8: Readiness and Observability

Goals:

  • Make matchmaking operations diagnosable.

Work:

  • Add readiness checks for Game DB, Identity DB, event bus, push, and worker lease.
  • Add metrics for queue depth, oldest wait time, matches per minute, average rank delta, tick latency, delivery failures, and claim conflicts.
  • Add runbooks for stuck queues, delivery failures, and worker scaling.

Done when:

  • Operators can see whether players can join, be matched, and receive match notifications.

Suggested Work Order

  1. Add baseline tests for join/leave/status and candidate selection.
  2. Add typed MatchmakingOptions with current defaults.
  3. Split handler and worker registration.
  4. Extract queue, selector, snapshot, and match creation services.
  5. Add deck eligibility validation.
  6. Add active-entry uniqueness and worker row claiming.
  7. Add outbox/reconciliation for event and push delivery.
  8. Add queue cleanup jobs.
  9. Add readiness checks, metrics, dashboards, and runbooks.

Completion Criteria

The refactor is complete when:

  • Matchmaking API and worker roles are registered explicitly.
  • Matchmaking behavior is controlled by typed config.
  • Queue joins are concurrency-safe and idempotent.
  • Multiple workers are either prevented or coordinated safely.
  • Match creation is transactional and idempotent.
  • Game-started events and match-found pushes are delivered through a retryable outbox.
  • Deck eligibility is validated before ranked matchmaking.
  • Stale queue and undelivered match states have maintenance jobs.
  • Readiness and metrics show queue health, worker health, and delivery health.

Immediate Low-Risk Next Steps

  • Add MatchmakingOptions with defaults matching current constants.
  • Extract MatchCandidateSelector and cover rank-window behavior with tests.
  • Split AddMatchmakingService internally into handler and worker registration methods without changing host behavior.
  • Add a database/model plan for one active searching entry per account.
  • Document the current single-worker assumption until row claiming or leases are implemented.