426 lines
16 KiB
Markdown
426 lines
16 KiB
Markdown
# SOR.Matchmaking 12-Factor Refactor Plan
|
|
|
|
Date: 2026-06-10
|
|
|
|
Scope reviewed:
|
|
- `Server/src/SOR.Matchmaking`
|
|
- Matchmaking persistence in `Server/src/SOR.Database/Contexts/GameDbContext.cs`
|
|
- Identity rank reads in `Server/src/SOR.Database/Contexts/SorIdentityDbContext.cs`
|
|
- Matchmaking wiring in `Server/src/SOR.Host`
|
|
- Push and event dependencies through `IClientPush` and `IEventBus`
|
|
|
|
Reference: https://12factor.net/
|
|
|
|
## Executive Summary
|
|
|
|
`SOR.Matchmaking` is a service module registered by `SOR.Host` when `ENABLED_SERVICES` includes `matchmaking`. It exposes handlers for joining, leaving, and checking matchmaking queue status, and it starts `MatchmakerService`, a polling background worker that pairs players by rank range, creates `GameSession` rows, publishes `GameStartedEvent`, and pushes `matchmaking.match_found` notifications to both players.
|
|
|
|
The module already has useful foundations:
|
|
|
|
- Queue state is durable in the `game.matchmaking_entries` table.
|
|
- Queue join is idempotent in the common case where an active searching entry already exists.
|
|
- Match creation snapshots ranked rules and board template data into the game session.
|
|
- Match found delivery uses existing `IClientPush`.
|
|
- Game lifecycle notification uses existing `IEventBus`.
|
|
- The worker is cancellation-aware and logs tick failures.
|
|
|
|
The main 12-Factor gaps are around process formation, config, and concurrency:
|
|
|
|
- Request handlers and the matchmaking worker are always registered together.
|
|
- Poll interval, rank range, range expansion, max range, and expansion cadence are hardcoded constants.
|
|
- Multiple matchmaking worker processes can load and match the same queue entries unless there is only one instance operationally.
|
|
- Active queue uniqueness is application-level only; concurrent joins can create duplicate searching entries.
|
|
- Match creation, queue-entry updates, event publish, and push delivery are not modeled with an outbox/idempotent delivery boundary.
|
|
- Join does not visibly validate that the selected deck exists, belongs to the account, and is valid for ranked play.
|
|
- Queue cleanup, stale entry expiration, abandoned initializing sessions, and retry/reconciliation jobs are not explicit.
|
|
- Readiness is host-level only and does not account for Game DB, Identity DB, push, or event bus dependencies.
|
|
|
|
The plan is to keep `SOR.Matchmaking` as the ranked queue module, but split request handlers from worker registration, add typed config, harden queue invariants with database/lease semantics, introduce a transaction/outbox boundary for match creation, and add observability and operational jobs.
|
|
|
|
## Current Structure Observations
|
|
|
|
### Strong Foundations
|
|
|
|
- `AddMatchmakingService` registers `GameDbContext`, `SorIdentityDbContext`, all handlers, and `MatchmakerService`.
|
|
- `MatchmakingJoinHandler` returns the existing searching entry when found.
|
|
- `MatchmakingLeaveHandler` marks searching entries cancelled instead of deleting them.
|
|
- `MatchmakerService` orders queue entries by wait time.
|
|
- Rank range expands over time to reduce indefinite waits.
|
|
- Game sessions snapshot ranked rules, board dimensions, turn timer, rewards, and board template JSON.
|
|
- `GameStartedEvent` is published after game session creation.
|
|
- Both matched players receive a push payload with session ID and opponent info.
|
|
|
|
### Main Problems
|
|
|
|
- Matchmaking algorithm parameters are hardcoded in `MatchmakerService`.
|
|
- `MatchmakerService` starts whenever the matchmaking module is enabled, which couples API handlers and worker role.
|
|
- The worker loads all searching entries into memory each tick.
|
|
- There is no distributed lock, row claim, or queue-group style ownership for matching work.
|
|
- There is no database constraint guaranteeing one active searching entry per account.
|
|
- `JoinQueueRequest` accepts a deck ID but does not visibly validate deck ownership or deck validity.
|
|
- Publishing `GameStartedEvent` can fail after the game session and queue entries are committed.
|
|
- Push delivery can fail after the match is committed.
|
|
- There is no durable delivery state for match-found notifications.
|
|
- Stale searching entries and old cancelled/matched entries do not appear to have retention cleanup.
|
|
- No visible tests cover concurrent joins or multiple workers.
|
|
|
|
## Target Matchmaking Shape
|
|
|
|
Keep `SOR.Matchmaking` focused on ranked queue orchestration, but split responsibilities:
|
|
|
|
```text
|
|
Server/src/SOR.Matchmaking/
|
|
Configuration/
|
|
MatchmakingOptions.cs
|
|
MatchmakingWorkerOptions.cs
|
|
MatchmakingRankOptions.cs
|
|
MatchmakingOptionsValidator.cs
|
|
Bootstrap/
|
|
MatchmakingRegistration.cs
|
|
MatchmakingHandlersRegistration.cs
|
|
MatchmakingWorkerRegistration.cs
|
|
MatchmakingHealthChecks.cs
|
|
Contracts/
|
|
MatchmakingMessages.cs
|
|
MatchmakingErrors.cs
|
|
Application/
|
|
Queue/
|
|
MatchmakingQueueService.cs
|
|
QueueEntryValidator.cs
|
|
QueueStatusService.cs
|
|
Matching/
|
|
MatchmakerWorker.cs
|
|
MatchCandidateSelector.cs
|
|
MatchCreationService.cs
|
|
RankedRuleSnapshotBuilder.cs
|
|
Delivery/
|
|
MatchFoundNotificationService.cs
|
|
MatchmakingOutboxService.cs
|
|
Persistence/
|
|
IMatchmakingRepository.cs
|
|
IDeckEligibilityReader.cs
|
|
IRankReader.cs
|
|
IMatchmakingLeaseStore.cs
|
|
IMatchmakingUnitOfWork.cs
|
|
Jobs/
|
|
ExpireStaleQueueEntriesJob.cs
|
|
ReconcileUndeliveredMatchesJob.cs
|
|
CleanupOldMatchmakingEntriesJob.cs
|
|
```
|
|
|
|
Desired process roles:
|
|
|
|
- `matchmaking-api`: handles join, leave, and status requests.
|
|
- `matchmaking-worker`: runs matching ticks and owns match creation.
|
|
- `matchmaking-jobs`: runs cleanup and reconciliation jobs.
|
|
|
|
Small deployments can co-locate these roles, but registration should make the role boundary explicit.
|
|
|
|
## 12-Factor Gap Map
|
|
|
|
### 1. Codebase
|
|
|
|
Mostly aligned. Matchmaking is one small module, but handler and worker concerns are co-registered.
|
|
|
|
Plan:
|
|
- Keep one module.
|
|
- Split API handlers, worker, and maintenance jobs into explicit registration methods.
|
|
- Extract matching algorithm and match creation into testable services.
|
|
|
|
### 2. Dependencies
|
|
|
|
Mostly aligned. Dependencies are explicit: Game DB, Identity DB, event bus, and client push.
|
|
|
|
Plan:
|
|
- Add abstractions for rank reads, deck eligibility, queue repository, match creation, and notification delivery.
|
|
- Keep `IEventBus` and `IClientPush` behind a match delivery/outbox service.
|
|
- Avoid direct Identity queries in the worker where a rank reader service is sufficient.
|
|
|
|
### 3. Config
|
|
|
|
Weakly aligned. All matchmaking behavior is hardcoded.
|
|
|
|
Plan:
|
|
- Add typed `MatchmakingOptions`.
|
|
- Move poll interval, batch size, rank range, range expansion, max range, stale queue timeout, and worker enablement into config.
|
|
- Validate options at startup.
|
|
|
|
Candidate config shape:
|
|
|
|
```json
|
|
{
|
|
"Matchmaking": {
|
|
"ApiEnabled": true,
|
|
"WorkerEnabled": true,
|
|
"PollIntervalMs": 5000,
|
|
"MaxEntriesPerTick": 200,
|
|
"InitialRankRange": 100,
|
|
"RangeExpandStep": 100,
|
|
"MaxRankRange": 500,
|
|
"ExpandEverySeconds": 30,
|
|
"StaleQueueEntryMinutes": 15,
|
|
"RequireValidDeck": true
|
|
}
|
|
}
|
|
```
|
|
|
|
### 4. Backing Services
|
|
|
|
Partially aligned. Matchmaking uses PostgreSQL, NATS, Redis/client push, and Identity rank data, but readiness does not reflect those dependencies.
|
|
|
|
Plan:
|
|
- Treat Game DB, Identity DB, event bus, and push delivery as attached resources.
|
|
- Add role-specific readiness checks.
|
|
- Define degraded mode: API may accept queue operations only when worker/delivery dependencies are healthy, or explicitly reject joins if worker is unavailable.
|
|
|
|
### 5. Build, Release, Run
|
|
|
|
Mostly aligned. There are no build-time assets and no migrations run here.
|
|
|
|
Plan:
|
|
- Keep schema migrations outside matchmaking startup.
|
|
- Include deployment checks for worker replica count and queue ownership mode.
|
|
- Version algorithm changes through config/release notes because matching behavior is player-visible.
|
|
|
|
### 6. Processes
|
|
|
|
Partially aligned. API request handling and worker polling are one process role today.
|
|
|
|
Plan:
|
|
- Split registration into:
|
|
- `AddMatchmakingCore`
|
|
- `AddMatchmakingHandlers`
|
|
- `AddMatchmakingWorker`
|
|
- `AddMatchmakingJobs`
|
|
- Let `SOR.Host` decide which roles start.
|
|
- Validate unsafe combinations, such as multiple workers without leases.
|
|
|
|
### 7. Port Binding
|
|
|
|
Aligned through `SOR.Host`. Matchmaking does not bind ports directly.
|
|
|
|
Plan:
|
|
- Keep port binding in Host/Gateway.
|
|
- Expose matchmaking readiness through host health checks.
|
|
|
|
### 8. Concurrency
|
|
|
|
Weakly aligned. This is the main risk area.
|
|
|
|
Plan:
|
|
- Add a database-backed active queue uniqueness rule for one searching entry per account.
|
|
- Add row claiming with `FOR UPDATE SKIP LOCKED`, optimistic status transitions, or a worker lease model.
|
|
- Ensure game session creation and queue entry resolution happen in one transaction.
|
|
- Make match creation idempotent per entry pair.
|
|
- Add tests for two workers attempting to match the same entries.
|
|
|
|
### 9. Disposability
|
|
|
|
Partially aligned. The worker accepts cancellation, but tick duration and shutdown boundaries are not explicit.
|
|
|
|
Plan:
|
|
- Make ticks bounded by max entries and command timeout.
|
|
- Stop starting new match attempts on shutdown.
|
|
- Complete or rollback the current transaction before exit.
|
|
- Move push/event retry to outbox/reconciliation so shutdown does not lose delivery.
|
|
|
|
### 10. Dev/Prod Parity
|
|
|
|
Partially aligned. Same code runs locally and in production, but worker count and timing behavior are not config-driven.
|
|
|
|
Plan:
|
|
- Use the same worker code locally and in production.
|
|
- Configure local shorter poll intervals if useful.
|
|
- Document single-worker versus multi-worker deployment behavior.
|
|
- Add a local test profile for queue scenarios.
|
|
|
|
### 11. Logs
|
|
|
|
Partially aligned. Worker starts/stops, tick errors, and found matches are logged.
|
|
|
|
Plan:
|
|
- Add structured logs and metrics for joins, leaves, matches, candidate search, stale entries, event publish, push delivery, and worker claim conflicts.
|
|
- Avoid logging sensitive deck contents.
|
|
|
|
Suggested fields:
|
|
|
|
- `entryId`
|
|
- `accountId`
|
|
- `opponentAccountId`
|
|
- `gameSessionId`
|
|
- `rankScore`
|
|
- `opponentRankScore`
|
|
- `rankRange`
|
|
- `waitSeconds`
|
|
- `workerId`
|
|
- `deliveryState`
|
|
|
|
### 12. Admin Processes
|
|
|
|
Partially aligned. Cleanup/reconciliation are operational jobs but not explicit.
|
|
|
|
Plan:
|
|
- Add admin/job processes for stale queue expiration, delivery reconciliation, old entry cleanup, and queue health reports.
|
|
- Add dry-run reports for cleanup.
|
|
- Add Admin visibility for current queue depth, age, rank distribution, and worker status.
|
|
|
|
## Refactor Phases
|
|
|
|
### Phase 0: Baseline Tests
|
|
|
|
Goals:
|
|
- Preserve current behavior before restructuring.
|
|
|
|
Work:
|
|
- Add tests for join idempotency, leave idempotency, and status.
|
|
- Add tests for rank range expansion.
|
|
- Add tests for closest-opponent selection.
|
|
- Add tests for session rule snapshot generation.
|
|
- Add tests for event/push failure behavior as it works today.
|
|
|
|
Done when:
|
|
- Current behavior is covered before refactors.
|
|
|
|
### Phase 1: Typed Options
|
|
|
|
Goals:
|
|
- Make matchmaking behavior configurable.
|
|
|
|
Work:
|
|
- Add `MatchmakingOptions`.
|
|
- Replace hardcoded poll/range constants with options.
|
|
- Add startup validation.
|
|
- Preserve current defaults.
|
|
|
|
Done when:
|
|
- Operators can tune queue behavior per environment without code changes.
|
|
|
|
### Phase 2: Split API and Worker Registration
|
|
|
|
Goals:
|
|
- Make process formation explicit.
|
|
|
|
Work:
|
|
- Split `AddMatchmakingService` into core, handlers, worker, and jobs.
|
|
- Keep current host behavior initially by calling both handlers and worker registration.
|
|
- Add host-level role support later if desired.
|
|
|
|
Done when:
|
|
- A deployment can run API-only, worker-only, or combined matchmaking roles intentionally.
|
|
|
|
### Phase 3: Extract Queue and Matching Services
|
|
|
|
Goals:
|
|
- Make matching logic testable and smaller.
|
|
|
|
Work:
|
|
- Extract `MatchmakingQueueService` for join/leave/status.
|
|
- Extract `MatchCandidateSelector` for rank-window selection.
|
|
- Extract `MatchCreationService` for session creation and queue resolution.
|
|
- Extract `RankedRuleSnapshotBuilder`.
|
|
|
|
Done when:
|
|
- Handlers and worker are orchestration shells over tested services.
|
|
|
|
### Phase 4: Deck Eligibility Validation
|
|
|
|
Goals:
|
|
- Prevent invalid ranked queue entries.
|
|
|
|
Work:
|
|
- Add `IDeckEligibilityReader`.
|
|
- Validate deck ownership, deck validity, and ranked eligibility on join.
|
|
- Decide whether already queued entries should be revalidated before match creation.
|
|
|
|
Done when:
|
|
- Players cannot queue ranked matchmaking with invalid or unowned decks.
|
|
|
|
### Phase 5: Concurrency Hardening
|
|
|
|
Goals:
|
|
- Make matchmaking safe under concurrent joins and multiple workers.
|
|
|
|
Work:
|
|
- Add database uniqueness or conditional insert for one searching entry per account.
|
|
- Add queue row claiming or worker leases.
|
|
- Use transactions around claiming, session creation, and entry resolution.
|
|
- Add multi-worker concurrency tests.
|
|
|
|
Done when:
|
|
- Multiple API instances and worker instances cannot duplicate queue entries or matches.
|
|
|
|
### Phase 6: Outbox and Delivery Reconciliation
|
|
|
|
Goals:
|
|
- Avoid losing game-started events or match-found pushes after DB commit.
|
|
|
|
Work:
|
|
- Add an outbox record for `GameStartedEvent` and match-found notifications.
|
|
- Publish/push from outbox with retry.
|
|
- Add reconciliation job for undelivered match notifications.
|
|
- Make push delivery idempotent from the client perspective.
|
|
|
|
Done when:
|
|
- A committed match cannot silently miss lifecycle event or player notification delivery.
|
|
|
|
### Phase 7: Queue Maintenance Jobs
|
|
|
|
Goals:
|
|
- Keep queue data healthy.
|
|
|
|
Work:
|
|
- Add stale searching entry expiration.
|
|
- Add cleanup for old cancelled/matched entries.
|
|
- Add abandoned initializing session detection handoff or integration with game inactivity logic.
|
|
- Add admin queue reports.
|
|
|
|
Done when:
|
|
- Queue state does not grow unbounded or remain stuck indefinitely.
|
|
|
|
### Phase 8: Readiness and Observability
|
|
|
|
Goals:
|
|
- Make matchmaking operations diagnosable.
|
|
|
|
Work:
|
|
- Add readiness checks for Game DB, Identity DB, event bus, push, and worker lease.
|
|
- Add metrics for queue depth, oldest wait time, matches per minute, average rank delta, tick latency, delivery failures, and claim conflicts.
|
|
- Add runbooks for stuck queues, delivery failures, and worker scaling.
|
|
|
|
Done when:
|
|
- Operators can see whether players can join, be matched, and receive match notifications.
|
|
|
|
## Suggested Work Order
|
|
|
|
1. Add baseline tests for join/leave/status and candidate selection.
|
|
2. Add typed `MatchmakingOptions` with current defaults.
|
|
3. Split handler and worker registration.
|
|
4. Extract queue, selector, snapshot, and match creation services.
|
|
5. Add deck eligibility validation.
|
|
6. Add active-entry uniqueness and worker row claiming.
|
|
7. Add outbox/reconciliation for event and push delivery.
|
|
8. Add queue cleanup jobs.
|
|
9. Add readiness checks, metrics, dashboards, and runbooks.
|
|
|
|
## Completion Criteria
|
|
|
|
The refactor is complete when:
|
|
|
|
- Matchmaking API and worker roles are registered explicitly.
|
|
- Matchmaking behavior is controlled by typed config.
|
|
- Queue joins are concurrency-safe and idempotent.
|
|
- Multiple workers are either prevented or coordinated safely.
|
|
- Match creation is transactional and idempotent.
|
|
- Game-started events and match-found pushes are delivered through a retryable outbox.
|
|
- Deck eligibility is validated before ranked matchmaking.
|
|
- Stale queue and undelivered match states have maintenance jobs.
|
|
- Readiness and metrics show queue health, worker health, and delivery health.
|
|
|
|
## Immediate Low-Risk Next Steps
|
|
|
|
- Add `MatchmakingOptions` with defaults matching current constants.
|
|
- Extract `MatchCandidateSelector` and cover rank-window behavior with tests.
|
|
- Split `AddMatchmakingService` internally into handler and worker registration methods without changing host behavior.
|
|
- Add a database/model plan for one active searching entry per account.
|
|
- Document the current single-worker assumption until row claiming or leases are implemented.
|