Files
sor/SOR.Matchmaking.12FactorPlan.md
2026-07-02 13:09:05 -04:00

426 lines
16 KiB
Markdown

# SOR.Matchmaking 12-Factor Refactor Plan
Date: 2026-06-10
Scope reviewed:
- `Server/src/SOR.Matchmaking`
- Matchmaking persistence in `Server/src/SOR.Database/Contexts/GameDbContext.cs`
- Identity rank reads in `Server/src/SOR.Database/Contexts/SorIdentityDbContext.cs`
- Matchmaking wiring in `Server/src/SOR.Host`
- Push and event dependencies through `IClientPush` and `IEventBus`
Reference: https://12factor.net/
## Executive Summary
`SOR.Matchmaking` is a service module registered by `SOR.Host` when `ENABLED_SERVICES` includes `matchmaking`. It exposes handlers for joining, leaving, and checking matchmaking queue status, and it starts `MatchmakerService`, a polling background worker that pairs players by rank range, creates `GameSession` rows, publishes `GameStartedEvent`, and pushes `matchmaking.match_found` notifications to both players.
The module already has useful foundations:
- Queue state is durable in the `game.matchmaking_entries` table.
- Queue join is idempotent in the common case where an active searching entry already exists.
- Match creation snapshots ranked rules and board template data into the game session.
- Match found delivery uses existing `IClientPush`.
- Game lifecycle notification uses existing `IEventBus`.
- The worker is cancellation-aware and logs tick failures.
The main 12-Factor gaps are around process formation, config, and concurrency:
- Request handlers and the matchmaking worker are always registered together.
- Poll interval, rank range, range expansion, max range, and expansion cadence are hardcoded constants.
- Multiple matchmaking worker processes can load and match the same queue entries unless there is only one instance operationally.
- Active queue uniqueness is application-level only; concurrent joins can create duplicate searching entries.
- Match creation, queue-entry updates, event publish, and push delivery are not modeled with an outbox/idempotent delivery boundary.
- Join does not visibly validate that the selected deck exists, belongs to the account, and is valid for ranked play.
- Queue cleanup, stale entry expiration, abandoned initializing sessions, and retry/reconciliation jobs are not explicit.
- Readiness is host-level only and does not account for Game DB, Identity DB, push, or event bus dependencies.
The plan is to keep `SOR.Matchmaking` as the ranked queue module, but split request handlers from worker registration, add typed config, harden queue invariants with database/lease semantics, introduce a transaction/outbox boundary for match creation, and add observability and operational jobs.
## Current Structure Observations
### Strong Foundations
- `AddMatchmakingService` registers `GameDbContext`, `SorIdentityDbContext`, all handlers, and `MatchmakerService`.
- `MatchmakingJoinHandler` returns the existing searching entry when found.
- `MatchmakingLeaveHandler` marks searching entries cancelled instead of deleting them.
- `MatchmakerService` orders queue entries by wait time.
- Rank range expands over time to reduce indefinite waits.
- Game sessions snapshot ranked rules, board dimensions, turn timer, rewards, and board template JSON.
- `GameStartedEvent` is published after game session creation.
- Both matched players receive a push payload with session ID and opponent info.
### Main Problems
- Matchmaking algorithm parameters are hardcoded in `MatchmakerService`.
- `MatchmakerService` starts whenever the matchmaking module is enabled, which couples API handlers and worker role.
- The worker loads all searching entries into memory each tick.
- There is no distributed lock, row claim, or queue-group style ownership for matching work.
- There is no database constraint guaranteeing one active searching entry per account.
- `JoinQueueRequest` accepts a deck ID but does not visibly validate deck ownership or deck validity.
- Publishing `GameStartedEvent` can fail after the game session and queue entries are committed.
- Push delivery can fail after the match is committed.
- There is no durable delivery state for match-found notifications.
- Stale searching entries and old cancelled/matched entries do not appear to have retention cleanup.
- No visible tests cover concurrent joins or multiple workers.
## Target Matchmaking Shape
Keep `SOR.Matchmaking` focused on ranked queue orchestration, but split responsibilities:
```text
Server/src/SOR.Matchmaking/
Configuration/
MatchmakingOptions.cs
MatchmakingWorkerOptions.cs
MatchmakingRankOptions.cs
MatchmakingOptionsValidator.cs
Bootstrap/
MatchmakingRegistration.cs
MatchmakingHandlersRegistration.cs
MatchmakingWorkerRegistration.cs
MatchmakingHealthChecks.cs
Contracts/
MatchmakingMessages.cs
MatchmakingErrors.cs
Application/
Queue/
MatchmakingQueueService.cs
QueueEntryValidator.cs
QueueStatusService.cs
Matching/
MatchmakerWorker.cs
MatchCandidateSelector.cs
MatchCreationService.cs
RankedRuleSnapshotBuilder.cs
Delivery/
MatchFoundNotificationService.cs
MatchmakingOutboxService.cs
Persistence/
IMatchmakingRepository.cs
IDeckEligibilityReader.cs
IRankReader.cs
IMatchmakingLeaseStore.cs
IMatchmakingUnitOfWork.cs
Jobs/
ExpireStaleQueueEntriesJob.cs
ReconcileUndeliveredMatchesJob.cs
CleanupOldMatchmakingEntriesJob.cs
```
Desired process roles:
- `matchmaking-api`: handles join, leave, and status requests.
- `matchmaking-worker`: runs matching ticks and owns match creation.
- `matchmaking-jobs`: runs cleanup and reconciliation jobs.
Small deployments can co-locate these roles, but registration should make the role boundary explicit.
## 12-Factor Gap Map
### 1. Codebase
Mostly aligned. Matchmaking is one small module, but handler and worker concerns are co-registered.
Plan:
- Keep one module.
- Split API handlers, worker, and maintenance jobs into explicit registration methods.
- Extract matching algorithm and match creation into testable services.
### 2. Dependencies
Mostly aligned. Dependencies are explicit: Game DB, Identity DB, event bus, and client push.
Plan:
- Add abstractions for rank reads, deck eligibility, queue repository, match creation, and notification delivery.
- Keep `IEventBus` and `IClientPush` behind a match delivery/outbox service.
- Avoid direct Identity queries in the worker where a rank reader service is sufficient.
### 3. Config
Weakly aligned. All matchmaking behavior is hardcoded.
Plan:
- Add typed `MatchmakingOptions`.
- Move poll interval, batch size, rank range, range expansion, max range, stale queue timeout, and worker enablement into config.
- Validate options at startup.
Candidate config shape:
```json
{
"Matchmaking": {
"ApiEnabled": true,
"WorkerEnabled": true,
"PollIntervalMs": 5000,
"MaxEntriesPerTick": 200,
"InitialRankRange": 100,
"RangeExpandStep": 100,
"MaxRankRange": 500,
"ExpandEverySeconds": 30,
"StaleQueueEntryMinutes": 15,
"RequireValidDeck": true
}
}
```
### 4. Backing Services
Partially aligned. Matchmaking uses PostgreSQL, NATS, Redis/client push, and Identity rank data, but readiness does not reflect those dependencies.
Plan:
- Treat Game DB, Identity DB, event bus, and push delivery as attached resources.
- Add role-specific readiness checks.
- Define degraded mode: API may accept queue operations only when worker/delivery dependencies are healthy, or explicitly reject joins if worker is unavailable.
### 5. Build, Release, Run
Mostly aligned. There are no build-time assets and no migrations run here.
Plan:
- Keep schema migrations outside matchmaking startup.
- Include deployment checks for worker replica count and queue ownership mode.
- Version algorithm changes through config/release notes because matching behavior is player-visible.
### 6. Processes
Partially aligned. API request handling and worker polling are one process role today.
Plan:
- Split registration into:
- `AddMatchmakingCore`
- `AddMatchmakingHandlers`
- `AddMatchmakingWorker`
- `AddMatchmakingJobs`
- Let `SOR.Host` decide which roles start.
- Validate unsafe combinations, such as multiple workers without leases.
### 7. Port Binding
Aligned through `SOR.Host`. Matchmaking does not bind ports directly.
Plan:
- Keep port binding in Host/Gateway.
- Expose matchmaking readiness through host health checks.
### 8. Concurrency
Weakly aligned. This is the main risk area.
Plan:
- Add a database-backed active queue uniqueness rule for one searching entry per account.
- Add row claiming with `FOR UPDATE SKIP LOCKED`, optimistic status transitions, or a worker lease model.
- Ensure game session creation and queue entry resolution happen in one transaction.
- Make match creation idempotent per entry pair.
- Add tests for two workers attempting to match the same entries.
### 9. Disposability
Partially aligned. The worker accepts cancellation, but tick duration and shutdown boundaries are not explicit.
Plan:
- Make ticks bounded by max entries and command timeout.
- Stop starting new match attempts on shutdown.
- Complete or rollback the current transaction before exit.
- Move push/event retry to outbox/reconciliation so shutdown does not lose delivery.
### 10. Dev/Prod Parity
Partially aligned. Same code runs locally and in production, but worker count and timing behavior are not config-driven.
Plan:
- Use the same worker code locally and in production.
- Configure local shorter poll intervals if useful.
- Document single-worker versus multi-worker deployment behavior.
- Add a local test profile for queue scenarios.
### 11. Logs
Partially aligned. Worker starts/stops, tick errors, and found matches are logged.
Plan:
- Add structured logs and metrics for joins, leaves, matches, candidate search, stale entries, event publish, push delivery, and worker claim conflicts.
- Avoid logging sensitive deck contents.
Suggested fields:
- `entryId`
- `accountId`
- `opponentAccountId`
- `gameSessionId`
- `rankScore`
- `opponentRankScore`
- `rankRange`
- `waitSeconds`
- `workerId`
- `deliveryState`
### 12. Admin Processes
Partially aligned. Cleanup/reconciliation are operational jobs but not explicit.
Plan:
- Add admin/job processes for stale queue expiration, delivery reconciliation, old entry cleanup, and queue health reports.
- Add dry-run reports for cleanup.
- Add Admin visibility for current queue depth, age, rank distribution, and worker status.
## Refactor Phases
### Phase 0: Baseline Tests
Goals:
- Preserve current behavior before restructuring.
Work:
- Add tests for join idempotency, leave idempotency, and status.
- Add tests for rank range expansion.
- Add tests for closest-opponent selection.
- Add tests for session rule snapshot generation.
- Add tests for event/push failure behavior as it works today.
Done when:
- Current behavior is covered before refactors.
### Phase 1: Typed Options
Goals:
- Make matchmaking behavior configurable.
Work:
- Add `MatchmakingOptions`.
- Replace hardcoded poll/range constants with options.
- Add startup validation.
- Preserve current defaults.
Done when:
- Operators can tune queue behavior per environment without code changes.
### Phase 2: Split API and Worker Registration
Goals:
- Make process formation explicit.
Work:
- Split `AddMatchmakingService` into core, handlers, worker, and jobs.
- Keep current host behavior initially by calling both handlers and worker registration.
- Add host-level role support later if desired.
Done when:
- A deployment can run API-only, worker-only, or combined matchmaking roles intentionally.
### Phase 3: Extract Queue and Matching Services
Goals:
- Make matching logic testable and smaller.
Work:
- Extract `MatchmakingQueueService` for join/leave/status.
- Extract `MatchCandidateSelector` for rank-window selection.
- Extract `MatchCreationService` for session creation and queue resolution.
- Extract `RankedRuleSnapshotBuilder`.
Done when:
- Handlers and worker are orchestration shells over tested services.
### Phase 4: Deck Eligibility Validation
Goals:
- Prevent invalid ranked queue entries.
Work:
- Add `IDeckEligibilityReader`.
- Validate deck ownership, deck validity, and ranked eligibility on join.
- Decide whether already queued entries should be revalidated before match creation.
Done when:
- Players cannot queue ranked matchmaking with invalid or unowned decks.
### Phase 5: Concurrency Hardening
Goals:
- Make matchmaking safe under concurrent joins and multiple workers.
Work:
- Add database uniqueness or conditional insert for one searching entry per account.
- Add queue row claiming or worker leases.
- Use transactions around claiming, session creation, and entry resolution.
- Add multi-worker concurrency tests.
Done when:
- Multiple API instances and worker instances cannot duplicate queue entries or matches.
### Phase 6: Outbox and Delivery Reconciliation
Goals:
- Avoid losing game-started events or match-found pushes after DB commit.
Work:
- Add an outbox record for `GameStartedEvent` and match-found notifications.
- Publish/push from outbox with retry.
- Add reconciliation job for undelivered match notifications.
- Make push delivery idempotent from the client perspective.
Done when:
- A committed match cannot silently miss lifecycle event or player notification delivery.
### Phase 7: Queue Maintenance Jobs
Goals:
- Keep queue data healthy.
Work:
- Add stale searching entry expiration.
- Add cleanup for old cancelled/matched entries.
- Add abandoned initializing session detection handoff or integration with game inactivity logic.
- Add admin queue reports.
Done when:
- Queue state does not grow unbounded or remain stuck indefinitely.
### Phase 8: Readiness and Observability
Goals:
- Make matchmaking operations diagnosable.
Work:
- Add readiness checks for Game DB, Identity DB, event bus, push, and worker lease.
- Add metrics for queue depth, oldest wait time, matches per minute, average rank delta, tick latency, delivery failures, and claim conflicts.
- Add runbooks for stuck queues, delivery failures, and worker scaling.
Done when:
- Operators can see whether players can join, be matched, and receive match notifications.
## Suggested Work Order
1. Add baseline tests for join/leave/status and candidate selection.
2. Add typed `MatchmakingOptions` with current defaults.
3. Split handler and worker registration.
4. Extract queue, selector, snapshot, and match creation services.
5. Add deck eligibility validation.
6. Add active-entry uniqueness and worker row claiming.
7. Add outbox/reconciliation for event and push delivery.
8. Add queue cleanup jobs.
9. Add readiness checks, metrics, dashboards, and runbooks.
## Completion Criteria
The refactor is complete when:
- Matchmaking API and worker roles are registered explicitly.
- Matchmaking behavior is controlled by typed config.
- Queue joins are concurrency-safe and idempotent.
- Multiple workers are either prevented or coordinated safely.
- Match creation is transactional and idempotent.
- Game-started events and match-found pushes are delivered through a retryable outbox.
- Deck eligibility is validated before ranked matchmaking.
- Stale queue and undelivered match states have maintenance jobs.
- Readiness and metrics show queue health, worker health, and delivery health.
## Immediate Low-Risk Next Steps
- Add `MatchmakingOptions` with defaults matching current constants.
- Extract `MatchCandidateSelector` and cover rank-window behavior with tests.
- Split `AddMatchmakingService` internally into handler and worker registration methods without changing host behavior.
- Add a database/model plan for one active searching entry per account.
- Document the current single-worker assumption until row claiming or leases are implemented.