Files
sor/SOR.Ai.12FactorPlan.md
2026-07-02 13:09:05 -04:00

21 KiB

SOR.Ai 12-Factor Refactor Plan

Date: 2026-06-10

Scope reviewed:

  • Server/src/SOR.Ai
  • Server/docs/SOR.Ai-Architecture-Plan.md
  • AI wiring in Server/src/SOR.Host
  • AI touchpoints in SOR.Game, SOR.Database, SOR.Networking.Abstractions, and SOR.Shared

Reference: https://12factor.net/

Executive Summary

SOR.Ai is a service module, not its own executable. It is registered by SOR.Host when the game service is enabled, while AiPlayerSessionManager is only started when ENABLED_SERVICES includes ai. That gives the project a useful split between AI message handlers and the dedicated AI worker loop, but the split is currently implicit and easy to misconfigure.

The project already has strong foundations:

  • AI players are modeled as normal accounts and participate through the same game message handler path as human clients.
  • Decision logic is separated into random, heuristic, MCTS, and neural engines.
  • The runtime worker uses a hosted service and cancellation tokens.
  • Neural inference is optional and falls back when no ONNX model is configured.
  • Training, replay export, calibration, lobby AI slots, and admin match services are separate namespaces.

The main 12-Factor gaps are structural and operational:

  • One DI module registers runtime worker infrastructure, lobby handlers, startup seeding, neural loading, admin-only services, self-play, calibration, and database contexts together.
  • AI account and deck seeding runs during normal application startup.
  • AiPlayerSessionManager keeps active sessions in process memory and assumes only one AI worker subscribes to game events.
  • Turn execution starts fire-and-forget tasks and does not centrally track all in-flight session work for shutdown.
  • Worker delays, polling intervals, difficulty profiles, strategy definitions, parallelism, and model contract details are mostly code constants instead of validated config.
  • Neural model loading is path-based and static-accessible through NeuralModelRegistry.Instance, making model artifacts a filesystem concern rather than an attached release resource.
  • Training and calibration jobs run inside the same module as runtime AI and need clearer process boundaries.

The plan is to keep SOR.Ai as the AI domain module, but split registration into explicit roles: game-facing AI handlers, AI runtime worker, admin AI services, and offline training/calibration jobs.

Current Structure Observations

Strong Foundations

  • SOR.Ai depends on SOR.Networking.Abstractions and invokes existing game handlers through IMessageHandler.
  • AiNoOpClientConnection lets AI use the normal authenticated game pipeline without a real socket.
  • AiAccountRegistry exposes IAiAccountRegistry so SOR.Game can detect AI accounts without referencing the AI project directly.
  • AiPlayerSessionManager is intentionally registered as a singleton but only hosted by the dedicated ai process role.
  • AiPlayerSession polls game state and drives placement, draw, and main-phase decisions.
  • Decision engines are grouped by behavior and difficulty: random, heuristic, Monte Carlo, and neural.
  • Neural model startup is tolerant of a missing model and does not hard-fail lower environments.
  • Training artifacts are represented through TrainingRecord, TrainingDataWriter, replay export, self-play, and calibration services.

Main Problems

  • AddAiService has too many responsibilities for a single registration method.
  • Calling AddAiService from game means AI startup seed and neural loader run on every game-enabled host, even when the host is not the AI worker.
  • The runtime role boundary depends on comments and ENABLED_SERVICES discipline rather than typed options and startup validation.
  • AiPlayerSessionManager stores active sessions in a ConcurrentDictionary<Guid, List<AiPlayerSession>>, which cannot survive process restart and cannot coordinate multiple AI workers.
  • NATS subscriptions are documented as no queue group, so multiple AI worker subscribers would all receive the same event.
  • AI session tasks are started with Task.Run and discarded.
  • ELO recording is also fire-and-forget.
  • Turn execution directly resolves handlers from scoped DI and creates concrete decision engines with new, which makes runtime orchestration harder to test and observe.
  • The neural model is loaded from Ai:NeuralModelPath and exposed through a static NeuralModelRegistry.Instance.
  • Difficulty profiles, think times, mistake rates, strategies, model input/output names, self-play parallelism, and polling behavior are not externally validated runtime config.
  • Seeding AI accounts/decks during app startup mixes release/admin work into the long-running web or worker process.
  • Training and calibration can be triggered from admin services, but they are not clearly isolated as one-off admin processes.

Target AI Shape

Keep SOR.Ai as the AI module, but organize it around explicit responsibilities:

Server/src/SOR.Ai/
  Configuration/
    AiOptions.cs
    AiRuntimeOptions.cs
    AiNeuralOptions.cs
    AiTrainingOptions.cs
    AiSeedingOptions.cs
    AiOptionsValidator.cs
  Bootstrap/
    AiCoreRegistration.cs
    AiHandlerRegistration.cs
    AiRuntimeRegistration.cs
    AiAdminRegistration.cs
    AiTrainingRegistration.cs
    AiHealthChecks.cs
  Runtime/
    AiPlayerSessionManager.cs
    AiSessionCoordinator.cs
    AiSessionLeaseStore.cs
    AiTurnExecutor.cs
    AiTurnDispatcher.cs
    AiRuntimeMetrics.cs
  Decisions/
    IAiDecisionEngineFactory.cs
    AiDecisionEngineFactory.cs
    Random/
    Heuristic/
    Mcts/
    Neural/
  Lobby/
    AiLobbyService.cs
    AiSlotAssigner.cs
    AiDeckResolver.cs
  Persistence/
    IAiAccountRepository.cs
    IAiDeckRepository.cs
    IAiEloRepository.cs
    IAiTrainingDataStore.cs
  Models/
    IAiModelStore.cs
    NeuralModelRegistry.cs
    NeuralModelVersion.cs
  Operations/
    SeedAiAccountsCommand.cs
    SeedAiDecksCommand.cs
    RunAiCalibrationCommand.cs
    ExportAiTrainingDataCommand.cs
  Admin/
    AiMatchService.cs

Desired process roles:

  • game: registers game handlers plus AI lobby handlers needed by player/admin messages.
  • ai-worker: starts the AI session runtime and owns turn execution.
  • admin: uses AI admin services but does not start AI runtime loops.
  • ai-jobs: runs seeding, calibration, replay export, self-play, and model-management jobs as one-off processes.

12-Factor Gap Map

1. Codebase

Partially aligned. SOR.Ai is one codebase module, but it currently mixes online runtime AI, message handlers, admin services, training, seeding, calibration, and neural model management.

Plan:

  • Keep one module, but split registration and namespaces by role.
  • Document which services are valid in each process type.
  • Add tests around role registration so game, ai-worker, admin, and ai-jobs do not accidentally start each other's work.

2. Dependencies

Partially aligned. Dependencies are explicit in SOR.Ai.csproj, including Microsoft.ML.OnnxRuntime, SOR.Game, database projects, shared abstractions, and ASP.NET Core hosting types.

Plan:

  • Keep ONNX as an explicit package dependency.
  • Introduce abstractions for model storage, session coordination, ELO persistence, and game action invocation.
  • Avoid static model access from decision code; inject a model evaluator or model registry abstraction.
  • Make the dependency on in-process game handlers explicit through an IAiGameActionInvoker.

3. Config

Weakly aligned. Ai:NeuralModelPath is configuration-driven, but many runtime settings are hardcoded.

Plan:

  • Add typed options under Ai.
  • Validate options at startup for active roles.
  • Move runtime constants into config where they affect operations.
  • Fail fast when required config for the active role is missing.

Candidate config shape:

{
  "Ai": {
    "Role": "runtime",
    "Runtime": {
      "PollIntervalMs": 1000,
      "TurnWatchdogMs": 10000,
      "MaxConcurrentTurns": 32,
      "SessionLeaseSeconds": 60
    },
    "Neural": {
      "ModelPath": "",
      "InputName": "features",
      "OutputName": "score",
      "RequiredForTiers": []
    },
    "Seeding": {
      "EnabledAtStartup": false
    },
    "Training": {
      "MaxParallelGames": 4,
      "OutputPath": ""
    }
  }
}

4. Backing Services

Partially aligned. PostgreSQL, NATS, and the ONNX model file are effectively backing services, but only database and NATS are treated that way operationally.

Plan:

  • Treat the neural model as an attached resource with version, checksum, and readiness state.
  • Move model artifacts to object storage, a mounted volume, or a release-provided path with explicit version metadata.
  • Treat active AI session ownership as a backing service concern if more than one worker is allowed.
  • Store training outputs in an attached data store rather than process-local paths for shared environments.

5. Build, Release, Run

Weakly aligned for AI-specific operations. Runtime startup currently performs seeding and model loading. Model artifact availability is not clearly separated from build or release.

Plan:

  • Keep the application build independent from the neural model artifact.
  • Promote model version and path through release configuration.
  • Move account/deck seeding to a release/admin command.
  • Make startup read-only except for runtime session claims and telemetry.
  • Add a release checklist for model version, model checksum, required database migrations, and AI worker count.

6. Processes

Partially aligned. The AI worker is a long-running process role, but role boundaries are spread across SOR.Host and AddAiService.

Plan:

  • Replace one broad AddAiService method with role-specific registration methods:
    • AddAiCore
    • AddAiGameHandlers
    • AddAiRuntimeWorker
    • AddAiAdminServices
    • AddAiTrainingJobs
  • Ensure only ai-worker starts AiPlayerSessionManager.
  • Ensure game hosts can process add/remove/configure AI messages without starting session loops.
  • Add startup validation that rejects unsafe combinations, such as multiple uncoordinated AI workers.

7. Port Binding

Mostly aligned through SOR.Host. SOR.Ai does not bind ports directly; it participates in host-provided HTTP, health, and NATS connectivity.

Plan:

  • Keep port binding in SOR.Host.
  • Add AI-specific health/readiness endpoints through host health checks.
  • Expose AI readiness without adding an AI-specific listener.

8. Concurrency

Weakly aligned for horizontal scale. The AI worker uses in-memory sessions and fire-and-forget task scheduling.

Plan:

  • Add bounded turn execution with MaxConcurrentTurns.
  • Track all session and turn tasks so shutdown can wait for or cancel them.
  • Replace untracked Task.Run calls with a supervised background task registry or channel-based worker queue.
  • Add a session lease/claim store if more than one AI worker can run.
  • Use NATS queue groups or durable consumer semantics when event fan-out is not desired.
  • Define idempotency for GameStarted, GameEnded, and ELO recording.

9. Disposability

Partially aligned. Cancellation tokens are present, but in-flight work is not centrally owned.

Plan:

  • Make StopAsync cancel polling loops, stop accepting new turns, and wait for active turn tasks up to a configured timeout.
  • Check cancellation before each game action invocation.
  • Persist enough session progress to recover after an AI worker restart.
  • Make model reload and disposal observable and cancellation-aware.
  • Move startup seeding out of normal process startup to reduce boot-time side effects.

10. Dev/Prod Parity

Partially aligned. Missing neural models fall back gracefully, which helps development, but production expectations are not encoded.

Plan:

  • Add environment-specific validation profiles.
  • Allow fallback neural behavior in dev/test, but optionally require model availability in production for configured tiers.
  • Provide local examples for DB, NATS, model path, and training output paths.
  • Keep self-play and calibration behavior consistent between local and production-like job runners.

11. Logs

Partially aligned. The module uses structured logging through ILogger, but runtime observability is thin.

Plan:

  • Add structured logs for session start/end, lease acquisition, turn decision, action dispatch, model version, fallback mode, calibration job status, and ELO updates.
  • Add metrics for active AI sessions, turn latency, decision latency, handler failures, model load status, fallback count, and self-play throughput.
  • Avoid logging full game state or sensitive account data.
  • Add correlation fields: gameSessionId, aiAccountId, seatIndex, difficulty, strategy, modelVersion.

12. Admin Processes

Weakly aligned. Seeding, calibration, training, model reload, and replay export are operational tasks but are registered with the same module as runtime services.

Plan:

  • Convert seeding to explicit commands or jobs.
  • Run calibration and self-play as cancellable background jobs with durable job records.
  • Keep admin UI as a trigger/control plane, not the owner of long-running compute.
  • Add job status, cancellation, retry, and output artifact tracking.
  • Make model reload an admin operation with validation and audit logging.

Refactor Phases

Phase 0: Baseline and Safety Tests

Goals:

  • Preserve current AI behavior before restructuring.
  • Create tests around the most fragile runtime paths.

Work:

  • Add tests for AiSlotAssigner, AiDeckResolver, AiDifficultyProfiles, ActionCandidateGenerator, and ScoringFunctions.
  • Add integration-style tests for add/remove/configure AI handlers.
  • Add a worker-registration test proving AiPlayerSessionManager only starts for the AI runtime role.
  • Add tests for missing neural model fallback.

Done when:

  • Existing behavior has executable coverage.
  • Refactoring can proceed without relying only on manual game testing.

Phase 1: Typed Configuration

Goals:

  • Make AI runtime behavior visible and validated.

Work:

  • Add AiOptions, AiRuntimeOptions, AiNeuralOptions, AiTrainingOptions, and AiSeedingOptions.
  • Bind config through IOptions.
  • Validate active-role requirements at startup.
  • Move polling intervals, watchdog timeouts, max concurrency, neural model path, training parallelism, and seeding-at-startup behavior into options.

Done when:

  • The AI role fails fast on missing required config.
  • Local dev config remains simple.
  • Production config can require a model for neural tiers if desired.

Phase 2: Split Service Registration by Role

Goals:

  • Make process formation explicit.

Work:

  • Break AddAiService into focused registration methods.
  • Keep handler registration available to game-enabled hosts.
  • Start runtime session management only through AddAiRuntimeWorker.
  • Register admin services separately from worker services.
  • Register training/calibration jobs separately from web/runtime services.

Done when:

  • game, ai-worker, admin, and ai-jobs can be reasoned about independently.
  • Startup comments are backed by code-level registration boundaries.

Phase 3: Move Seeding Out of Startup

Goals:

  • Keep runtime startup disposable and side-effect-light.

Work:

  • Convert AI account/deck seeding into explicit commands.
  • Keep a guarded development-only startup seed option if needed.
  • Make seed commands idempotent and auditable.
  • Add release documentation for when seed commands must run.

Done when:

  • Production runtime startup no longer mutates seed data by default.
  • Seeding is a 12-Factor admin process.

Phase 4: Supervise Runtime Sessions

Goals:

  • Make AI worker concurrency bounded, observable, and disposable.

Work:

  • Introduce AiSessionCoordinator to own session lifecycle.
  • Replace untracked Task.Run with tracked tasks or channel-based workers.
  • Add bounded turn execution via options.
  • Make ELO updates idempotent and supervised.
  • Add graceful shutdown behavior that cancels sessions and waits for active turns.

Done when:

  • The worker can stop without abandoning unknown in-flight tasks.
  • Turn concurrency can be tuned per deployment.
  • Worker logs and metrics show active sessions and active turns.

Phase 5: Add Distributed Session Ownership

Goals:

  • Make AI runtime safe to scale beyond one process.

Work:

  • Decide whether the supported production model is exactly one AI worker or multiple workers.
  • If exactly one worker, enforce that operationally through deployment docs and readiness warnings.
  • If multiple workers, add a lease/claim store for GameSessionId ownership.
  • Use NATS queue groups or durable consumers where duplicate event delivery would be unsafe.
  • Add idempotency checks for duplicate GameStarted and GameEnded events.

Done when:

  • Running two AI workers cannot cause duplicate turn execution.
  • Session ownership survives worker restart.

Phase 6: Extract Turn Execution and Game Action Invocation

Goals:

  • Make AI turns testable without a full hosted worker.

Work:

  • Extract AiTurnExecutor from AiPlayerSession.
  • Extract IAiGameActionInvoker for handler lookup and authenticated no-op connection creation.
  • Extract IAiDecisionEngineFactory instead of constructing engines directly in sessions.
  • Keep the actual game action messages unchanged.

Done when:

  • A turn can be unit-tested with fake game state, fake decision engine, and fake action invoker.
  • Runtime session code mostly coordinates polling, cancellation, and lifecycle.

Phase 7: Model Artifact Management

Goals:

  • Treat neural models as release resources, not incidental local files.

Work:

  • Replace static NeuralModelRegistry.Instance access with injected model evaluation services.
  • Add model metadata: version, checksum, loaded timestamp, input contract, output contract.
  • Add readiness state for required neural models.
  • Add explicit reload operation with audit logging and failure reporting.
  • Decide whether models are local release files, mounted volumes, or object-store artifacts.

Done when:

  • The active model version is visible in health/logs/admin.
  • Missing or invalid model behavior is controlled by config.
  • Neural evaluation no longer depends on global static state.

Phase 8: Training and Calibration Jobs

Goals:

  • Move compute-heavy AI work into proper one-off or worker jobs.

Work:

  • Give calibration and self-play durable job records.
  • Store output artifacts through an attached resource.
  • Configure max parallel games, timeouts, and cancellation.
  • Make admin UI trigger jobs and display status instead of owning long-running compute.
  • Add replay export and training data retention policies.

Done when:

  • Training/calibration can run independently from the web/admin process lifecycle.
  • Jobs are resumable or explicitly retryable.
  • Output artifacts are discoverable by version.

Phase 9: Observability and Runbooks

Goals:

  • Make AI operations diagnosable in production.

Work:

  • Add AI-specific readiness checks.
  • Add metrics for session count, turn latency, decision latency, action failure count, fallback count, model status, and job throughput.
  • Add an operations runbook covering worker count, model deployment, seed jobs, calibration jobs, and failure recovery.
  • Document local dev setup for AI with and without a neural model.

Done when:

  • Operators can answer: which games have AI sessions, which model is loaded, whether neural fallback is active, how many turns are pending, and whether training jobs are healthy.

Suggested Work Order

  1. Add baseline tests for pure decision and handler behavior.
  2. Add typed options and active-role validation.
  3. Split DI registration by AI role.
  4. Move seeding to explicit admin/release commands.
  5. Supervise runtime tasks and add bounded concurrency.
  6. Add session ownership semantics for multi-worker safety.
  7. Extract turn execution and action invocation.
  8. Replace static neural registry access with injected model services.
  9. Move calibration/training to durable job workflows.
  10. Add AI readiness checks, metrics, and runbooks.

Completion Criteria

The refactor is complete when:

  • SOR.Ai has explicit registration paths for game handlers, runtime worker, admin services, and jobs.
  • Runtime startup is read-only except for leases, telemetry, and normal worker state.
  • AI worker concurrency and shutdown are bounded and observable.
  • Multiple AI workers are either prevented or coordinated safely.
  • Neural model artifacts are versioned, validated, and visible through health/admin surfaces.
  • Training, calibration, replay export, and seeding run as explicit admin processes.
  • Config required for active roles is validated at startup.
  • Tests cover decision behavior, AI lobby handlers, neural fallback, and worker registration.

Immediate Low-Risk Next Steps

  • Create AiOptions classes and bind them without changing behavior.
  • Add role-specific extension methods that internally call the current registrations, then move services gradually.
  • Add tests for AiDifficultyProfiles and missing neural model fallback.
  • Add a runtime warning when ENABLED_SERVICES starts the AI worker without the required game/AI handler registrations.
  • Add a short runbook documenting the current single-AI-worker assumption.