- tools/train_neural.py: PyTorch MLP training script that reads JSONL from GameReplayExporter and exports a compliant ONNX model. Architecture is 1024->512->256->128->1 (tanh) with LayerNorm + Dropout. AdamW optimizer with cosine LR annealing, gradient clipping, best-checkpoint restore. Post-export validation via onnxruntime confirms tensor names and shapes. - tools/requirements.txt: pinned deps (torch 2.3.1, onnx 1.16.1, onnxruntime 1.18.0 matching the C# package, numpy 1.26.4) - tools/README.md: end-to-end workflow docs covering self-play, data export, training options, deployment (hot-swap vs restart), ONNX contract table, and iterative training guidance Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>
146 lines
4.1 KiB
Markdown
146 lines
4.1 KiB
Markdown
# SOR Neural AI Training Tools
|
|
|
|
This directory contains the Python training pipeline for the SOR neural decision engine
|
|
(tiers 9-10). The workflow is:
|
|
|
|
```
|
|
self-play (admin) -> JSONL export (admin) -> train_neural.py -> model.onnx -> hot-swap (admin)
|
|
```
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
Python 3.11+ with pip:
|
|
|
|
```bash
|
|
pip install -r tools/requirements.txt
|
|
```
|
|
|
|
GPU training is supported via CUDA. Use `--cpu` to force CPU-only mode.
|
|
|
|
---
|
|
|
|
## Step 1 — Generate self-play data (Admin portal)
|
|
|
|
1. Navigate to **Admin -> AI -> Calibration**.
|
|
2. Click **Run All Pairs** (or a specific difficulty pair) to trigger in-process self-play games.
|
|
3. Wait for the run to complete — check the calibration table for results.
|
|
|
|
Alternatively, trigger a full calibration run via the API:
|
|
```
|
|
POST /api/ai/calibration/run-all
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2 — Export training data (Admin portal)
|
|
|
|
1. Navigate to **Admin -> AI -> Calibration**.
|
|
2. Click **Export Training Data** and enter an output path on the server
|
|
(e.g. `/var/sor/training/data.jsonl`).
|
|
3. The exporter replays completed game sessions and writes one JSON line per action:
|
|
|
|
```json
|
|
{
|
|
"GameSessionId": "3fa85f64...",
|
|
"TurnNumber": 5,
|
|
"ActorAccountId": "...",
|
|
"Features": [0.0, 1.0, ...],
|
|
"ActionType": "AiPlayCardAction",
|
|
"ActionJson": "{}",
|
|
"Outcome": 1.0
|
|
}
|
|
```
|
|
|
|
`Outcome` is `+1.0` if the acting player won the game, `-1.0` if they lost.
|
|
|
|
---
|
|
|
|
## Step 3 — Train the model
|
|
|
|
```bash
|
|
cd /path/to/SOR
|
|
python tools/train_neural.py --data /var/sor/training/data.jsonl --out /var/sor/models/model.onnx --epochs 50 --batch-size 512 --lr 1e-3
|
|
```
|
|
|
|
### Options
|
|
|
|
| Flag | Default | Description |
|
|
|------|---------|-------------|
|
|
| `--data` | *(required)* | Path to the JSONL export file |
|
|
| `--out` | `model.onnx` | Output ONNX file path |
|
|
| `--epochs` | `50` | Number of training epochs |
|
|
| `--batch-size` | `512` | Mini-batch size |
|
|
| `--lr` | `0.001` | AdamW initial learning rate |
|
|
| `--val-split` | `0.1` | Fraction of data held out for validation |
|
|
| `--dropout` | `0.2` | Dropout probability in the MLP |
|
|
| `--seed` | `42` | Random seed |
|
|
| `--cpu` | off | Force CPU training (ignores CUDA) |
|
|
|
|
The script prints per-epoch train/val loss and marks the best checkpoint with `*`.
|
|
It restores the best weights before exporting.
|
|
|
|
### Model architecture
|
|
|
|
```
|
|
Input [1, 1024] float32 (FeatureExtractor output)
|
|
Linear 1024 -> 512 + LayerNorm + ReLU + Dropout
|
|
Linear 512 -> 256 + LayerNorm + ReLU + Dropout
|
|
Linear 256 -> 128 + LayerNorm + ReLU
|
|
Linear 128 -> 1 + Tanh
|
|
Output [1, 1] float32 (score in [-1, 1])
|
|
```
|
|
|
|
Total parameters: ~920k
|
|
|
|
---
|
|
|
|
## Step 4 — Deploy the model
|
|
|
|
### Option A: Hot-swap via admin portal (no restart)
|
|
|
|
1. Copy the `model.onnx` file to the path configured in `appsettings`:
|
|
```json
|
|
"Ai": { "NeuralModelPath": "/var/sor/models/model.onnx" }
|
|
```
|
|
2. Navigate to **Admin -> AI -> Neural Model**.
|
|
3. Click **Hot-swap Model**. The old session is disposed atomically and the new
|
|
model is loaded from disk. All subsequent tier 9-10 decisions use the new model.
|
|
|
|
### Option B: Restart the service
|
|
|
|
The model is loaded at startup by `AiNeuralStartupService`. If the file exists at the
|
|
configured path, it will be loaded automatically on the next deploy.
|
|
|
|
---
|
|
|
|
## ONNX contract
|
|
|
|
The exported model must satisfy:
|
|
|
|
| Property | Value |
|
|
|----------|-------|
|
|
| Input tensor name | `features` |
|
|
| Input shape | `[1, 1024]` (or `[batch, 1024]` with dynamic batch) |
|
|
| Input dtype | `float32` |
|
|
| Output tensor name | `score` |
|
|
| Output shape | `[1, 1]` (or `[batch, 1]`) |
|
|
| Output dtype | `float32` |
|
|
| Output range | `[-1, 1]` (tanh) |
|
|
|
|
These names are defined as constants in `NeuralModelRegistry.cs` (`InputName`, `OutputName`).
|
|
The Python script and the C# inference code must agree on these names exactly.
|
|
|
|
---
|
|
|
|
## Iterative training
|
|
|
|
For best results, run multiple rounds:
|
|
|
|
1. Start with random-engine games (all difficulties) to generate diverse data.
|
|
2. Train an initial model and deploy it.
|
|
3. Run additional self-play with the neural engine enabled (tiers 9-10 play each other).
|
|
4. Append new exports to the existing JSONL and retrain with more epochs.
|
|
5. Monitor calibration win rates in the admin portal after each deployment.
|