Files
aevolution/docs/engineering/VALIDATION.md
T

121 lines
7.1 KiB
Markdown

# Validation report
Recorded 2026-09-19. Latest verified Linux delivery: **run 1402 passed** on
`4c2cc04ac47d37c6191a9ef02d4bbba716a930dc`. Native checks apply to base commit `8ab48d3` plus the
foundation-delivery working changes committed with this report. Simulation rules
and snapshot format remain version 1; the only C++ change is benchmark metadata.
Results below distinguish executed checks from pending Linux/cluster gates.
## Tools and host
- Windows NT 10.0.26200, AMD Ryzen 7 9800X3D, 8 cores / 16 logical processors.
- MSVC 19.44.35217.0, CMake 3.31.6-msvc6, Ninja 1.12.1.
- clang-format and clang-tidy 19.1.5 (Visual Studio LLVM tools).
- kubectl 1.31.4 / Kustomize 5.4.2 for offline manifest rendering.
## Executed native checks
| Command | Observed result |
|---|---|
| `./tools/windows-check.ps1 -Preset debug -Action check` | Configure/build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset release -Action check` | Configure/build passed; 5/5 CTests passed under NDEBUG |
| `./tools/windows-check.ps1 -Preset analysis -Action check` | clang-tidy/warnings-as-errors build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset asan -Action check` | MSVC AddressSanitizer build passed; 5/5 CTests passed; this is NOT UBSan |
| `./tools/windows-check.ps1 -Preset debug -Action format` | Passed with clang-format 19; repeated twice in one shell after helper fix |
| `./build/release/evo-sim --seed 123456 --entities 8 --ticks 1000000 --verify-replay` | Exit 0; replay_verified=true; hash fab22c2e8ae1c379 |
| `./build/release/evo-benchmark` | Exit 0; MSVC 1944 Release, seed 123456, 10000 entities, 1000 ticks, 0.0210666 seconds, hash 3bad6db1612cbd1 |
| Git Bash `bash -n tools/ci/linux-check.sh` and `bash -n tools/ci/publish-image.sh` | Syntax checks passed; no Linux execution implied |
| `kubectl kustomize deploy/kubernetes/dev` | Rendered Job, ServiceAccount and NetworkPolicy in evolution-dev; final render pins the verified run-1402 image |
CTest covers the independently frozen 1000-tick fingerprint d61ad5bf0c928c76,
seeded replay properties, malformed/semantic snapshot rejection, entity lifecycle,
capacity/overflow, telemetry and CLI file continuation/alias protection. The
expected-failure harness confirms tests remain active in Release.
The benchmark is a single uncontrolled wall-clock observation, not a performance
threshold or an MMO capacity estimate. Its initial compiler output accidentally
used hexadecimal; the stream now returns to decimal and reports MSVC 1944.
## Gitea evidence
The maintainer's [run 1399](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1399)
on `8ab48d3a5639121693f5881b600482e870b49f05` completed successfully. API job records
confirm verify (configure/build/CTest) and publish (Docker build/push) succeeded
on runner theraborn-runners-runner-0, label ubuntu-22.04. This is evidence for the
original workflow, not the expanded sanitizer/analysis jobs added in this change.
Exact Linux tool versions and expanded-run results appear below; a runner label
is not a compiler version.
## Limitations and next gates
- Local Docker Desktop has no running Linux engine. Container checks must run on
the configured Gitea runner; no local container success is claimed.
- Expanded Linux GCC14/Clang19, ASan/UBSan, TSan and libFuzzer evidence is recorded
below. TSan currently validates a sequential kernel, not future concurrency.
- Sandbox process restrictions stalled native Ninja and blocked Git Bash signal
pipes. Those attempts were stopped; checks reran successfully outside sandbox.
- Repeated VsDevCmd initialization grew PATH beyond cmd's limit. The helper now
reuses the matching x64 environment and deduplicates PATH; repeated checks pass.
- The maintainer reports Argo setup, but this workspace has no Kubernetes context.
No dev Job completion or cluster admission result is recorded yet.
- No biology, network/load, durable server recovery or backup restore exists.
Those remain M2, M7/M8 and later gates, not failing or passing foundation tests.
Update this report with run URL, tested commit, image digest and Job result when
those checks execute. Never overwrite a golden merely to make CI pass.
## Expanded Linux run 1400
[Run 1400](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1400),
commit 9e412736d91ad103b7a951f99b811c629c5ac61e: GCC14 Debug and Release, Clang19
analysis and ASan/UBSan each passed 5/5 tests. Formatting passed. The seeded fuzz
smoke completed 10000 cases with no reported failure. CMake 3.28.3, GCC 14.2.0,
Clang/format/tidy 19.1.1, Linux 6.8.0-110-generic x86-64. Million-tick hash matched
Windows (fab22c2e8ae1c379); benchmark hash matched too (3bad6db1612cbd1), with
0.0674131 seconds observed for 10000 entities * 1000 ticks on the CI host.
TSan compiled but all executions failed before main: its runtime could not call
personality(ADDR_NO_RANDOMIZE) inside the Docker build sandbox. The image job was
skipped, as required. This is not recorded as a TSan pass. The follow-up isolates
execution in a container with the narrowly constrained profile from ADR-0020;
its result must be recorded after execution.
## Successful delivery run 1402
[Run 1402](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1402),
commit 4c2cc04ac47d37c6191a9ef02d4bbba716a930dc: every job succeeded. The general
checks reused Docker's valid cache for unchanged checked sources; their full
execution also passed in run 1401. TSan executed all five CTests successfully in
2.71 seconds using the scoped profile and writable tmpfs outputs. Run 1401 had
first exposed a non-root tmpfs permission error; setting explicit mode=1777 on
these isolated scratch mounts fixed it. No test was disabled or ignored.
The runtime image built, ran the restricted replay smoke test and was pushed to:
`registry.aeternalisgames.com/aeternalis/aevolution:sha-4c2cc04ac47d37c6191a9ef02d4bbba716a930dc-run-6-1`
Registry digest:
`sha256:8fce2b462c9907f38d3fb832afeb70d0bff068dd832b2171e861186dbcb8e6b9`.
The dev manifest pins this digest. This proves build/test/image delivery; Argo
application mapping and Kubernetes Job completion remain unverified locally.
## M1.5 native ECS experiment
Opt-in target evo-ecs-experiment (EnTT4 pinned in ADR-0021), based on 1c6e6e1 plus
this experiment's changes. MSVC Release contract checks passed, including 72
seed/density/count combinations, hand-calculated arithmetic, canonical equality,
restore continuation, retired-ID rejection, malformed snapshot checks and
allocator balance. The final clang-tidy build/test passed after tightening the
process-boundary diagnostic catch. MSVC ASan passed on the final source.
Baseline Release CTest remained 5/5
and format-check included the experiment. No simulation rule/golden changed.
Commands: configure release/analysis/asan with `-B build/ecs[-analysis|-asan]
-DEVO_BUILD_ECS_EXPERIMENT=ON`, build target evo-ecs-experiment, then
`ctest --test-dir <directory> -R ecs-experiment --output-on-failure`.
Seven measured samples plus warmup are retained in results/ecs-msvc1944.jsonl;
protocol, build observations and scope limits are in ECS_EVALUATION.md. The
Linux experiment CI job is pending its first pushed run; no Linux experiment
success is claimed yet.