Files
aevolution/docs/engineering/VALIDATION.md
T

257 lines
16 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Validation report
Recorded 2026-09-19. Latest verified Linux delivery: **run 1402 passed** on
`4c2cc04ac47d37c6191a9ef02d4bbba716a930dc`. Native checks apply to base commit `8ab48d3` plus the
foundation-delivery working changes committed with this report. Simulation rules
and snapshot format remain version 1; the only C++ change is benchmark metadata.
Results below distinguish executed checks from pending Linux/cluster gates.
## Tools and host
- Windows NT 10.0.26200, AMD Ryzen 7 9800X3D, 8 cores / 16 logical processors.
- MSVC 19.44.35217.0, CMake 3.31.6-msvc6, Ninja 1.12.1.
- clang-format and clang-tidy 19.1.5 (Visual Studio LLVM tools).
- kubectl 1.31.4 / Kustomize 5.4.2 for offline manifest rendering.
## Executed native checks
| Command | Observed result |
|---|---|
| `./tools/windows-check.ps1 -Preset debug -Action check` | Configure/build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset release -Action check` | Configure/build passed; 5/5 CTests passed under NDEBUG |
| `./tools/windows-check.ps1 -Preset analysis -Action check` | clang-tidy/warnings-as-errors build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset asan -Action check` | MSVC AddressSanitizer build passed; 5/5 CTests passed; this is NOT UBSan |
| `./tools/windows-check.ps1 -Preset debug -Action format` | Passed with clang-format 19; repeated twice in one shell after helper fix |
| `./build/release/evo-sim --seed 123456 --entities 8 --ticks 1000000 --verify-replay` | Exit 0; replay_verified=true; hash fab22c2e8ae1c379 |
| `./build/release/evo-benchmark` | Exit 0; MSVC 1944 Release, seed 123456, 10000 entities, 1000 ticks, 0.0210666 seconds, hash 3bad6db1612cbd1 |
| Git Bash `bash -n tools/ci/linux-check.sh` and `bash -n tools/ci/publish-image.sh` | Syntax checks passed; no Linux execution implied |
| `kubectl kustomize deploy/kubernetes/dev` | Rendered Job, ServiceAccount and NetworkPolicy in evolution-dev; final render pins the verified run-1402 image |
CTest covers the independently frozen 1000-tick fingerprint d61ad5bf0c928c76,
seeded replay properties, malformed/semantic snapshot rejection, entity lifecycle,
capacity/overflow, telemetry and CLI file continuation/alias protection. The
expected-failure harness confirms tests remain active in Release.
The benchmark is a single uncontrolled wall-clock observation, not a performance
threshold or an MMO capacity estimate. Its initial compiler output accidentally
used hexadecimal; the stream now returns to decimal and reports MSVC 1944.
## Gitea evidence
The maintainer's [run 1399](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1399)
on `8ab48d3a5639121693f5881b600482e870b49f05` completed successfully. API job records
confirm verify (configure/build/CTest) and publish (Docker build/push) succeeded
on runner theraborn-runners-runner-0, label ubuntu-22.04. This is evidence for the
original workflow, not the expanded sanitizer/analysis jobs added in this change.
Exact Linux tool versions and expanded-run results appear below; a runner label
is not a compiler version.
## Limitations and next gates
- Local Docker Desktop has no running Linux engine. Container checks must run on
the configured Gitea runner; no local container success is claimed.
- Expanded Linux GCC14/Clang19, ASan/UBSan, TSan and libFuzzer evidence is recorded
below. TSan currently validates a sequential kernel, not future concurrency.
- Sandbox process restrictions stalled native Ninja and blocked Git Bash signal
pipes. Those attempts were stopped; checks reran successfully outside sandbox.
- Repeated VsDevCmd initialization grew PATH beyond cmd's limit. The helper now
reuses the matching x64 environment and deduplicates PATH; repeated checks pass.
- The maintainer reports Argo setup, but this workspace has no Kubernetes context.
No dev Job completion or cluster admission result is recorded yet.
- No biology, network/load, durable server recovery or backup restore exists.
Those remain M2, M7/M8 and later gates, not failing or passing foundation tests.
Update this report with run URL, tested commit, image digest and Job result when
those checks execute. Never overwrite a golden merely to make CI pass.
## Expanded Linux run 1400
[Run 1400](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1400),
commit 9e412736d91ad103b7a951f99b811c629c5ac61e: GCC14 Debug and Release, Clang19
analysis and ASan/UBSan each passed 5/5 tests. Formatting passed. The seeded fuzz
smoke completed 10000 cases with no reported failure. CMake 3.28.3, GCC 14.2.0,
Clang/format/tidy 19.1.1, Linux 6.8.0-110-generic x86-64. Million-tick hash matched
Windows (fab22c2e8ae1c379); benchmark hash matched too (3bad6db1612cbd1), with
0.0674131 seconds observed for 10000 entities * 1000 ticks on the CI host.
TSan compiled but all executions failed before main: its runtime could not call
personality(ADDR_NO_RANDOMIZE) inside the Docker build sandbox. The image job was
skipped, as required. This is not recorded as a TSan pass. The follow-up isolates
execution in a container with the narrowly constrained profile from ADR-0020;
its result must be recorded after execution.
## Successful delivery run 1402
[Run 1402](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1402),
commit 4c2cc04ac47d37c6191a9ef02d4bbba716a930dc: every job succeeded. The general
checks reused Docker's valid cache for unchanged checked sources; their full
execution also passed in run 1401. TSan executed all five CTests successfully in
2.71 seconds using the scoped profile and writable tmpfs outputs. Run 1401 had
first exposed a non-root tmpfs permission error; setting explicit mode=1777 on
these isolated scratch mounts fixed it. No test was disabled or ignored.
The runtime image built, ran the restricted replay smoke test and was pushed to:
`registry.aeternalisgames.com/aeternalis/aevolution:sha-4c2cc04ac47d37c6191a9ef02d4bbba716a930dc-run-6-1`
Registry digest:
`sha256:8fce2b462c9907f38d3fb832afeb70d0bff068dd832b2171e861186dbcb8e6b9`.
The dev manifest pins this digest. This proves build/test/image delivery; Argo
application mapping and Kubernetes Job completion remain unverified locally.
## M1.5 native ECS experiment
Opt-in target evo-ecs-experiment (EnTT4 pinned in ADR-0021), based on 1c6e6e1 plus
this experiment's changes. MSVC Release contract checks passed, including 72
seed/density/count combinations, hand-calculated arithmetic, canonical equality,
restore continuation, retired-ID rejection, malformed snapshot checks and
allocator balance. The final clang-tidy build/test passed after tightening the
process-boundary diagnostic catch. MSVC ASan passed on the final source.
Baseline Release CTest remained 5/5
and format-check included the experiment. No simulation rule/golden changed.
Commands: configure release/analysis/asan with `-B build/ecs[-analysis|-asan]
-DEVO_BUILD_ECS_EXPERIMENT=ON`, build target evo-ecs-experiment, then
`ctest --test-dir <directory> -R ecs-experiment --output-on-failure`.
Seven measured samples plus warmup are retained in results/ecs-msvc1944.jsonl;
protocol, build observations and scope limits are in ECS_EVALUATION.md. The
Linux experiment CI job is pending its first pushed run; no Linux experiment
success is claimed yet.
## M1.5 Linux evidence
[Run 1404](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1404),
source bf7d60b7cd174736d38aea42616064e55e0c5f05: every job passed. The new ci-ecs
stage explicitly prepared the pinned header/license, then built and ran the
experiment under GCC14 Release, Clang19 ASan/UBSan and Clang19 analysis. All three
contract invocations passed. Raw measured GCC samples are retained at
results/ecs-gcc142.jsonl; all 56 fingerprints match corresponding Windows samples.
No production rule or snapshot-format change occurred. This supersedes the
pending Linux experiment note above. The final follow-up edits only record
results and decision scope; production ECS adoption remains deferred.
## M1.6 sequential scheduler (2026-09-19)
Native commands: `tools/windows-check.ps1 -Preset release -Action configure`,
clang-format on changed C++ files, `cmake --build --preset release`,
`ctest --preset release`; then `tools/windows-check.ps1 -Preset analysis -Action check`,
`tools/windows-check.ps1 -Preset asan -Action check` and Release `-Action format`.
All three configurations passed 5/5 CTests; clang-tidy and format-check passed.
Tests execute under NDEBUG and cover six batch permutations across 32 seeds,
an independent direct-lifecycle reference, ordered receipts, restore continuation,
rejected conflicts and malformed admission, the 1,024-command boundary, full entity
capacity and tick/ID exhaustion. Golden `d61ad5bf0c928c76` remains unchanged.
The copy-and-commit implementation allocates before publishing world state and
statically checks noexcept publication/receipt moves. Allocation-failure injection
was not run. No scheduler throughput claim, concurrency validation, durable input
journal or Kubernetes execution is implied.
[Gitea run 1407](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1407)
passed all four jobs for source `b620edfd6f8097172acf055402a24620cb078358`:
Linux checks, the ECS experiment, separate TSan, and image smoke/publication.
This closes the pending Linux validation for M1.6. The maintainer's independent
dev image pin from `20e609d` was preserved during rebase. Kustomize renders the
three expected dev resources successfully; no cluster execution is claimed.
## M1.7 spatial queries (2026-09-19)
Release and MSVC ASan passed 5/5 CTests. Tests cover periodic-image reference
comparison (32 seeds × 3 lifecycle/restore phases × 24 queries), hand-checked
seams/corners/inclusive radii, empty/full capacity, invalid centers and maximum
u32 radius. Queries leave snapshots unchanged; existing v1 goldens remain intact.
Native commands use `tools/windows-check.ps1 -Preset <release|analysis|asan>
-Action check` and Release `-Action format`. Initial analysis flagged an unnecessary
test-helper vector copy; the helper now takes a const reference. Final analysis,
Release, ASan and formatting passed after applying the helper's required line
wrapping. Linux validation is recorded below.
`build/release/evo-spatial-benchmark.exe` produced 72 measured records plus
metadata in results/spatial-msvc1944.jsonl. One warmup and three samples per
scenario; 1k/10k/100k entities, 128 queries, uniform/clustered positions, four radii.
Timing includes result allocation, destruction and checksum work, excludes setup;
no performance threshold. Desktop CPU load was not isolated. No cluster changes
are required for this library increment.
[Gitea run 1409](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1409)
passed all four jobs for source `9b19509673d88683ee72e093e42534bfe151f928`:
Linux checks, ECS regression, separate TSan and image smoke/publication.
All 72 spatial benchmark checksums and match counts agree with Windows.
Raw Linux measurements are in results/spatial-gcc142.jsonl; 24 native scenarios
also have identical checksums across their three repetitions. No deployment
manifest, rules version, snapshot format or golden was changed.
## M2.1 finite inventory ledger (2026-09-19)
Native Release, analysis and MSVC ASan each passed 6/6 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. Initial clang-tidy rejected the Element enum's
oversized base type; changed to u8 and reran all configurations successfully.
The new independent ledger test checks typed energy/mass separation, finite-source
exhaustion, explicit heat transfer, intermediate overdraft, whole-transaction
rollback, retry/duplicate action semantics, invalid reservoir IDs, maximum batch
and reservoir counts, opening-total overflow and exact u64-limit transfers.
32 seeded runs × 31,250 transactions = one million transactions, each checked
against independent reference balances, a replay branch and recomputed per-channel
totals. This is transfer replay, not serialized recovery or an ecosystem run.
Final ledger tests took 0.63s Release and 7.62s ASan locally; not controlled
performance measurements. Existing particle goldens remain unchanged.
Allocation-failure injection, durable ledger recovery and cross-ledger ownership
are not tested or implemented. Rates and conversions are absent, so rounding
residuals cannot occur in this transfer-only scope.
[Gitea run 1411](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1411),
source `8fd3591b0e37aaecf00a6410a5926745a031cb6f`, passed
the Linux checks, ECS regression and separate TSan jobs. Each main Linux
configuration passed 6/6 CTests. Ledger-suite times were 3.34s GCC Debug, 0.37s GCC
Release, 2.46s Clang analysis and 8.78s Clang ASan/UBSan; these are CI test
durations, not controlled performance benchmarks. The image smoke/publication
job also passed, completing all four jobs. No deployment image pin was changed.
## M2.2 finite producer process (2026-09-19)
Native Release, clang-tidy and MSVC ASan passed 7/7 CTests; format-check passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. The producer suite compares hand-calculated ticks
and 32 seeds × 1,000 ticks against a small-domain cumulative arithmetic reference,
an independent replay branch and per-tick energy/element conservation audits.
It covers invalid config/ticks with unchanged state, fractional capture/growth,
efficiency endpoints, finite source exhaustion, all four limiting elements,
zero capture, maximum u64 capture with 50% carried conversion, and maximum recipe
and growth-cost multiplication. Particle goldens remain unchanged.
Allocation-failure injection and tick-counter exhaustion were not exercised;
the latter has an explicit guard but no arbitrary-state restore API. Tests use
in-memory copies, not serialized recovery. No photosynthetic chemistry, metabolic
model, database integration, throughput guarantee or deployment is claimed.
[Gitea run 1413](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1413)
passed all four jobs for source `af293b8b50cbb506a49dfff10c6cc92307e16aca`:
Linux checks, ECS regression, separate TSan and runtime image smoke/publication.
The initial push was delayed by a tool approval-review usage limit; retry succeeded
after the maintainer resumed work. No deployment image pin was changed.
## M2.3 shared feeding community (2026-09-20)
Native Release, clang-tidy and MSVC ASan each passed 8/8 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. The new suite checks hand-calculated capture/feeding,
producer-to-consumer and consumer-to-consumer predation, heat/waste outputs,
terminal actor rejection, full-tick rollback, competing resource requests,
duplicate sequence/consumer admission, per-channel intake caps, invalid actors,
config/capacity limits and u64 assimilation endpoints. 32 seeds × 200 ticks compare
reordered input batches, receipts, full balances and producer progress, with
independently recomputed energy and elemental totals. Standalone producer and
existing particle golden tests remain unchanged and passing.
No serialized recovery, allocation-failure injection, spatial reach/perception,
physical injury, corpse handling or persistent-world integration was tested or
implemented. These are co-located synthetic process fixtures.
[Gitea run 1415](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1415)
passed all four jobs for source `b7dd466f57f385d6a8ffccfaa69f639077545587`:
Linux checks, ECS regression, separate TSan and image smoke/publication.
No deployment image pin was changed.