Files
aevolution/docs/engineering/VALIDATION.md
T

16 KiB
Raw Blame History

Validation report

Recorded 2026-09-19. Latest verified Linux delivery: run 1402 passed on 4c2cc04ac47d37c6191a9ef02d4bbba716a930dc. Native checks apply to base commit 8ab48d3 plus the foundation-delivery working changes committed with this report. Simulation rules and snapshot format remain version 1; the only C++ change is benchmark metadata. Results below distinguish executed checks from pending Linux/cluster gates.

Tools and host

  • Windows NT 10.0.26200, AMD Ryzen 7 9800X3D, 8 cores / 16 logical processors.
  • MSVC 19.44.35217.0, CMake 3.31.6-msvc6, Ninja 1.12.1.
  • clang-format and clang-tidy 19.1.5 (Visual Studio LLVM tools).
  • kubectl 1.31.4 / Kustomize 5.4.2 for offline manifest rendering.

Executed native checks

Command Observed result
./tools/windows-check.ps1 -Preset debug -Action check Configure/build passed; 5/5 CTests passed
./tools/windows-check.ps1 -Preset release -Action check Configure/build passed; 5/5 CTests passed under NDEBUG
./tools/windows-check.ps1 -Preset analysis -Action check clang-tidy/warnings-as-errors build passed; 5/5 CTests passed
./tools/windows-check.ps1 -Preset asan -Action check MSVC AddressSanitizer build passed; 5/5 CTests passed; this is NOT UBSan
./tools/windows-check.ps1 -Preset debug -Action format Passed with clang-format 19; repeated twice in one shell after helper fix
./build/release/evo-sim --seed 123456 --entities 8 --ticks 1000000 --verify-replay Exit 0; replay_verified=true; hash fab22c2e8ae1c379
./build/release/evo-benchmark Exit 0; MSVC 1944 Release, seed 123456, 10000 entities, 1000 ticks, 0.0210666 seconds, hash 3bad6db1612cbd1
Git Bash bash -n tools/ci/linux-check.sh and bash -n tools/ci/publish-image.sh Syntax checks passed; no Linux execution implied
kubectl kustomize deploy/kubernetes/dev Rendered Job, ServiceAccount and NetworkPolicy in evolution-dev; final render pins the verified run-1402 image

CTest covers the independently frozen 1000-tick fingerprint d61ad5bf0c928c76, seeded replay properties, malformed/semantic snapshot rejection, entity lifecycle, capacity/overflow, telemetry and CLI file continuation/alias protection. The expected-failure harness confirms tests remain active in Release.

The benchmark is a single uncontrolled wall-clock observation, not a performance threshold or an MMO capacity estimate. Its initial compiler output accidentally used hexadecimal; the stream now returns to decimal and reports MSVC 1944.

Gitea evidence

The maintainer's run 1399 on 8ab48d3a5639121693f5881b600482e870b49f05 completed successfully. API job records confirm verify (configure/build/CTest) and publish (Docker build/push) succeeded on runner theraborn-runners-runner-0, label ubuntu-22.04. This is evidence for the original workflow, not the expanded sanitizer/analysis jobs added in this change. Exact Linux tool versions and expanded-run results appear below; a runner label is not a compiler version.

Limitations and next gates

  • Local Docker Desktop has no running Linux engine. Container checks must run on the configured Gitea runner; no local container success is claimed.
  • Expanded Linux GCC14/Clang19, ASan/UBSan, TSan and libFuzzer evidence is recorded below. TSan currently validates a sequential kernel, not future concurrency.
  • Sandbox process restrictions stalled native Ninja and blocked Git Bash signal pipes. Those attempts were stopped; checks reran successfully outside sandbox.
  • Repeated VsDevCmd initialization grew PATH beyond cmd's limit. The helper now reuses the matching x64 environment and deduplicates PATH; repeated checks pass.
  • The maintainer reports Argo setup, but this workspace has no Kubernetes context. No dev Job completion or cluster admission result is recorded yet.
  • No biology, network/load, durable server recovery or backup restore exists. Those remain M2, M7/M8 and later gates, not failing or passing foundation tests.

Update this report with run URL, tested commit, image digest and Job result when those checks execute. Never overwrite a golden merely to make CI pass.

Expanded Linux run 1400

Run 1400, commit 9e412736d9: GCC14 Debug and Release, Clang19 analysis and ASan/UBSan each passed 5/5 tests. Formatting passed. The seeded fuzz smoke completed 10000 cases with no reported failure. CMake 3.28.3, GCC 14.2.0, Clang/format/tidy 19.1.1, Linux 6.8.0-110-generic x86-64. Million-tick hash matched Windows (fab22c2e8ae1c379); benchmark hash matched too (3bad6db1612cbd1), with 0.0674131 seconds observed for 10000 entities * 1000 ticks on the CI host.

TSan compiled but all executions failed before main: its runtime could not call personality(ADDR_NO_RANDOMIZE) inside the Docker build sandbox. The image job was skipped, as required. This is not recorded as a TSan pass. The follow-up isolates execution in a container with the narrowly constrained profile from ADR-0020; its result must be recorded after execution.

Successful delivery run 1402

Run 1402, commit 4c2cc04ac4: every job succeeded. The general checks reused Docker's valid cache for unchanged checked sources; their full execution also passed in run 1401. TSan executed all five CTests successfully in 2.71 seconds using the scoped profile and writable tmpfs outputs. Run 1401 had first exposed a non-root tmpfs permission error; setting explicit mode=1777 on these isolated scratch mounts fixed it. No test was disabled or ignored.

The runtime image built, ran the restricted replay smoke test and was pushed to:

registry.aeternalisgames.com/aeternalis/aevolution:sha-4c2cc04ac47d37c6191a9ef02d4bbba716a930dc-run-6-1

Registry digest: sha256:8fce2b462c9907f38d3fb832afeb70d0bff068dd832b2171e861186dbcb8e6b9.

The dev manifest pins this digest. This proves build/test/image delivery; Argo application mapping and Kubernetes Job completion remain unverified locally.

M1.5 native ECS experiment

Opt-in target evo-ecs-experiment (EnTT4 pinned in ADR-0021), based on 1c6e6e1 plus this experiment's changes. MSVC Release contract checks passed, including 72 seed/density/count combinations, hand-calculated arithmetic, canonical equality, restore continuation, retired-ID rejection, malformed snapshot checks and allocator balance. The final clang-tidy build/test passed after tightening the process-boundary diagnostic catch. MSVC ASan passed on the final source. Baseline Release CTest remained 5/5 and format-check included the experiment. No simulation rule/golden changed.

Commands: configure release/analysis/asan with -B build/ecs[-analysis|-asan] -DEVO_BUILD_ECS_EXPERIMENT=ON, build target evo-ecs-experiment, then ctest --test-dir <directory> -R ecs-experiment --output-on-failure. Seven measured samples plus warmup are retained in results/ecs-msvc1944.jsonl; protocol, build observations and scope limits are in ECS_EVALUATION.md. The Linux experiment CI job is pending its first pushed run; no Linux experiment success is claimed yet.

M1.5 Linux evidence

Run 1404, source bf7d60b7cd: every job passed. The new ci-ecs stage explicitly prepared the pinned header/license, then built and ran the experiment under GCC14 Release, Clang19 ASan/UBSan and Clang19 analysis. All three contract invocations passed. Raw measured GCC samples are retained at results/ecs-gcc142.jsonl; all 56 fingerprints match corresponding Windows samples. No production rule or snapshot-format change occurred. This supersedes the pending Linux experiment note above. The final follow-up edits only record results and decision scope; production ECS adoption remains deferred.

M1.6 sequential scheduler (2026-09-19)

Native commands: tools/windows-check.ps1 -Preset release -Action configure, clang-format on changed C++ files, cmake --build --preset release, ctest --preset release; then tools/windows-check.ps1 -Preset analysis -Action check, tools/windows-check.ps1 -Preset asan -Action check and Release -Action format. All three configurations passed 5/5 CTests; clang-tidy and format-check passed. Tests execute under NDEBUG and cover six batch permutations across 32 seeds, an independent direct-lifecycle reference, ordered receipts, restore continuation, rejected conflicts and malformed admission, the 1,024-command boundary, full entity capacity and tick/ID exhaustion. Golden d61ad5bf0c928c76 remains unchanged.

The copy-and-commit implementation allocates before publishing world state and statically checks noexcept publication/receipt moves. Allocation-failure injection was not run. No scheduler throughput claim, concurrency validation, durable input journal or Kubernetes execution is implied.

Gitea run 1407 passed all four jobs for source b620edfd6f8097172acf055402a24620cb078358: Linux checks, the ECS experiment, separate TSan, and image smoke/publication. This closes the pending Linux validation for M1.6. The maintainer's independent dev image pin from 20e609d was preserved during rebase. Kustomize renders the three expected dev resources successfully; no cluster execution is claimed.

M1.7 spatial queries (2026-09-19)

Release and MSVC ASan passed 5/5 CTests. Tests cover periodic-image reference comparison (32 seeds × 3 lifecycle/restore phases × 24 queries), hand-checked seams/corners/inclusive radii, empty/full capacity, invalid centers and maximum u32 radius. Queries leave snapshots unchanged; existing v1 goldens remain intact. Native commands use tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check and Release -Action format. Initial analysis flagged an unnecessary test-helper vector copy; the helper now takes a const reference. Final analysis, Release, ASan and formatting passed after applying the helper's required line wrapping. Linux validation is recorded below.

build/release/evo-spatial-benchmark.exe produced 72 measured records plus metadata in results/spatial-msvc1944.jsonl. One warmup and three samples per scenario; 1k/10k/100k entities, 128 queries, uniform/clustered positions, four radii. Timing includes result allocation, destruction and checksum work, excludes setup; no performance threshold. Desktop CPU load was not isolated. No cluster changes are required for this library increment.

Gitea run 1409 passed all four jobs for source 9b19509673d88683ee72e093e42534bfe151f928: Linux checks, ECS regression, separate TSan and image smoke/publication. All 72 spatial benchmark checksums and match counts agree with Windows. Raw Linux measurements are in results/spatial-gcc142.jsonl; 24 native scenarios also have identical checksums across their three repetitions. No deployment manifest, rules version, snapshot format or golden was changed.

M2.1 finite inventory ledger (2026-09-19)

Native Release, analysis and MSVC ASan each passed 6/6 CTests; formatting passed. Commands: tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check and Release -Action format. Initial clang-tidy rejected the Element enum's oversized base type; changed to u8 and reran all configurations successfully.

The new independent ledger test checks typed energy/mass separation, finite-source exhaustion, explicit heat transfer, intermediate overdraft, whole-transaction rollback, retry/duplicate action semantics, invalid reservoir IDs, maximum batch and reservoir counts, opening-total overflow and exact u64-limit transfers. 32 seeded runs × 31,250 transactions = one million transactions, each checked against independent reference balances, a replay branch and recomputed per-channel totals. This is transfer replay, not serialized recovery or an ecosystem run. Final ledger tests took 0.63s Release and 7.62s ASan locally; not controlled performance measurements. Existing particle goldens remain unchanged.

Allocation-failure injection, durable ledger recovery and cross-ledger ownership are not tested or implemented. Rates and conversions are absent, so rounding residuals cannot occur in this transfer-only scope.

Gitea run 1411, source 8fd3591b0e37aaecf00a6410a5926745a031cb6f, passed the Linux checks, ECS regression and separate TSan jobs. Each main Linux configuration passed 6/6 CTests. Ledger-suite times were 3.34s GCC Debug, 0.37s GCC Release, 2.46s Clang analysis and 8.78s Clang ASan/UBSan; these are CI test durations, not controlled performance benchmarks. The image smoke/publication job also passed, completing all four jobs. No deployment image pin was changed.

M2.2 finite producer process (2026-09-19)

Native Release, clang-tidy and MSVC ASan passed 7/7 CTests; format-check passed. Commands: tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check and Release -Action format. The producer suite compares hand-calculated ticks and 32 seeds × 1,000 ticks against a small-domain cumulative arithmetic reference, an independent replay branch and per-tick energy/element conservation audits. It covers invalid config/ticks with unchanged state, fractional capture/growth, efficiency endpoints, finite source exhaustion, all four limiting elements, zero capture, maximum u64 capture with 50% carried conversion, and maximum recipe and growth-cost multiplication. Particle goldens remain unchanged.

Allocation-failure injection and tick-counter exhaustion were not exercised; the latter has an explicit guard but no arbitrary-state restore API. Tests use in-memory copies, not serialized recovery. No photosynthetic chemistry, metabolic model, database integration, throughput guarantee or deployment is claimed. Gitea run 1413 passed all four jobs for source af293b8b50cbb506a49dfff10c6cc92307e16aca: Linux checks, ECS regression, separate TSan and runtime image smoke/publication. The initial push was delayed by a tool approval-review usage limit; retry succeeded after the maintainer resumed work. No deployment image pin was changed.

M2.3 shared feeding community (2026-09-20)

Native Release, clang-tidy and MSVC ASan each passed 8/8 CTests; formatting passed. Commands: tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check and Release -Action format. The new suite checks hand-calculated capture/feeding, producer-to-consumer and consumer-to-consumer predation, heat/waste outputs, terminal actor rejection, full-tick rollback, competing resource requests, duplicate sequence/consumer admission, per-channel intake caps, invalid actors, config/capacity limits and u64 assimilation endpoints. 32 seeds × 200 ticks compare reordered input batches, receipts, full balances and producer progress, with independently recomputed energy and elemental totals. Standalone producer and existing particle golden tests remain unchanged and passing.

No serialized recovery, allocation-failure injection, spatial reach/perception, physical injury, corpse handling or persistent-world integration was tested or implemented. These are co-located synthetic process fixtures.

Gitea run 1415 passed all four jobs for source b7dd466f57f385d6a8ffccfaa69f639077545587: Linux checks, ECS regression, separate TSan and image smoke/publication. No deployment image pin was changed.

M2.4 terminal inventory and decomposition (2026-09-20)

Native Release, analysis and MSVC ASan passed 8/8 CTests; formatting passed. Commands: tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check and Release -Action format. Tests preserve corpse inventories on death, suppress producer activity immediately, reject repeated/invalid terminal events and roll back death with later invalid feeding. Sixteen seeds × 256 ticks compare a closed-form depletion reference, reversed terminal-event order, per-tick budgets, inventories and conservation. Additional cases cover shared corpse/waste limits, mixed and zero channel caps, u64-maximum inventories, diagnostic rollback, and returned nutrients funding producer structure on the next tick.

The existing feeding, standalone producer, ledger and particle tests pass unchanged. This does not validate reaction chemistry, actual death causes, corpse scavenging, serialized recovery, allocation-failure injection or persistent history. There is no deployment change. Linux CI is pending source push.