Files
aevolution/docs/engineering/VALIDATION.md
T
dingoandClaude Opus 5 22fb904c3e Reject CLI options that do not belong to the selected mode
Mode validation ran in one direction only. Under --ecosystem, --ticks was
accepted and then ignored; without it, --episodes and --tick-limit were
accepted and then ignored. Either way the process exited zero while
running the other workload, so a measurement script could report success
for a run it never performed.

Record which options were actually supplied and reject any that belong to
the other mode, in both directions. An option that is accepted and then
ignored is now a parse error rather than a silent substitution.

Six CLI cases cover this. cli-ecosystem-rejects-ticks and
cli-episodes-require-ecosystem are the genuine regressions, reproducing
the two reported invocations; the entities, save and verify-replay cases
were already rejected and add coverage. The remaining shared options were
audited: --seed and --output are valid in both modes, and --load
conflicts were already enforced.

Native Release, clang-tidy analysis, ASan and format checks pass 16/16.
The ecosystem report is unchanged: 32 episodes still fingerprint
2cd4a15f9fbe9949, and golden d61ad5bf0c928c76 is intact.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 18:36:46 -04:00

365 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Validation report
Recorded 2026-09-19. Latest verified Linux delivery: **run 1402 passed** on
`4c2cc04ac47d37c6191a9ef02d4bbba716a930dc`. Native checks apply to base commit `8ab48d3` plus the
foundation-delivery working changes committed with this report. Simulation rules
and snapshot format remain version 1; the only C++ change is benchmark metadata.
Results below distinguish executed checks from pending Linux/cluster gates.
## Tools and host
- Windows NT 10.0.26200, AMD Ryzen 7 9800X3D, 8 cores / 16 logical processors.
- MSVC 19.44.35217.0, CMake 3.31.6-msvc6, Ninja 1.12.1.
- clang-format and clang-tidy 19.1.5 (Visual Studio LLVM tools).
- kubectl 1.31.4 / Kustomize 5.4.2 for offline manifest rendering.
## Executed native checks
| Command | Observed result |
|---|---|
| `./tools/windows-check.ps1 -Preset debug -Action check` | Configure/build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset release -Action check` | Configure/build passed; 5/5 CTests passed under NDEBUG |
| `./tools/windows-check.ps1 -Preset analysis -Action check` | clang-tidy/warnings-as-errors build passed; 5/5 CTests passed |
| `./tools/windows-check.ps1 -Preset asan -Action check` | MSVC AddressSanitizer build passed; 5/5 CTests passed; this is NOT UBSan |
| `./tools/windows-check.ps1 -Preset debug -Action format` | Passed with clang-format 19; repeated twice in one shell after helper fix |
| `./build/release/evo-sim --seed 123456 --entities 8 --ticks 1000000 --verify-replay` | Exit 0; replay_verified=true; hash fab22c2e8ae1c379 |
| `./build/release/evo-benchmark` | Exit 0; MSVC 1944 Release, seed 123456, 10000 entities, 1000 ticks, 0.0210666 seconds, hash 3bad6db1612cbd1 |
| Git Bash `bash -n tools/ci/linux-check.sh` and `bash -n tools/ci/publish-image.sh` | Syntax checks passed; no Linux execution implied |
| `kubectl kustomize deploy/kubernetes/dev` | Rendered Job, ServiceAccount and NetworkPolicy in evolution-dev; final render pins the verified run-1402 image |
CTest covers the independently frozen 1000-tick fingerprint d61ad5bf0c928c76,
seeded replay properties, malformed/semantic snapshot rejection, entity lifecycle,
capacity/overflow, telemetry and CLI file continuation/alias protection. The
expected-failure harness confirms tests remain active in Release.
The benchmark is a single uncontrolled wall-clock observation, not a performance
threshold or an MMO capacity estimate. Its initial compiler output accidentally
used hexadecimal; the stream now returns to decimal and reports MSVC 1944.
## Gitea evidence
The maintainer's [run 1399](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1399)
on `8ab48d3a5639121693f5881b600482e870b49f05` completed successfully. API job records
confirm verify (configure/build/CTest) and publish (Docker build/push) succeeded
on runner theraborn-runners-runner-0, label ubuntu-22.04. This is evidence for the
original workflow, not the expanded sanitizer/analysis jobs added in this change.
Exact Linux tool versions and expanded-run results appear below; a runner label
is not a compiler version.
## Limitations and next gates
- Local Docker Desktop has no running Linux engine. Container checks must run on
the configured Gitea runner; no local container success is claimed.
- Expanded Linux GCC14/Clang19, ASan/UBSan, TSan and libFuzzer evidence is recorded
below. TSan currently validates a sequential kernel, not future concurrency.
- Sandbox process restrictions stalled native Ninja and blocked Git Bash signal
pipes. Those attempts were stopped; checks reran successfully outside sandbox.
- Repeated VsDevCmd initialization grew PATH beyond cmd's limit. The helper now
reuses the matching x64 environment and deduplicates PATH; repeated checks pass.
- The maintainer reports Argo setup, but this workspace has no Kubernetes context.
No dev Job completion or cluster admission result is recorded yet.
- No biology, network/load, durable server recovery or backup restore exists.
Those remain M2, M7/M8 and later gates, not failing or passing foundation tests.
Update this report with run URL, tested commit, image digest and Job result when
those checks execute. Never overwrite a golden merely to make CI pass.
## Expanded Linux run 1400
[Run 1400](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1400),
commit 9e412736d91ad103b7a951f99b811c629c5ac61e: GCC14 Debug and Release, Clang19
analysis and ASan/UBSan each passed 5/5 tests. Formatting passed. The seeded fuzz
smoke completed 10000 cases with no reported failure. CMake 3.28.3, GCC 14.2.0,
Clang/format/tidy 19.1.1, Linux 6.8.0-110-generic x86-64. Million-tick hash matched
Windows (fab22c2e8ae1c379); benchmark hash matched too (3bad6db1612cbd1), with
0.0674131 seconds observed for 10000 entities * 1000 ticks on the CI host.
TSan compiled but all executions failed before main: its runtime could not call
personality(ADDR_NO_RANDOMIZE) inside the Docker build sandbox. The image job was
skipped, as required. This is not recorded as a TSan pass. The follow-up isolates
execution in a container with the narrowly constrained profile from ADR-0020;
its result must be recorded after execution.
## Successful delivery run 1402
[Run 1402](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1402),
commit 4c2cc04ac47d37c6191a9ef02d4bbba716a930dc: every job succeeded. The general
checks reused Docker's valid cache for unchanged checked sources; their full
execution also passed in run 1401. TSan executed all five CTests successfully in
2.71 seconds using the scoped profile and writable tmpfs outputs. Run 1401 had
first exposed a non-root tmpfs permission error; setting explicit mode=1777 on
these isolated scratch mounts fixed it. No test was disabled or ignored.
The runtime image built, ran the restricted replay smoke test and was pushed to:
`registry.aeternalisgames.com/aeternalis/aevolution:sha-4c2cc04ac47d37c6191a9ef02d4bbba716a930dc-run-6-1`
Registry digest:
`sha256:8fce2b462c9907f38d3fb832afeb70d0bff068dd832b2171e861186dbcb8e6b9`.
The dev manifest pins this digest. This proves build/test/image delivery; Argo
application mapping and Kubernetes Job completion remain unverified locally.
## M1.5 native ECS experiment
Opt-in target evo-ecs-experiment (EnTT4 pinned in ADR-0021), based on 1c6e6e1 plus
this experiment's changes. MSVC Release contract checks passed, including 72
seed/density/count combinations, hand-calculated arithmetic, canonical equality,
restore continuation, retired-ID rejection, malformed snapshot checks and
allocator balance. The final clang-tidy build/test passed after tightening the
process-boundary diagnostic catch. MSVC ASan passed on the final source.
Baseline Release CTest remained 5/5
and format-check included the experiment. No simulation rule/golden changed.
Commands: configure release/analysis/asan with `-B build/ecs[-analysis|-asan]
-DEVO_BUILD_ECS_EXPERIMENT=ON`, build target evo-ecs-experiment, then
`ctest --test-dir <directory> -R ecs-experiment --output-on-failure`.
Seven measured samples plus warmup are retained in results/ecs-msvc1944.jsonl;
protocol, build observations and scope limits are in ECS_EVALUATION.md. The
Linux experiment CI job is pending its first pushed run; no Linux experiment
success is claimed yet.
## M1.5 Linux evidence
[Run 1404](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1404),
source bf7d60b7cd174736d38aea42616064e55e0c5f05: every job passed. The new ci-ecs
stage explicitly prepared the pinned header/license, then built and ran the
experiment under GCC14 Release, Clang19 ASan/UBSan and Clang19 analysis. All three
contract invocations passed. Raw measured GCC samples are retained at
results/ecs-gcc142.jsonl; all 56 fingerprints match corresponding Windows samples.
No production rule or snapshot-format change occurred. This supersedes the
pending Linux experiment note above. The final follow-up edits only record
results and decision scope; production ECS adoption remains deferred.
## M1.6 sequential scheduler (2026-09-19)
Native commands: `tools/windows-check.ps1 -Preset release -Action configure`,
clang-format on changed C++ files, `cmake --build --preset release`,
`ctest --preset release`; then `tools/windows-check.ps1 -Preset analysis -Action check`,
`tools/windows-check.ps1 -Preset asan -Action check` and Release `-Action format`.
All three configurations passed 5/5 CTests; clang-tidy and format-check passed.
Tests execute under NDEBUG and cover six batch permutations across 32 seeds,
an independent direct-lifecycle reference, ordered receipts, restore continuation,
rejected conflicts and malformed admission, the 1,024-command boundary, full entity
capacity and tick/ID exhaustion. Golden `d61ad5bf0c928c76` remains unchanged.
The copy-and-commit implementation allocates before publishing world state and
statically checks noexcept publication/receipt moves. Allocation-failure injection
was not run. No scheduler throughput claim, concurrency validation, durable input
journal or Kubernetes execution is implied.
[Gitea run 1407](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1407)
passed all four jobs for source `b620edfd6f8097172acf055402a24620cb078358`:
Linux checks, the ECS experiment, separate TSan, and image smoke/publication.
This closes the pending Linux validation for M1.6. The maintainer's independent
dev image pin from `20e609d` was preserved during rebase. Kustomize renders the
three expected dev resources successfully; no cluster execution is claimed.
## M1.7 spatial queries (2026-09-19)
Release and MSVC ASan passed 5/5 CTests. Tests cover periodic-image reference
comparison (32 seeds × 3 lifecycle/restore phases × 24 queries), hand-checked
seams/corners/inclusive radii, empty/full capacity, invalid centers and maximum
u32 radius. Queries leave snapshots unchanged; existing v1 goldens remain intact.
Native commands use `tools/windows-check.ps1 -Preset <release|analysis|asan>
-Action check` and Release `-Action format`. Initial analysis flagged an unnecessary
test-helper vector copy; the helper now takes a const reference. Final analysis,
Release, ASan and formatting passed after applying the helper's required line
wrapping. Linux validation is recorded below.
`build/release/evo-spatial-benchmark.exe` produced 72 measured records plus
metadata in results/spatial-msvc1944.jsonl. One warmup and three samples per
scenario; 1k/10k/100k entities, 128 queries, uniform/clustered positions, four radii.
Timing includes result allocation, destruction and checksum work, excludes setup;
no performance threshold. Desktop CPU load was not isolated. No cluster changes
are required for this library increment.
[Gitea run 1409](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1409)
passed all four jobs for source `9b19509673d88683ee72e093e42534bfe151f928`:
Linux checks, ECS regression, separate TSan and image smoke/publication.
All 72 spatial benchmark checksums and match counts agree with Windows.
Raw Linux measurements are in results/spatial-gcc142.jsonl; 24 native scenarios
also have identical checksums across their three repetitions. No deployment
manifest, rules version, snapshot format or golden was changed.
## M2.1 finite inventory ledger (2026-09-19)
Native Release, analysis and MSVC ASan each passed 6/6 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. Initial clang-tidy rejected the Element enum's
oversized base type; changed to u8 and reran all configurations successfully.
The new independent ledger test checks typed energy/mass separation, finite-source
exhaustion, explicit heat transfer, intermediate overdraft, whole-transaction
rollback, retry/duplicate action semantics, invalid reservoir IDs, maximum batch
and reservoir counts, opening-total overflow and exact u64-limit transfers.
32 seeded runs × 31,250 transactions = one million transactions, each checked
against independent reference balances, a replay branch and recomputed per-channel
totals. This is transfer replay, not serialized recovery or an ecosystem run.
Final ledger tests took 0.63s Release and 7.62s ASan locally; not controlled
performance measurements. Existing particle goldens remain unchanged.
Allocation-failure injection, durable ledger recovery and cross-ledger ownership
are not tested or implemented. Rates and conversions are absent, so rounding
residuals cannot occur in this transfer-only scope.
[Gitea run 1411](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1411),
source `8fd3591b0e37aaecf00a6410a5926745a031cb6f`, passed
the Linux checks, ECS regression and separate TSan jobs. Each main Linux
configuration passed 6/6 CTests. Ledger-suite times were 3.34s GCC Debug, 0.37s GCC
Release, 2.46s Clang analysis and 8.78s Clang ASan/UBSan; these are CI test
durations, not controlled performance benchmarks. The image smoke/publication
job also passed, completing all four jobs. No deployment image pin was changed.
## M2.2 finite producer process (2026-09-19)
Native Release, clang-tidy and MSVC ASan passed 7/7 CTests; format-check passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. The producer suite compares hand-calculated ticks
and 32 seeds × 1,000 ticks against a small-domain cumulative arithmetic reference,
an independent replay branch and per-tick energy/element conservation audits.
It covers invalid config/ticks with unchanged state, fractional capture/growth,
efficiency endpoints, finite source exhaustion, all four limiting elements,
zero capture, maximum u64 capture with 50% carried conversion, and maximum recipe
and growth-cost multiplication. Particle goldens remain unchanged.
Allocation-failure injection and tick-counter exhaustion were not exercised;
the latter has an explicit guard but no arbitrary-state restore API. Tests use
in-memory copies, not serialized recovery. No photosynthetic chemistry, metabolic
model, database integration, throughput guarantee or deployment is claimed.
[Gitea run 1413](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1413)
passed all four jobs for source `af293b8b50cbb506a49dfff10c6cc92307e16aca`:
Linux checks, ECS regression, separate TSan and runtime image smoke/publication.
The initial push was delayed by a tool approval-review usage limit; retry succeeded
after the maintainer resumed work. No deployment image pin was changed.
## M2.3 shared feeding community (2026-09-20)
Native Release, clang-tidy and MSVC ASan each passed 8/8 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. The new suite checks hand-calculated capture/feeding,
producer-to-consumer and consumer-to-consumer predation, heat/waste outputs,
terminal actor rejection, full-tick rollback, competing resource requests,
duplicate sequence/consumer admission, per-channel intake caps, invalid actors,
config/capacity limits and u64 assimilation endpoints. 32 seeds × 200 ticks compare
reordered input batches, receipts, full balances and producer progress, with
independently recomputed energy and elemental totals. Standalone producer and
existing particle golden tests remain unchanged and passing.
No serialized recovery, allocation-failure injection, spatial reach/perception,
physical injury, corpse handling or persistent-world integration was tested or
implemented. These are co-located synthetic process fixtures.
[Gitea run 1415](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1415)
passed all four jobs for source `b7dd466f57f385d6a8ffccfaa69f639077545587`:
Linux checks, ECS regression, separate TSan and image smoke/publication.
No deployment image pin was changed.
## M2.4 terminal inventory and decomposition (2026-09-20)
Native Release, analysis and MSVC ASan passed 8/8 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and Release `-Action format`. Tests preserve corpse inventories on death, suppress
producer activity immediately, reject repeated/invalid terminal events and roll
back death with later invalid feeding. Sixteen seeds × 256 ticks compare a
closed-form depletion reference, reversed terminal-event order, per-tick budgets,
inventories and conservation. Additional cases cover shared corpse/waste limits,
mixed and zero channel caps, u64-maximum inventories, diagnostic rollback, and
returned nutrients funding producer structure on the next tick.
The existing feeding, standalone producer, ledger and particle tests pass unchanged.
This does not validate reaction chemistry, actual death causes, corpse scavenging,
serialized recovery, allocation-failure injection or persistent history. There is
no deployment change.
[Gitea run 1417](https://git.aeternalisgames.com/dingo/AEvolution/actions/runs/1417)
passed all four jobs for source `e17a9a7279d6bde736261c0c33abe46108e1f622`:
Linux checks, ECS regression, separate TSan and image smoke/publication.
## M2.5 funded reproduction (2026-09-21)
Native Release, analysis and MSVC ASan passed 8/8 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and debug `-Action format`. The first format run rejected three wrapped test
declarations; clang-format 19.1.5 rewrote them and Release was rebuilt and rerun.
Tests cover hand-calculated endowment and construction overhead, the endowment
floor, per-channel insufficiency with abundant energy, zero and oversized
investments, reserved slots rejected as feeding targets, corpses and ancestry
subjects, the producer rejected as a parent, terminated and unknown parents,
zero/duplicate sequences, two intents naming one parent, oversized batches,
capacity exhaustion rejecting the whole tick, absent identity reuse after a child
dies and decomposes, ancestry surviving both deaths, and construction endpoints of
0 and 1000 including a paid-for zero-endowment child. 32 seeds × 4 ticks × 2 births
compare reversed intent order against balances, producer progress, birth receipts
and independently recomputed conservation. A u64-maximum investment endows exactly
and conserves. Default zero capacity leaves the M2.4 reservoir layout, feeding,
producer, ledger and particle tests unchanged; golden `d61ad5bf0c928c76` is intact.
Review of `c84a701` found one defect (2026-09-22): births validated each parent
against the progressively modified candidate, so a child allocated earlier in the
same batch was already active and could fund a second birth. Two generations could
therefore appear in one tick, which ADR-0028 forbids. Eligibility is now fixed on
entry to the reproduction phase, after terminal facts and feeding, and checked for
every intent before any birth is applied. The contract itself was already correct
and did not change. The regression test admits one intent whose parent is created
by the preceding intent and requires `unborn_actor` with the entire tick unchanged;
it was confirmed to fail against the unguarded implementation and to pass with the
guard. The same pair succeeds across two ticks, recording the grandchild ancestry.
All four native configurations were rerun after the fix.
Allocation-failure injection, serialized recovery of states/parents/origins and
long-run ecosystem behavior were not exercised. This fixture defines no
reproductive strategy, fecundity, mutation, gestation, spatial placement or account
ownership, and no deployment manifest or image pin changed. Linux validation is
pending its first pushed run; no Gitea result is claimed for this increment yet.
## M2.6 measured ecosystem episodes (2026-09-22)
Native Release, analysis and MSVC ASan each passed 11/11 CTests; formatting passed.
Commands: `tools/windows-check.ps1 -Preset <release|analysis|asan> -Action check`
and debug `-Action format`. Formatting rejected 25 wrapped declarations on the
first run; clang-format 19.1.5 rewrote them and Release was rebuilt and rerun.
Three CTest cases are new: `ecosystem-contracts`, `cli-ecosystem` pinned to
fingerprint `2cd4a15f9fbe9949`, and `cli-ecosystem-conflict`, which requires the
CLI to reject `--ecosystem` combined with proof-run options.
The suite checks a solitary founder that only ages, a hand-checked predation
retirement, births exhausting exactly the reserved slots, tick-limit termination
with survivors, configuration rejection, and two episodes interleaved without
observing each other. Across 64 seeds it steps tick by tick, requiring that the
driver never provokes a rejected batch and that conserved totals are identical at
every tick, then replays each seed and requires an identical report. That ensemble
advanced 22,804 ticks with 879 births, 730 predation deaths, 277 lifespan deaths
and extinction in all 64 episodes, episode length 73 to 463 ticks.
`build/release/evo-sim.exe --ecosystem --seed 0 --episodes 3000` advanced
**1,093,007 ticks** in 1.34 s, with 41,138 births, 33,984 predation deaths, 13,154
lifespan deaths, extinction in all 3,000 episodes, episode length 50 to 497 ticks
and fingerprint `130d7725c0dc8b16`. Deaths equal founders plus births exactly
(6,000 + 41,138 = 47,138), so every process died once and no producer was grazed
out: the producer always retains captured energy, so its whole inventory never
reaches zero. The elapsed time is one uncontrolled wall-clock observation, not a
performance threshold.
Review of `eaf58f2` found one defect (2026-09-22): the CLI validated mode options
in one direction only. `--ticks` was accepted and ignored under `--ecosystem`, and
`--episodes`/`--tick-limit` were accepted and ignored without it, so a measurement
script could exit zero while running the other workload. The parser now records
which options were supplied and rejects any that do not belong to the selected
mode, in both directions. Six CLI cases cover this; `cli-ecosystem-rejects-ticks`
and `cli-episodes-require-ecosystem` are the two genuine regressions, while the
entities, save and verify-replay cases were already rejected and add coverage.
The remaining shared options were audited: `--seed` and `--output` are valid in
both modes, and `--load` conflicts were already enforced. CTest is now 16 cases,
all four native configurations were rerun, and `2cd4a15f9fbe9949` is unchanged.
This measures arithmetic conservation and turnover, not ecological plausibility.
Strategy constants, lifespans and the targeting rule are chosen inputs. There is
no maintenance cost, starvation rule, perception, spatial gate, genome or
heritable variation, so lineages differ only by founder strategy. Episode state is
not serialized, allocation failure is not injected, and no population abstraction
exists. Golden `d61ad5bf0c928c76` and every earlier suite are unchanged. No
deployment manifest or image pin changed. Linux validation is pending its first
pushed run; no Gitea result or cross-platform fingerprint agreement is claimed yet.