Files
aevolution/docs/engineering/TEST_STRATEGY.md
T
dingo da39fc8095
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m33s
Foundation / checks (network) (push) Successful in 21m15s
Foundation / checks (client) (push) Successful in 22m8s
Foundation / checks (tsan) (push) Successful in 25m2s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 38m24s
Foundation / checks (postgres) (push) Canceled after 38m24s
M12.6: add bounded regional cellular census and lineage queries
2026-10-18 13:57:52 -04:00

141 lines
10 KiB
Markdown

# Validation strategy
M12.6a's regional census tests compare per-category counts and body/reserve
inventories with direct individual observations. Cover active/lingering/suspended
controlled lives, real funded wildlife, exact cohort conversion/restoration,
permanent death and empty owners, all 256 resident slots, mutation-current versus
immutable birth genomes, zero-carrier ancestry and sparse IDs whose numeric order
differs from ancestry. Unknown anchor/scope and older biological rules must fail.
Existing migration tests additionally verify nonresident historical counts,
per-member migrant inventory, repeated return trips and imported cohort tags.
Full native regression and relevant ASan/UBSan/Clang checks qualify this portable
projection; durable commit-bound history, complete world coverage and client
queries remain separate M12.6/12.7 gates. No terminal-extinction outcome exists.
M12.4f's checkpoint contract exercises explicit portable bundle-9 to 10 upgrades
for raw owners and a real nonempty cohort with retained policy/sequence. It
requires complete candidate equality, canonical restore, continued batch-2
biology and bundle-preserving representation changes. Negative cases cover
old/new payload confusion, malformed source/lease/digest bindings, exhaustion,
repeated upgrade and new genesis. The private PostgreSQL population suite must
continue refusing these portable candidates through schema 45 and existing
prepared/generic publication paths without advancing the head. Durable admission
and upgrade crash/retry recovery remain separate gates.
M12.4g's private PostgreSQL suite upgrades raw/profile-5 bundle-9 sources using
schema 46. It rehearses raw upgrade/trace-3 continuation and rollback, retains a
nonempty cohort, refuses pending-interval upgrades and wrong publication paths,
and recovers after a pre-transaction SIGKILL and an ambiguous exact retry. Changed
requests and stale writers fail. Missing/corrupt objects quarantine; a canonical
but unequal cohort with matching biological totals fails full upgrade replay.
Batch-1/2 cross-binding is refused in both directions. The process-loss and
host-recreation/materialization matrix also runs with batch 2. Full historical
migration/backup/DR/permission regression remains required for schema admission.
M12.4h's private freeze matrix uses funded player-derived wildlife and both
bundle-10 representations. It requires unchanged source/destination heads and
stock, exact request retries, conflicting/new-work refusal, pending-choice/
interval guards, new-epoch recovery, explicit abort without reopening, and real
SIGKILL before/after preparation. Unknown preparation/abort replies pause the
owning host through exact reconciliation; drain/recreation retains the freeze.
Missing and same-length corrupt capsules fail, and canonical false material
fails full biological replay. SQL checksum rejection is checked independently of
the C++ decoder. The metric has one fixed unlabelled series and remains bounded.
These are preparation/abort checks, not proof of destination activation or atomic
paired publication; full historical persistence regression is also required.
M12.4i's private staging matrix covers both bundle-10 representations and actual
player-origin wildlife. It requires unchanged biological heads, complete ghost
replay, exact/conflicting retries, refusal of moved destination heads and pending
intervals, publication/profile/life-reservation guards and exclusive endpoint
roles. Stale destination epochs fail; replacement retains the read-only ghost.
Missing source capsules, missing/same-length corrupt ghosts and valid but false
ghost objects fail recovery. Real lock-timeout ambiguity, SIGKILL before/after
stage commit and independent-connection stage/abort races preserve one outcome.
The owning destination host retains exact uncertain work across restart and
resumes only its original biology after confirmed source abort. Full schema,
permission, backup/DR and historical regression remain required. These tests do
not qualify paired commit, destination activation or linked ecological movement.
M12.4j's private paired-publication matrix uses real wildlife, both representations
and return transfers. It requires complete source/destination equality with the
portable pair, one shared receipt and unchanged completed ticks. An injected
second-insert failure must roll back both heads, receipt and pointer settlement.
Wrong publishers, unstaged/aborted requests, stale fences and conflicting exact
retries fail. Tests cover genuine lock-timeout ambiguity, SIGKILL before/after
commit, fresh-epoch exact retry, independent-connection commit/abort races and
receipt recovery that rejects missing counterpart capsules and valid but false
biological objects. Owning hosts must reload both actual heads before readiness,
retain complete recovery after replacement and continue biology afterwards.
Independent network partitions, linked ecology and actual elapsed-day stability
remain separate evidence; full schema/permission/backup/DR regression is required.
M12.4k adds an explicit disposable loopback TLS mode and test-owned opaque relays.
Both representations exercise request loss, committed receipt loss, wrong-host
rejection, lost preparation/staging/abort replies, either owner's actual lease
expiry, an unchanged healthy-peer epoch, replacement and healed stale-owner
fencing. Independent Unix-socket observations prove the database outcome; relay
byte counters alone are insufficient. Full and sanitizer CI paths run this mode
in a separate cluster. ASan/UBSan and TSan exercise the actual relay/host matrix.
Multi-host/kernel-network failures, object-store outages and orchestrator drills
remain separate; see [the scope](M12_MIGRATION_TRANSPORT_FAULTS.md).
M12.4l adds source/destination capsule and result directory-sync failures, plus
actual object read-permission denial after paired commit. Both representations
and transfer directions must preserve the independently observed heads and
pending roles before publication. Current-result I/O failure pauses both hosts
for complete retry under the same epochs; unavailable counterpart replay evidence
quarantines and restoring access cannot clear quarantine. Explicit lease retirement
and replacement must recover exact bytes under newer epochs, and exact retry must
return the original receipt. Runtime
checks require real I/O errors, without changing immutable bytes or skipping
under elevated filesystem privileges. See [storage scope](M12_MIGRATION_STORAGE_FAULTS.md).
Tests defend contracts, not directory coverage. All checks remain active in
Release builds. Current proof has no actual network, biology or distributed
recovery, so those categories below are milestone gates rather than passing
test labels attached to empty targets.
| Category | Foundation evidence | Future gate |
|---|---|---|
| Unit | PRNG reference vectors, entity lifecycle, capacity/error boundaries | Genome development and energetic transfer rules |
| Deterministic regression | Frozen canonical fingerprint plus independent reference calculation; uninterrupted vs restored execution | Versioned ecosystem baseline ensembles |
| Property-style | Many explicit seeds and tick split points; coordinate and ID invariants | Conservation, nonnegative mass/population and phenotype prerequisites |
| Serialization | Exact byte round-trip, malformed lengths/versions/checksum/IDs/coordinates; validate before publication | Compatible migrations, verified object archive and replay range |
| Integration | CLI seed/ticks and replay verification; file save/load comparison | Durable death/reproduction acknowledgment and worker process kill |
| Fuzz | Optional libFuzzer snapshot decoder with bounded input and sanitizer instrumentation | Authenticated protocol/parser and migration fuzzing |
| Static/sanitizers | clang-format/tidy; ASan/UBSan and separate TSan configurations | TSan on real scheduler and I/O lifecycle |
| Benchmark/load | Seed/count/tick/time/compiler report for kernel; no noisy portable time threshold | 20+ clients, region load, interest bandwidth, recovery backlog |
| Biological regression | Absent until M2; do not call synthetic particles an ecosystem | Million-tick balances, predator/prey cycles, starvation, diversity and extinction |
| Disaster recovery | Absent until persistent services exist | Off-host verified backup restore; database/object consistency; old owner fencing |
Reference fixture, rules/format 1: seed 123456, create 8 particles at (0,0), then
1,000 ticks yields canonical payload FNV-1a `d61ad5bf0c928c76`. Expected positions
by ID: (49,10), (985,993), (39,33), (1008,0), (16,1), (7,3), (1004,1007),
(995,15). The initial value was computed independently in JavaScript with
explicit wrapping BigInt arithmetic, not copied from the C++ test result.
SplitMix64 seed-zero vectors begin `e220a8397b1dcdaf`, `6e789e6aa1b965f4`,
`06c45d188009454f`, `f88bb8a8724c81ec`.
Changing a golden requires a stated rule/format change, compatibility decision,
independent review and updated documentation. A new compiler or optimization
mode must reproduce existing integer fixtures. Floating-point biology later
needs error envelopes and ensemble comparisons in addition to exact fixtures.
Unexecuted configurations are not validation. Record exact tools, commands,
results and limitations in VALIDATION.md at the end of each foundation change.
The cohort qualification runs its complete matrix under sanitizers; its wall
timeout is a bounded CI execution guard, not a portable performance claim.
Budgets are 180 s normally, 600 s with ASan/UBSan and 900 s with TSan. The larger
migration contract gets 180/360/600 s respectively. Cases, seeds, checks and
instrumentation remain identical. The checks job allows 90 minutes for the full
cold/shared-runner build, instrumented suites and remaining fuzz/benchmark steps.
Population and migration restart scripts honor the runtime `EVO_TEST_TMPDIR`
override and create distinct random fixture subdirectories there. Without the
override they retain the configured build-directory output paths. The non-root,
networkless, read-only TSan container sets the override to its existing bounded
`/tmp` tmpfs; no source/build mount is made writable and no sandbox permission is
broadened. Fixture path randomness has no biological or test-result meaning.