9.0 KiB
M1.5 ECS evaluation protocol
Scope: synthetic composition storage, not biological rules. Public SimulationWorld, its golden fingerprint, rules version and snapshot format remain unchanged.
Workload and controls
Both adapters store a stable u64 ID and counter, with optional Motion and Capture counter components. These are workload labels, not organism classes or energy. Seed 123456 selects capabilities independently at 10% and 100% prevalence. Run 1000 and 10000 entities, 100 update passes, one warmup and seven measured samples. Alternate adapter order between samples to reduce simple ordering bias.
Both adapters use three ordered system passes per tick. One adds 3 to every common counter; the next adds Motion.rate then increments that rate by 1, and adds Capture.rate then increments it by 2. These independent integer operations have a closed-form reference. No wall clock enters state. After updates, remove every tenth original entity and create the same number with new monotonic IDs. Each adapter rejects stale IDs and preserves external identity despite storage relocation/native handle recycling.
Measure population creation (excluding the empty store constructor), updates, churn, canonical encoding and fresh restore separately. Export sorts by external ID and explicitly encodes fields little endian. Decode checks exact size, bounded count, ID ordering and capability flags. This in-memory experimental format is not production persistence. Compare full bytes after every correctness phase and after restore/continuation; fingerprints in reports are convenient checks, not substitutes for equality.
The row adapter extends the proof's sorted-row strategy with optional fields; it is not a benchmark of the unmodified production binary. EnTT uses sparse views and an external-ID-to-handle vector. That map's allocation is counted. Neither adapter reserves ahead to conceal growth; both start empty. No owning groups or per-tick sorting is applied because updates do not interact across entities. A production interaction scheduler must specify stable conflict order.
EnTT receives a small allocator-traits bridge backed by the counting resource. Direct polymorphic_allocator failed compilation because allocator-aware storage construction received a second injected allocator. The bridge preserves normal allocator-traits construction and tracks rebound allocations; it is benchmark instrumentation, not a new production allocator. Lifetime-balance tests apply to both adapters. Allocation failures abort the experiment at its host boundary; these internal fixture adapters do not claim transactional recovery.
External IDs are capped at 200000 in this experiment. The EnTT lookup vector retains space for retired IDs until store destruction. This is deliberately measured, not an acceptable unbounded mapping for a persistent region. A production EnTT adapter would need a separately evaluated bounded/sparse map.
Memory and evidence
Track allocation calls, live requested bytes and peak requested bytes for store containers through a dedicated std::pmr resource. Exclude canonical snapshot scratch/output, stack, allocator metadata and process RSS. Check all tracked bytes return to zero on adapter destruction. Do not call this total memory. Report construction/iteration/churn/export/restore timings for every sample, seed, count, density, compiler and build type; summarize medians and ranges. Measure cold/warm target build time and executable size separately. Do not compare wall times between Windows and Linux as if hosts/flags were controlled.
Acceptance and decision
Correctness gates: hand-checkable arithmetic fixture; density/seed ensembles; empty/singleton stores; stale/duplicate identities; churn; byte equality; malformed codec inputs; restore then continued equality; allocator balance. Timing has no CI threshold. Regression checks remain active under NDEBUG.
Prefer the simpler strategy when it satisfies measured needs. A candidate's query speed cannot excuse unstable IDs, hidden iteration-order semantics, unbounded adapter retention or more memory than the region budget supports. Document the measured recommendation in ADR-0005; any later production adapter needs a separate contract review and unchanged replay tests.
Reproduction
After initializing the compiler environment (Windows: run the existing check helper once in the same shell):
cmake -P tools/prepare-entt.cmake
cmake --preset release -B build/ecs -DEVO_BUILD_ECS_EXPERIMENT=ON
cmake --build build/ecs --target evo-ecs-experiment --parallel 2
ctest --test-dir build/ecs -R ecs-experiment --output-on-failure
./build/ecs/evo-ecs-experiment --measure
Preparation is the only network step. Repeating configure/build uses the verified local header offline. Use .exe on Windows. Never place EnTT headers in include/evo.
Windows observations, 2026-09-19
MSVC 19.44.35217.0 (/O2, NDEBUG), CMake 3.31.6, Ninja 1.12.1; Windows 10.0.26200 on Ryzen 7 9800X3D (8 cores / 16 logical processors). Seven samples after warmup; raw JSONL includes every sample and fingerprint. Measurements were recorded after compiler jobs completed. There is no controlled CPU isolation or portable performance threshold.
10000 entities, median elapsed milliseconds; update column is 100 ticks, churn removes/replaces 1000 individuals. Memory is live requested container bytes.
| Store / density | Create ms | Update ms | Churn ms | Export ms | Restore ms | Live bytes |
|---|---|---|---|---|---|---|
| Rows / 10% | 0.3366 | 0.7365 | 1.3883 | 0.3019 | 0.3056 | 485520 |
| EnTT4 / 10% | 0.4178 | 0.9868 | 0.0952 | 0.6435 | 0.5067 | 638608 |
| Rows / 100% | 0.2699 | 0.9237 | 1.3829 | 0.2873 | 0.2561 | 485520 |
| EnTT4 / 100% | 0.5777 | 7.0615 | 0.1399 | 0.6269 | 0.6842 | 866632 |
Observed update min–max: rows 10% 0.7312–0.8778 ms; EnTT 10% 0.9824–1.0396 ms; rows 100% 0.9203–0.9358 ms; EnTT 100% 7.0491–7.1502 ms. These are seven-sample ranges, not confidence intervals.
All paired samples have identical canonical bytes. EnTT wins this individual churn case by roughly 10–15x; rows win the measured simple updates/export and use fewer live bytes. EnTT's peak bytes can be lower than the row vector's temporary reallocation peak: at 10% density, 638608 versus 809200. Do not equate live bytes, peak requested bytes and process RSS.
The combined comparison executable is 99328 bytes; it contains both adapters and verification code, so this is not an isolated EnTT binary-size delta. Clean target build observed 2.7621 seconds, warm no-op 0.0475 seconds (configuration excluded; OS file cache not flushed). No claim of allocator-overhead/RSS measurement is made.
Recommendation: keep the current bounded row proof while designing the deterministic scheduler. Preserve a future adapter boundary. Revisit with real component sizes, dynamic capabilities, interacting systems, EnTT groups, batched structural changes and a bounded persistent-ID mapping before production adoption. Do not extrapolate these synthetic counts into organism/player capacity.
Linux observations, Gitea run 1404
Run 1404,
source bf7d60b7cd: GCC14 Release experiment,
Clang19 ASan/UBSan and clang-tidy contract runs all passed. Baseline regression,
TSan and image smoke/publish jobs also passed. The experiment itself remains
single-threaded; baseline TSan does not certify future ECS concurrency.
CMake 3.28.3, GCC 14.2.0, Clang 19.1.1; Linux x86-64 Docker runner. CPU model and cgroup limits were not captured for these samples, so they are exploratory within-runner comparisons, not a controlled hardware benchmark. All 56 sample fingerprints match Windows. Raw Linux JSONL.
10000 entities, medians in milliseconds; updates cover 100 ticks:
| Store / density | Create ms | Update ms | Churn ms | Export ms | Restore ms | Live bytes |
|---|---|---|---|---|---|---|
| Rows / 10% | 0.5208 | 7.6433 | 10.8217 | 1.1124 | 1.0239 | 655360 |
| EnTT4 / 10% | 1.0889 | 3.2116 | 0.1932 | 1.6101 | 1.4745 | 710392 |
| Rows / 100% | 0.8512 | 7.9037 | 10.8592 | 1.1100 | 1.0152 | 655360 |
| EnTT4 / 100% | 1.4721 | 18.9315 | 0.2737 | 1.7558 | 1.9766 | 968672 |
Update min-max: rows 10% 7.5747-7.7266 ms; EnTT 10% 3.1476-3.2407 ms; rows 100% 7.3748-9.3729 ms; EnTT 100% 18.7963-19.0924 ms. Combined executable size: 180256 bytes (unstripped); this is not a per-adapter size delta. Allocation instrumentation contributes overhead to creation/churn/restore timings.
Unlike Windows, EnTT sparse updates win on this Linux run (~2.4x), as well as churn (~40-56x). Rows retain cheaper dense updates, export and lower live requested bytes. Different vector growth policies also change live/peak allocation totals. These results strengthen the case for keeping the production choice open: selecting a library from one compiler's synthetic loop would be unjustified. The scoped ADR-0005 recommendation still retains rows for the bounded proof and scheduler contract, with production adoption gated on realistic workloads.