Files
aevolution/docs/adr/0040-resolution-equivalence.md
T
2026-10-08 12:08:53 -04:00

12 KiB

[Reviewed]

ADR-0040: Measuring what aggregation costs, and refusing to call it equivalence

Status

Adopted for M4.5, which closes M4. It adds no library code: the whole increment is a measurement, and needing to change a contract in order to measure it would have been a finding rather than a step. No M1 to M4.4 contract changes, and ADR-0038's refusal to choose an ecology is unchanged: the model compared here lives in the test fixture as a declared measurement input.

Context

Everything M4 built proves its arithmetic closes. None of it says whether a cohort behaves like the individuals it replaced, and ECOLOGY_MODEL is blunt about the difference: passing conservation alone is insufficient ecological evidence, exact seeded repeatability does not imply exact fidelity to individual ecology, and what is wanted is matched ensembles with published error envelopes.

The named risk is claiming identical trajectories at different resolution. It is a risk precisely because the tempting experiment supports the claim: pick a fixture where averaging happens not to bite, observe agreement, and report equivalence. The honest experiment has to be built so that it can fail, and has to report the size of the disagreement rather than its absence.

There is a second, quieter problem. Comparing two implementations of "the same" model measures the difference between the implementations as well as the difference between the resolutions, and those are not separable after the fact. A divergence of five per cent could be aggregation error or could be a transcription mistake in one of the two runs, and no amount of staring at the number distinguishes them.

Decision

The rule is written once and applied at both resolutions. The individual run applies it to one member at a time; the aggregate run applies it to a class average and multiplies by the count. Nothing else differs, and in particular the aggregate run sub-steps internally at one tick so that interval length is not a second source of error confounded with the first.

Anything that decides between members has to be part of the rule, not a consequence of how each run happens to iterate. Breeding is the case that matters: when there are fewer free slots than eligible parents, the allocation order decides which parents reproduce, and two runs ordering differently diverge for a reason that has nothing to do with averaging. The rule therefore states the order, oldest first, which both resolutions can honour because classes partition ages. Ordering inside a class is the one thing only the individual run can see, and that is a resolution effect rather than an implementation difference.

Control runs establish that the two are the same rule before any divergence is attributed to resolution. When every class holds exactly one member, a class average is that member, so the aggregate run must reproduce the individual run exactly, tick by tick. There are two of them, because one is not enough. The first exercises ordinary growth. The second sits at the capacity boundary, where eligible parents outnumber free slots, because that is the only place the allocation order is observable at all; a fixture that never contends never checks it. Both compare the whole distribution of members rather than censuses: whoever breeds hands the same investment to a child either way, so every total agrees while the distribution does not, and the disagreement only reaches a census dozens of ticks later.

These controls are the experiment's instrument check. Without them a measured divergence means nothing, and if either fails, every other number in this ADR is void.

Divergence is then produced deliberately by coarsening the classes, which is the resolution axis. Width is varied and the envelope is reported per width, so the relationship between declared resolution and error is visible rather than a single number standing in for it.

The scarce quantity is a single axis. A shared environment supplies a fixed amount of energy per tick divided equally among the living, and everything else is abundant, so a member that cannot pay its upkeep starves. That makes the mechanism attributable: at individual resolution the weakest starve first, and at class resolution the average pays for everyone until it cannot, at which point the class goes together. This is the "population averages hide viable outliers" risk of ADR-0036 arriving as a measurable quantity rather than as a warning.

The tests assert an envelope, and they also assert that the trajectories differ. Asserting the difference is the part that protects the claim: a later change that quietly made the runs agree would mean either that the fixture stopped exercising aggregation or that the comparison stopped comparing, and both should fail loudly rather than look like progress. Nothing in this ADR asserts equality outside the one-member-per-class control, and the word equivalence is not used for what is measured.

The envelope is published as a results file alongside the existing benchmark series, so the numbers are reviewable and comparable across platforms rather than living only in a commit message.

What the measurement found

The controls hold: with one member per class the two runs agree exactly, tick by tick, in every channel and across the whole distribution of members, including at the capacity boundary where twelve eligible parents contend for one free slot. The instrument was verified rather than assumed, by three separate mutations. A one-unit drift introduced into the aggregate upkeep is caught. Allocating contested slots in member insertion order, which is what the individual run originally did, is caught. Walking the classes in stored order, youngest first, which is what the aggregate run originally did, is caught. Since either allocation order alone fails a control, the divergence reported below is attributable to averaging rather than to the two runs disagreeing about who breeds.

Coarsening then produces two regimes rather than a gradient. At class widths of ten and fifty the aggregate population survives ten thousand ticks and tracks the individual run within 375 parts per thousand on population and 43 on energy. At widths of twenty-five, one hundred and two hundred the aggregate population dies out entirely while the individual run sustains seventy-five members. That is a false extinction produced by resolution alone: a whole class starves the moment its average cannot pay upkeep, where individuals would have starved a few at a time and the survivors would have kept the shared per-capita share up.

The collapse is not monotone in class width. Width twenty-five destroys the population while width fifty sustains it. Whether a class dips below upkeep together depends on how its width lands against the population cycle that the shared intake drives, and this fixture does not characterise that further. The consequence for a tier policy is the useful part: a resolution cannot be chosen by assuming that degradation worsens smoothly as classes widen, and a width that works is not evidence that every narrower width works. M4.6 inherits that.

Conservation was exact at every tick on both sides in every run, including the ones whose populations died out, which is the point ECOLOGY_MODEL makes about conservation being insufficient evidence. Four of these five runs conserve perfectly and three of them are ecologically worthless.

Alternatives Considered

Asserting the aggregate run matches the individual run within a tolerance and leaving it there is the shape this milestone is named against: it reads as equivalence, it says nothing about where the error comes from, and it passes just as well when the fixture has been chosen so that nothing is averaged. Deriving the aggregate fluxes from a separately written population model would be a more realistic aggregate, and would make every divergence uninterpretable, because the two models would differ in ways nobody could enumerate. Varying the interval length as the resolution axis was considered and rejected for this increment: it measures a real error but confounds it with the averaging error, and the averaging error is the one M4 is built on. Using a fitted tolerance derived from the observed numbers would make the test unfalsifiable; the asserted bounds are round numbers chosen above the measurement with room for platform variation, and the measurement itself is what is published.

Benefits

The cost of aggregation in this fixture is a number rather than an assurance, and it is attributable to averaging because the control rules out an implementation difference. The relationship between class width and error is visible, so a caller choosing a resolution has something to choose against. A change that made the comparison stop comparing fails the suite.

Costs

The fixture is one model, one scarce axis and one seed regime, so its numbers describe itself and nothing wider. The control is limited to a short horizon, because one class per age exhausts the sixty-four-class bound within a few tens of ticks; the instrument is therefore checked over tens of ticks and used over ten thousand. The aggregate run sub-steps at one tick, which is not how a cheap aggregate would be run in production, so this measures the averaging cost alone and not the cost of a long interval.

Risks

This is not a fidelity claim and does not become one by being repeated. The measured envelope belongs to this fixture: another model, another scarcity regime or another initial distribution would give other numbers, and nothing here bounds them. Agreement inside the control proves the two runs implement one rule, not that the rule resembles biology. The asserted bounds are chosen, not derived, so they are a regression guard and not a theoretical limit. Interval-length error is unmeasured. Attribution rests on the controls covering every place the two runs could order things differently: breeding allocation is covered because it is the only stage that depends on order, and the environment is sized so that intake never depletes it, which is asserted rather than assumed. A future stage that introduced another order-sensitive decision would need its own control before these numbers could be trusted again. Diversity is compared as counts per genome, which is not a species concept and settles nothing about speciation, and this fixture runs one genome, so the mixed-genome case ECOLOGY_MODEL also asks for is not covered here. The non-monotonicity is reported from five widths, which locates it but does not map it; the boundary between the regimes is not characterised and no mechanism beyond whole-class starvation is claimed. Nothing here is persistent, no population is distributed across owners, and crash behaviour at conversion boundaries remains what ADR-0037 and ADR-0039 already scoped in one process.

Consequences

Test the controls first, so a failing instrument fails before any envelope is reported: with one member per class the aggregate run must equal the individual run exactly, every tick, across the whole distribution of members and not merely in totals, both in ordinary growth and at the capacity boundary where eligible parents outnumber free slots. Test that the boundary was actually reached, or the second control is only the first with fewer members. Then run the scenario ECOLOGY_MODEL asks for, a hundred funded individuals collapsed, advanced over ten thousand ticks and materialized, asserting that conservation is exact at every step on both sides, that genome identity is preserved and no genome is invented or lost, and that population, biomass and diversity divergence stay inside the published envelope. Assert that the two runs do not agree, so equivalence cannot be read as identity. Report the envelope per class width to a results file. No genome schema, mutation catalog, archive, ledger, episode driver, ensemble, cohort record, collapse, interval accounting, materialization, renderer or deployment changes.