Files
aevolution/docs/adr/0040-resolution-equivalence.md
T
2026-10-08 12:08:53 -04:00

196 lines
12 KiB
Markdown

[Reviewed]
# ADR-0040: Measuring what aggregation costs, and refusing to call it equivalence
## Status
Adopted for M4.5, which closes M4. It adds no library code: the whole increment is
a measurement, and needing to change a contract in order to measure it would have
been a finding rather than a step. No M1 to M4.4 contract changes, and ADR-0038's
refusal to choose an ecology is unchanged: the model compared here lives in the
test fixture as a declared measurement input.
## Context
Everything M4 built proves its arithmetic closes. None of it says whether a cohort
behaves like the individuals it replaced, and ECOLOGY_MODEL is blunt about the
difference: passing conservation alone is insufficient ecological evidence, exact
seeded repeatability does not imply exact fidelity to individual ecology, and what
is wanted is matched ensembles with published error envelopes.
The named risk is claiming identical trajectories at different resolution. It is a
risk precisely because the tempting experiment supports the claim: pick a fixture
where averaging happens not to bite, observe agreement, and report equivalence.
The honest experiment has to be built so that it can fail, and has to report the
size of the disagreement rather than its absence.
There is a second, quieter problem. Comparing two implementations of "the same"
model measures the difference between the implementations as well as the difference
between the resolutions, and those are not separable after the fact. A divergence
of five per cent could be aggregation error or could be a transcription mistake in
one of the two runs, and no amount of staring at the number distinguishes them.
## Decision
The rule is written once and applied at both resolutions. The individual run
applies it to one member at a time; the aggregate run applies it to a class average
and multiplies by the count. Nothing else differs, and in particular the aggregate
run sub-steps internally at one tick so that interval length is not a second source
of error confounded with the first.
Anything that decides between members has to be part of the rule, not a
consequence of how each run happens to iterate. Breeding is the case that matters:
when there are fewer free slots than eligible parents, the allocation order decides
which parents reproduce, and two runs ordering differently diverge for a reason that
has nothing to do with averaging. The rule therefore states the order, oldest first,
which both resolutions can honour because classes partition ages. Ordering inside a
class is the one thing only the individual run can see, and that is a resolution
effect rather than an implementation difference.
Control runs establish that the two are the same rule before any divergence is
attributed to resolution. When every class holds exactly one member, a class average
is that member, so the aggregate run must reproduce the individual run exactly, tick
by tick. There are two of them, because one is not enough. The first exercises
ordinary growth. The second sits at the capacity boundary, where eligible parents
outnumber free slots, because that is the only place the allocation order is
observable at all; a fixture that never contends never checks it. Both compare the
whole distribution of members rather than censuses: whoever breeds hands the same
investment to a child either way, so every total agrees while the distribution does
not, and the disagreement only reaches a census dozens of ticks later.
These controls are the experiment's instrument check. Without them a measured
divergence means nothing, and if either fails, every other number in this ADR is
void.
Divergence is then produced deliberately by coarsening the classes, which is the
resolution axis. Width is varied and the envelope is reported per width, so the
relationship between declared resolution and error is visible rather than a single
number standing in for it.
The scarce quantity is a single axis. A shared environment supplies a fixed amount
of energy per tick divided equally among the living, and everything else is
abundant, so a member that cannot pay its upkeep starves. That makes the mechanism
attributable: at individual resolution the weakest starve first, and at class
resolution the average pays for everyone until it cannot, at which point the class
goes together. This is the "population averages hide viable outliers" risk of
ADR-0036 arriving as a measurable quantity rather than as a warning.
The tests assert an envelope, and they also assert that the trajectories differ.
Asserting the difference is the part that protects the claim: a later change that
quietly made the runs agree would mean either that the fixture stopped exercising
aggregation or that the comparison stopped comparing, and both should fail loudly
rather than look like progress. Nothing in this ADR asserts equality outside the
one-member-per-class control, and the word equivalence is not used for what is
measured.
The envelope is published as a results file alongside the existing benchmark
series, so the numbers are reviewable and comparable across platforms rather than
living only in a commit message.
## What the measurement found
The controls hold: with one member per class the two runs agree exactly, tick by
tick, in every channel and across the whole distribution of members, including at
the capacity boundary where twelve eligible parents contend for one free slot. The
instrument was verified rather than assumed, by three separate mutations. A
one-unit drift introduced into the aggregate upkeep is caught. Allocating contested
slots in member insertion order, which is what the individual run originally did, is
caught. Walking the classes in stored order, youngest first, which is what the
aggregate run originally did, is caught. Since either allocation order alone fails a
control, the divergence reported below is attributable to averaging rather than to
the two runs disagreeing about who breeds.
Coarsening then produces two regimes rather than a gradient. At class widths of ten
and fifty the aggregate population survives ten thousand ticks and tracks the
individual run within 375 parts per thousand on population and 43 on energy. At
widths of twenty-five, one hundred and two hundred the aggregate population dies
out entirely while the individual run sustains seventy-five members. That is a
false extinction produced by resolution alone: a whole class starves the moment its
average cannot pay upkeep, where individuals would have starved a few at a time and
the survivors would have kept the shared per-capita share up.
The collapse is not monotone in class width. Width twenty-five destroys the
population while width fifty sustains it. Whether a class dips below upkeep
together depends on how its width lands against the population cycle that the
shared intake drives, and this fixture does not characterise that further. The
consequence for a tier policy is the useful part: a resolution cannot be chosen by
assuming that degradation worsens smoothly as classes widen, and a width that works
is not evidence that every narrower width works. M4.6 inherits that.
Conservation was exact at every tick on both sides in every run, including the ones
whose populations died out, which is the point ECOLOGY_MODEL makes about
conservation being insufficient evidence. Four of these five runs conserve
perfectly and three of them are ecologically worthless.
## Alternatives Considered
Asserting the aggregate run matches the individual run within a tolerance and
leaving it there is the shape this milestone is named against: it reads as
equivalence, it says nothing about where the error comes from, and it passes just
as well when the fixture has been chosen so that nothing is averaged. Deriving the
aggregate fluxes from a separately written population model would be a more
realistic aggregate, and would make every divergence uninterpretable, because the
two models would differ in ways nobody could enumerate. Varying the interval length
as the resolution axis was considered and rejected for this increment: it measures
a real error but confounds it with the averaging error, and the averaging error is
the one M4 is built on. Using a fitted tolerance derived from the observed numbers
would make the test unfalsifiable; the asserted bounds are round numbers chosen
above the measurement with room for platform variation, and the measurement itself
is what is published.
## Benefits
The cost of aggregation in this fixture is a number rather than an assurance, and
it is attributable to averaging because the control rules out an implementation
difference. The relationship between class width and error is visible, so a caller
choosing a resolution has something to choose against. A change that made the
comparison stop comparing fails the suite.
## Costs
The fixture is one model, one scarce axis and one seed regime, so its numbers
describe itself and nothing wider. The control is limited to a short horizon,
because one class per age exhausts the sixty-four-class bound within a few tens of
ticks; the instrument is therefore checked over tens of ticks and used over ten
thousand. The aggregate run sub-steps at one tick, which is not how a cheap
aggregate would be run in production, so this measures the averaging cost alone and
not the cost of a long interval.
## Risks
This is not a fidelity claim and does not become one by being repeated. The
measured envelope belongs to this fixture: another model, another scarcity regime
or another initial distribution would give other numbers, and nothing here bounds
them. Agreement inside the control proves the two runs implement one rule, not that
the rule resembles biology. The asserted bounds are chosen, not derived, so they
are a regression guard and not a theoretical limit. Interval-length error is
unmeasured. Attribution rests on the controls covering every place the two runs
could order things differently: breeding allocation is covered because it is the
only stage that depends on order, and the environment is sized so that intake never
depletes it, which is asserted rather than assumed. A future stage that introduced
another order-sensitive decision would need its own control before these numbers
could be trusted again. Diversity is compared as counts per genome, which is not a species
concept and settles nothing about speciation, and this fixture runs one genome, so
the mixed-genome case ECOLOGY_MODEL also asks for is not covered here. The
non-monotonicity is reported from five widths, which locates it but does not map
it; the boundary between the regimes is not characterised and no mechanism beyond
whole-class starvation is claimed. Nothing here is persistent, no
population is distributed across owners, and crash behaviour at conversion
boundaries remains what ADR-0037 and ADR-0039 already scoped in one process.
## Consequences
Test the controls first, so a failing instrument fails before any envelope is
reported: with one member per class the aggregate run must equal the individual run
exactly, every tick, across the whole distribution of members and not merely in
totals, both in ordinary growth and at the capacity boundary where eligible parents
outnumber free slots. Test that the boundary was actually reached, or the second
control is only the first with fewer members. Then run the scenario
ECOLOGY_MODEL asks for, a hundred funded individuals collapsed, advanced over ten
thousand ticks and materialized, asserting that conservation is exact at every step
on both sides, that genome identity is preserved and no genome is invented or lost,
and that population, biomass and diversity divergence stay inside the published
envelope. Assert that the two runs do not agree, so equivalence cannot be read as
identity. Report the envelope per class width to a results file. No genome schema,
mutation catalog, archive, ledger, episode driver, ensemble, cohort record,
collapse, interval accounting, materialization, renderer or deployment changes.