196 lines
12 KiB
Markdown
196 lines
12 KiB
Markdown
[Reviewed]
|
|
# ADR-0040: Measuring what aggregation costs, and refusing to call it equivalence
|
|
|
|
## Status
|
|
|
|
Adopted for M4.5, which closes M4. It adds no library code: the whole increment is
|
|
a measurement, and needing to change a contract in order to measure it would have
|
|
been a finding rather than a step. No M1 to M4.4 contract changes, and ADR-0038's
|
|
refusal to choose an ecology is unchanged: the model compared here lives in the
|
|
test fixture as a declared measurement input.
|
|
|
|
## Context
|
|
|
|
Everything M4 built proves its arithmetic closes. None of it says whether a cohort
|
|
behaves like the individuals it replaced, and ECOLOGY_MODEL is blunt about the
|
|
difference: passing conservation alone is insufficient ecological evidence, exact
|
|
seeded repeatability does not imply exact fidelity to individual ecology, and what
|
|
is wanted is matched ensembles with published error envelopes.
|
|
|
|
The named risk is claiming identical trajectories at different resolution. It is a
|
|
risk precisely because the tempting experiment supports the claim: pick a fixture
|
|
where averaging happens not to bite, observe agreement, and report equivalence.
|
|
The honest experiment has to be built so that it can fail, and has to report the
|
|
size of the disagreement rather than its absence.
|
|
|
|
There is a second, quieter problem. Comparing two implementations of "the same"
|
|
model measures the difference between the implementations as well as the difference
|
|
between the resolutions, and those are not separable after the fact. A divergence
|
|
of five per cent could be aggregation error or could be a transcription mistake in
|
|
one of the two runs, and no amount of staring at the number distinguishes them.
|
|
|
|
## Decision
|
|
|
|
The rule is written once and applied at both resolutions. The individual run
|
|
applies it to one member at a time; the aggregate run applies it to a class average
|
|
and multiplies by the count. Nothing else differs, and in particular the aggregate
|
|
run sub-steps internally at one tick so that interval length is not a second source
|
|
of error confounded with the first.
|
|
|
|
Anything that decides between members has to be part of the rule, not a
|
|
consequence of how each run happens to iterate. Breeding is the case that matters:
|
|
when there are fewer free slots than eligible parents, the allocation order decides
|
|
which parents reproduce, and two runs ordering differently diverge for a reason that
|
|
has nothing to do with averaging. The rule therefore states the order, oldest first,
|
|
which both resolutions can honour because classes partition ages. Ordering inside a
|
|
class is the one thing only the individual run can see, and that is a resolution
|
|
effect rather than an implementation difference.
|
|
|
|
Control runs establish that the two are the same rule before any divergence is
|
|
attributed to resolution. When every class holds exactly one member, a class average
|
|
is that member, so the aggregate run must reproduce the individual run exactly, tick
|
|
by tick. There are two of them, because one is not enough. The first exercises
|
|
ordinary growth. The second sits at the capacity boundary, where eligible parents
|
|
outnumber free slots, because that is the only place the allocation order is
|
|
observable at all; a fixture that never contends never checks it. Both compare the
|
|
whole distribution of members rather than censuses: whoever breeds hands the same
|
|
investment to a child either way, so every total agrees while the distribution does
|
|
not, and the disagreement only reaches a census dozens of ticks later.
|
|
|
|
These controls are the experiment's instrument check. Without them a measured
|
|
divergence means nothing, and if either fails, every other number in this ADR is
|
|
void.
|
|
|
|
Divergence is then produced deliberately by coarsening the classes, which is the
|
|
resolution axis. Width is varied and the envelope is reported per width, so the
|
|
relationship between declared resolution and error is visible rather than a single
|
|
number standing in for it.
|
|
|
|
The scarce quantity is a single axis. A shared environment supplies a fixed amount
|
|
of energy per tick divided equally among the living, and everything else is
|
|
abundant, so a member that cannot pay its upkeep starves. That makes the mechanism
|
|
attributable: at individual resolution the weakest starve first, and at class
|
|
resolution the average pays for everyone until it cannot, at which point the class
|
|
goes together. This is the "population averages hide viable outliers" risk of
|
|
ADR-0036 arriving as a measurable quantity rather than as a warning.
|
|
|
|
The tests assert an envelope, and they also assert that the trajectories differ.
|
|
Asserting the difference is the part that protects the claim: a later change that
|
|
quietly made the runs agree would mean either that the fixture stopped exercising
|
|
aggregation or that the comparison stopped comparing, and both should fail loudly
|
|
rather than look like progress. Nothing in this ADR asserts equality outside the
|
|
one-member-per-class control, and the word equivalence is not used for what is
|
|
measured.
|
|
|
|
The envelope is published as a results file alongside the existing benchmark
|
|
series, so the numbers are reviewable and comparable across platforms rather than
|
|
living only in a commit message.
|
|
|
|
## What the measurement found
|
|
|
|
The controls hold: with one member per class the two runs agree exactly, tick by
|
|
tick, in every channel and across the whole distribution of members, including at
|
|
the capacity boundary where twelve eligible parents contend for one free slot. The
|
|
instrument was verified rather than assumed, by three separate mutations. A
|
|
one-unit drift introduced into the aggregate upkeep is caught. Allocating contested
|
|
slots in member insertion order, which is what the individual run originally did, is
|
|
caught. Walking the classes in stored order, youngest first, which is what the
|
|
aggregate run originally did, is caught. Since either allocation order alone fails a
|
|
control, the divergence reported below is attributable to averaging rather than to
|
|
the two runs disagreeing about who breeds.
|
|
|
|
Coarsening then produces two regimes rather than a gradient. At class widths of ten
|
|
and fifty the aggregate population survives ten thousand ticks and tracks the
|
|
individual run within 375 parts per thousand on population and 43 on energy. At
|
|
widths of twenty-five, one hundred and two hundred the aggregate population dies
|
|
out entirely while the individual run sustains seventy-five members. That is a
|
|
false extinction produced by resolution alone: a whole class starves the moment its
|
|
average cannot pay upkeep, where individuals would have starved a few at a time and
|
|
the survivors would have kept the shared per-capita share up.
|
|
|
|
The collapse is not monotone in class width. Width twenty-five destroys the
|
|
population while width fifty sustains it. Whether a class dips below upkeep
|
|
together depends on how its width lands against the population cycle that the
|
|
shared intake drives, and this fixture does not characterise that further. The
|
|
consequence for a tier policy is the useful part: a resolution cannot be chosen by
|
|
assuming that degradation worsens smoothly as classes widen, and a width that works
|
|
is not evidence that every narrower width works. M4.6 inherits that.
|
|
|
|
Conservation was exact at every tick on both sides in every run, including the ones
|
|
whose populations died out, which is the point ECOLOGY_MODEL makes about
|
|
conservation being insufficient evidence. Four of these five runs conserve
|
|
perfectly and three of them are ecologically worthless.
|
|
|
|
## Alternatives Considered
|
|
|
|
Asserting the aggregate run matches the individual run within a tolerance and
|
|
leaving it there is the shape this milestone is named against: it reads as
|
|
equivalence, it says nothing about where the error comes from, and it passes just
|
|
as well when the fixture has been chosen so that nothing is averaged. Deriving the
|
|
aggregate fluxes from a separately written population model would be a more
|
|
realistic aggregate, and would make every divergence uninterpretable, because the
|
|
two models would differ in ways nobody could enumerate. Varying the interval length
|
|
as the resolution axis was considered and rejected for this increment: it measures
|
|
a real error but confounds it with the averaging error, and the averaging error is
|
|
the one M4 is built on. Using a fitted tolerance derived from the observed numbers
|
|
would make the test unfalsifiable; the asserted bounds are round numbers chosen
|
|
above the measurement with room for platform variation, and the measurement itself
|
|
is what is published.
|
|
|
|
## Benefits
|
|
|
|
The cost of aggregation in this fixture is a number rather than an assurance, and
|
|
it is attributable to averaging because the control rules out an implementation
|
|
difference. The relationship between class width and error is visible, so a caller
|
|
choosing a resolution has something to choose against. A change that made the
|
|
comparison stop comparing fails the suite.
|
|
|
|
## Costs
|
|
|
|
The fixture is one model, one scarce axis and one seed regime, so its numbers
|
|
describe itself and nothing wider. The control is limited to a short horizon,
|
|
because one class per age exhausts the sixty-four-class bound within a few tens of
|
|
ticks; the instrument is therefore checked over tens of ticks and used over ten
|
|
thousand. The aggregate run sub-steps at one tick, which is not how a cheap
|
|
aggregate would be run in production, so this measures the averaging cost alone and
|
|
not the cost of a long interval.
|
|
|
|
## Risks
|
|
|
|
This is not a fidelity claim and does not become one by being repeated. The
|
|
measured envelope belongs to this fixture: another model, another scarcity regime
|
|
or another initial distribution would give other numbers, and nothing here bounds
|
|
them. Agreement inside the control proves the two runs implement one rule, not that
|
|
the rule resembles biology. The asserted bounds are chosen, not derived, so they
|
|
are a regression guard and not a theoretical limit. Interval-length error is
|
|
unmeasured. Attribution rests on the controls covering every place the two runs
|
|
could order things differently: breeding allocation is covered because it is the
|
|
only stage that depends on order, and the environment is sized so that intake never
|
|
depletes it, which is asserted rather than assumed. A future stage that introduced
|
|
another order-sensitive decision would need its own control before these numbers
|
|
could be trusted again. Diversity is compared as counts per genome, which is not a species
|
|
concept and settles nothing about speciation, and this fixture runs one genome, so
|
|
the mixed-genome case ECOLOGY_MODEL also asks for is not covered here. The
|
|
non-monotonicity is reported from five widths, which locates it but does not map
|
|
it; the boundary between the regimes is not characterised and no mechanism beyond
|
|
whole-class starvation is claimed. Nothing here is persistent, no
|
|
population is distributed across owners, and crash behaviour at conversion
|
|
boundaries remains what ADR-0037 and ADR-0039 already scoped in one process.
|
|
|
|
## Consequences
|
|
|
|
Test the controls first, so a failing instrument fails before any envelope is
|
|
reported: with one member per class the aggregate run must equal the individual run
|
|
exactly, every tick, across the whole distribution of members and not merely in
|
|
totals, both in ordinary growth and at the capacity boundary where eligible parents
|
|
outnumber free slots. Test that the boundary was actually reached, or the second
|
|
control is only the first with fewer members. Then run the scenario
|
|
ECOLOGY_MODEL asks for, a hundred funded individuals collapsed, advanced over ten
|
|
thousand ticks and materialized, asserting that conservation is exact at every step
|
|
on both sides, that genome identity is preserved and no genome is invented or lost,
|
|
and that population, biomass and diversity divergence stay inside the published
|
|
envelope. Assert that the two runs do not agree, so equivalence cannot be read as
|
|
identity. Report the envelope per class width to a results file. No genome schema,
|
|
mutation catalog, archive, ledger, episode driver, ensemble, cohort record,
|
|
collapse, interval accounting, materialization, renderer or deployment changes.
|