Complete M7.4b lineage checkpoint profile; M7.4 done

ADR-0062 adds a canonical LineageArchive codec (parents before children,
organisms by id, FNV) that decodes by replaying admit/record_birth/
record_death and requires byte-identical re-encoding; it never overwrites
an existing archive. LineageRegion binds the environmental producer and
consumer accounts to archived organisms with identity high-waters,
enforces closure (lifecycle vs deaths, exact living set, tick bounds)
and records deaths in the tick that causes them.

Checkpoint profile 2 (environmental-lineage-v1, history profile 1)
publishes that container; schema v4 admits it with a 4,268,144-byte cap
and requires each commit to keep its predecessor's profile, RNG, history
and durability declarations. Linux adds provision/publish/recover for
the profile on the shared publication path.

Qualification found WSL2 stepping CLOCK_REALTIME back ~0.5 s every ~32 s,
which made two 1 s-lease tests flaky under ASan. Tests now wait on
database-time expiry; ADR-0060 records the slewed-clock requirement.

Validated: Windows CTest 57/57; Linux GCC Debug/Release, clang-tidy and
ASan/UBSan CTest 57/57 each; PostgreSQL suites 2x Release, 2x analysis,
5x ASan; SQL and C++ mutations each caught. No Docker/Gitea CI, remote
store, process kill or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 16:43:10 -04:00
co-authored by Claude Opus 5.5
parent 5d4f59f901
commit a2c4543df2
33 changed files with 1622 additions and 69 deletions
+13
View File
@@ -529,3 +529,16 @@ codec and history-capable binding). M7.4a is implemented and locally qualified
retained-build allowlists and restored-trajectory equality are covered.
Next: M7.4b. M7.4 stays open until it passes.
M7.4b implemented and locally qualified (2026-09-28); M7.4 is complete. ADR-0062
adds:
- A canonical lineage archive codec that decodes by replaying the archive's own
rules.
- `LineageRegion`, which binds the environmental accounts to archived organisms,
keeps identity high-waters, enforces closure, and records deaths in the
causing tick.
- Checkpoint profile 2, and schema v4 (profile-specific caps; the profile is
fixed per region).
Recovery restores retained genomes, ancestry and dead organisms, and continues
identically. Next: M7.5 durable decisions, history and outbox rows in the same
transaction, and commit-ID resolution.
+24
View File
@@ -788,3 +788,27 @@ builds. Windows ran 56/56; Linux Release, clang-tidy and ASan/UBSan PostgreSQL
suites passed in WSL. M7.4 stays open for M7.4b (lineage archive codec and
history-capable binding). No real world was provisioned; there is no deployment,
host or process-kill test yet.
## M7.4b lineage checkpoint profile; M7.4 complete (2026-09-28)
ADR-0062 adds a canonical `LineageArchive` codec that decodes by replaying the
archive's own admission, birth and death rules. It also adds `LineageRegion`,
which:
- binds the two environmental accounts to archived organisms;
- keeps identity high-waters;
- enforces closure over lifecycle, the living set and ticks;
- records deaths in the tick that causes them.
Checkpoint profile 2 (`environmental-lineage-v1`) and schema v4 publish and recover
it. A region can never switch profile. Recovery restores retained genomes,
ancestry and dead organisms, and continues identically. A full archive encodes to
2,543,648 of at most 3,219,496 bytes.
While qualifying, WSL2 was measured stepping the wall clock back about 0.5 s every
~32 s. Four database-time tests now wait on the database's own expiry judgement
instead of assuming a local sleep equals database time; ADR-0060 records the
deployment requirement. Windows ran 57/57. Linux GCC Debug/Release, clang-tidy
and ASan/UBSan CTest each ran 57/57. The PostgreSQL suites passed 2/2/5 times.
Next: M7.5 durable decisions, history and outbox rows, and commit-ID resolution.
Retained ancestry still needs M7.5 history and M7.7/M7.9 kill-recovery evidence.
+3 -1
View File
@@ -2,7 +2,9 @@
## Status
Adopted for M3.5. It does not settle NPC mutation, which the roadmap lists as a
Adopted for M3.5. ADR-0062 (M7.4b) adds the whole-archive codec, which
decodes by replaying these same admission, birth and death rules and never
overwrites an existing archive. It does not settle NPC mutation, which the roadmap lists as a
dependency and OPEN_QUESTIONS leaves open until M12: the archive records an origin
event and never an actor, so whether an NPC may diverge changes nothing here.
Durable storage, retention tiers and cross-version migration remain M7 and M8.
+13 -1
View File
@@ -89,7 +89,19 @@ pre-created by an operator. This adds a second migration and a chain of prefix c
## Risks
Grants made after the last migrate run are not audited until it runs again, so
operators must rerun it after role changes. A frozen VM/hypervisor can stop all
operators must rerun it after role changes.
Database time is the server's wall clock (`clock_timestamp()`), so steps count.
M7.4b measured WSL2 stepping it back about 0.5 s every ~32 s: 3.66 s over 240 s,
roughly a 1.5% rate difference from CLOCK_MONOTONIC.
- Backward steps only lengthen leases in database time. Exclusivity holds, and a
holder's monotonic deadline still fences it early.
- Forward steps shorten leases. A successor can then acquire while the old holder
still believes it has authority. Commits are still rejected, but provisional
output could overlap.
Database hosts therefore need slewed, not stepped, time. A forward step larger
than the safety margin is outside the declared failure model. Tests wait on
database-time expiry instead of assuming a local sleep equals database time. A frozen VM/hypervisor can stop all
guest clocks. Database commits stay fenced, but
provisional output could then be late. Gateways still need bounded-validity
registration (ADR-0058). The bare head kernel accepts any 32-byte digest until M7.4
@@ -0,0 +1,96 @@
# ADR-0062: Lineage archive codec and history-capable checkpoint profile
## Status
Accepted for M7.4b, 2026-09-28. With ADR-0061 this completes M7.4. Durable
decision and history rows are still M7.5. The M7 retained-ancestry exit claim
still needs M7.7/M7.9 process-kill recovery evidence.
## Context
REGION_DURABILITY.md requires a history-capable profile before retained ancestry
can be claimed. That profile must validate:
- organism-to-genome bindings and parent closure;
- lifecycle agreement between the simulation and the archive;
- identity high-waters and archived dead members.
`LineageArchive` (ADR-0034) had no codec and deliberately has no assignment, so
an older copy cannot be written back. The M6 environmental fixture has two
organism accounts but no genome linkage, and it uses no randomness.
## Decision
**Archive codec v1** (`LineageArchive::encode/decode`, schemas/lineage-archive-v1.md):
- Genome v1 records are ordered by (ancestry depth, id), so parents precede
children. Organisms follow in id order, as 33-byte records.
- A trailing FNV-1a64 detects corruption. The caps are the existing 4096 genomes
and 65536 organisms, at most 3,219,496 bytes, checked before allocation.
- Decoding constructs a new archive by replaying `admit`, `record_birth` and
`record_death`, so retention, parent presence, death finality and tick order are
re-established by the archive's own rules. A byte-identical re-encoding is
required, so equal content has exactly one encoding.
- Decoding never overwrites an existing archive. Refusing an older checkpoint is
the committed chain's job (ADR-0061): recovery only ever loads the head.
**`LineageRegion`** (processes, headless) holds the environmental community, its
archive, the binding of the producer and consumer accounts to archived organisms,
and the next organism and genome identities.
*Closure* requires:
- both bound organisms are recorded;
- each terminal flag equals the archive's death state;
- no birth or death is after the current tick;
- the archive's living organisms are exactly the bound, non-terminal ones, so
there is no unowned living state;
- both high-waters are above every recorded identity.
`advance` records each terminal transition as an archive death at that tick. A
checkpoint therefore cannot split a death from the state that caused it. The
container is schemas/environmental-lineage-v1.md: header, binding, envelope,
archive and FNV.
**Checkpoint profile 2** is `environmental-lineage-v1`, with history profile 1
(archive 1 + genome schema 1). It uses the same nine version slots and zero RNG
streams, and its payload cap is 72 + 1 MiB + 3,219,496 bytes. The manifest
structure is unchanged, and profile and history profile must agree.
Schema v4 replaces the SQL parser to admit profile 2 with its cap. Publication
additionally requires the predecessor's exact profile, RNG, history and durability
declarations. Grants are unchanged. Linux adds
`provision/publish/recover_lineage_checkpoint` on the same publication and
recovery path as profile 1; each path refuses the other profile.
## Alternatives Considered
- **Extending environmental-v1 with optional sections**: forbidden, because empty
sections cannot stand in for missing history.
- **Admission-order encoding**: it is not recoverable from the archive and would
make equal content encode differently.
- **Deaths recorded by the host after the tick** could be split from the state
across a checkpoint.
- **A separate history object** would need its own atomic publication; one
container shares the manifest's digest and fence.
## Benefits
- Retained genomes, ancestry and dead organisms now survive checkpoint recovery
and must agree with the simulation state that references them.
- Decode reuses the archive's invariants rather than duplicating them.
- Existing environmental-v1 regions are unaffected.
## Costs
- The whole archive is rewritten with every batch. A full archive is about 2.5 MiB
in the measured test, up to about 3.2 MiB.
- Decode is O(genomes²) in the worst case because of sorted insertion.
- Archive memory is bounded by the fixed caps rather than charged to a budget.
## Risks
- The environmental fixture has no reproduction, so births within a batch are not
exercised. A reproducing profile needs its own closure rules.
- Durable history rows (M7.5) must agree with the checkpoint's archive. Only the
checkpoint side exists yet.
- FNV is not authentication; SHA-256 at the manifest covers the payload.
## Consequences
- M7.5 adds death decisions, history rows and outbox rows derived from these
transitions, in the same transaction.
- M7.7 recovers this profile after a process kill.
- M7.9 retains builds that can decode archive 1 and genome schema 1, and
qualifies retained ancestry end to end.
- A new genome schema or archive format needs a new history profile value and a
migration.
+1
View File
@@ -72,3 +72,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0059: PostgreSQL bootstrap](0059-postgresql-bootstrap.md) - accepted for M7.2 optional adapter, pinned server/client, migrations and Secret boundary.
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
- [ADR-0062: Lineage checkpoint profile](0062-lineage-checkpoint-profile.md) - accepted for M7.4b archive codec, LineageRegion closure and history-capable profile 2.
+7 -3
View File
@@ -179,6 +179,10 @@ a local filesystem object store:
- Unsupported builds are refused by both the database and recovery.
- Recovery never creates genesis or picks an older head.
Two parts remain open. M7.4b adds the lineage archive codec and a history-capable
profile; until then the history rows of this audit stay open. No process-kill
test was run yet (7.7).
M7.4b (ADR-0062) closes the checkpoint side of the history audit. The lineage
archive codec and the `environmental-lineage-v1` profile validate:
- organism-to-genome bindings and parent closure (by archive replay);
- lifecycle agreement, with deaths recorded in the causing tick;
- the living set, identity high-waters and retained dead members.
Recovery restores all of these. Durable history rows (M7.5) and process-kill
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
+2 -1
View File
@@ -259,7 +259,8 @@ API. Absence is also reversible: a lineage with no carrier can gain one again.
Capacity is bounded and admission fails loudly at the limit. Refusing to record is
not deleting; a silent drop would be. Durable storage, retention tiers and
cross-version migration are M7 and M8 work, and nothing here is serialized. NPC
cross-version migration are M7 and M8 work. M7.4b (ADR-0062) serializes the
whole archive canonically for history-capable region checkpoints. NPC
mutation stays undecided, because the archive records an origin event and never an
actor.
+55
View File
@@ -2158,3 +2158,58 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
These are local socket and local disk only.
- Not executed: Docker/Gitea CI, a remote object store or PVC, power-loss fsync
semantics, read-back verification fault injection, process kill, and TSan.
## M7.4b lineage checkpoint profile - 2026-09-28
- Windows MSVC Release CTest passed 57/57, including the new `lineage-region-contracts`
and extended lineage and region-checkpoint contracts. The repository clang-format
19 check passed.
- WSL Ubuntu (GCC 14.2.0, clang 19.1.1, PostgreSQL 18.6, OpenSSL 3.0.13):
- The Linux `checks` build set (separate build dirs) passed CTest 57/57 in each
of GCC14 Debug, GCC14 Release, clang19 analysis and clang19 ASan/UBSan.
- The PostgreSQL integration suite passed 2× Release, 2× analysis and 5×
ASan/UBSan.
- Archive codec coverage:
- canonical encoding regardless of admission order, including a child with a
smaller ID than its parent;
- checksum, magic, version, count cap, length and organism-count framing;
- genome damage, unknown genome, invalid alive/died forms, death before birth,
reordered organisms (non-canonical) and a child encoded before its parent;
- a full 4096-genome/65536-organism round trip at 2,543,648 bytes.
- `LineageRegion` coverage:
- closure refusals: missing, identical or dead binding; low high-waters; stray
living organism; future birth; archive death while the community is active;
- the death recorded at tick 20 in the same advance; a refused tick changes
nothing;
- restore plus 36 continuation ticks identical to the original;
- container, binding, envelope and archive damage refused.
- Profile 2 coverage:
- the 4,268,144-byte cap is asserted in C++ and enforced by SQL (the maximum is
accepted, +1 refused); the profile-1 1 MiB+1 is refused;
- mismatched profile/history is EVL06;
- lineage publish (death included), recover and continue, and a second
publication;
- the environmental path refuses a lineage predecessor before upload;
- the database refuses a profile switch;
- a host of the other profile reports unsupported.
- Migrations: clean→v4; v1→v4; v3 with a committed genesis upgraded to v4 with
the commit retained; v2 heads still block. Tampered v2/v3/v4 checksums and an
unknown v5 are refused.
- Mutations (restored afterwards), each failing at the expected line: removing
the SQL profile/history continuity check (store test 392), the living-set
closure check (lineage region 58), or death recording (lineage region 78).
- Flake found and fixed. The ASan PostgreSQL suite failed intermittently: 1 of 3,
then 1 of 5 runs, at the queued-writer and expired-renewal assertions.
- Cause: a 4-minute probe measured WSL2 stepping CLOCK_REALTIME back 470-596 ms
every ~32 s (3.66 s total, roughly 1.5% versus CLOCK_MONOTONIC).
- PostgreSQL `clock_timestamp()` follows that wall clock, so a 1.2 s local sleep
could be under 1 s of database time.
- Tests now poll `clock_timestamp() >= expires_at`, holding the row lock where
relevant. Afterwards the suite ran 5/5 under ASan.
- The fencing logic itself was unchanged; the requirement is recorded in
ADR-0060.
- Local measurements, Release:
- manifest publish p50 0.72 ms and p99 1.24 ms;
- checkpoint publication 3.5-5.0 ms for 23-37 KiB payloads.
- Not executed: Docker/Gitea CI, remote storage, process kill, reproduction within
a batch (the fixture has none), durable history rows (M7.5) and TSan.
+5
View File
@@ -107,6 +107,11 @@ Operator steps:
connection. It uploads and verifies the tick-0 payload, then calls the
owner-only `provision_region_checkpoint(manifest)`.
Schema v4 (ADR-0062) additionally admits profile 2 (`environmental-lineage-v1`,
payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
profile, history, RNG and durability declarations. Provision history-capable
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
There is no operator CLI for provisioning yet, and no real world has been
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
have no payload.