Complete M7.4b lineage checkpoint profile; M7.4 done
ADR-0062 adds a canonical LineageArchive codec (parents before children, organisms by id, FNV) that decodes by replaying admit/record_birth/ record_death and requires byte-identical re-encoding; it never overwrites an existing archive. LineageRegion binds the environmental producer and consumer accounts to archived organisms with identity high-waters, enforces closure (lifecycle vs deaths, exact living set, tick bounds) and records deaths in the tick that causes them. Checkpoint profile 2 (environmental-lineage-v1, history profile 1) publishes that container; schema v4 admits it with a 4,268,144-byte cap and requires each commit to keep its predecessor's profile, RNG, history and durability declarations. Linux adds provision/publish/recover for the profile on the shared publication path. Qualification found WSL2 stepping CLOCK_REALTIME back ~0.5 s every ~32 s, which made two 1 s-lease tests flaky under ASan. Tests now wait on database-time expiry; ADR-0060 records the slewed-clock requirement. Validated: Windows CTest 57/57; Linux GCC Debug/Release, clang-tidy and ASan/UBSan CTest 57/57 each; PostgreSQL suites 2x Release, 2x analysis, 5x ASan; SQL and C++ mutations each caught. No Docker/Gitea CI, remote store, process kill or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -529,3 +529,16 @@ codec and history-capable binding). M7.4a is implemented and locally qualified
|
||||
retained-build allowlists and restored-trajectory equality are covered.
|
||||
|
||||
Next: M7.4b. M7.4 stays open until it passes.
|
||||
|
||||
M7.4b implemented and locally qualified (2026-09-28); M7.4 is complete. ADR-0062
|
||||
adds:
|
||||
- A canonical lineage archive codec that decodes by replaying the archive's own
|
||||
rules.
|
||||
- `LineageRegion`, which binds the environmental accounts to archived organisms,
|
||||
keeps identity high-waters, enforces closure, and records deaths in the
|
||||
causing tick.
|
||||
- Checkpoint profile 2, and schema v4 (profile-specific caps; the profile is
|
||||
fixed per region).
|
||||
Recovery restores retained genomes, ancestry and dead organisms, and continues
|
||||
identically. Next: M7.5 durable decisions, history and outbox rows in the same
|
||||
transaction, and commit-ID resolution.
|
||||
|
||||
@@ -788,3 +788,27 @@ builds. Windows ran 56/56; Linux Release, clang-tidy and ASan/UBSan PostgreSQL
|
||||
suites passed in WSL. M7.4 stays open for M7.4b (lineage archive codec and
|
||||
history-capable binding). No real world was provisioned; there is no deployment,
|
||||
host or process-kill test yet.
|
||||
|
||||
## M7.4b lineage checkpoint profile; M7.4 complete (2026-09-28)
|
||||
|
||||
ADR-0062 adds a canonical `LineageArchive` codec that decodes by replaying the
|
||||
archive's own admission, birth and death rules. It also adds `LineageRegion`,
|
||||
which:
|
||||
- binds the two environmental accounts to archived organisms;
|
||||
- keeps identity high-waters;
|
||||
- enforces closure over lifecycle, the living set and ticks;
|
||||
- records deaths in the tick that causes them.
|
||||
|
||||
Checkpoint profile 2 (`environmental-lineage-v1`) and schema v4 publish and recover
|
||||
it. A region can never switch profile. Recovery restores retained genomes,
|
||||
ancestry and dead organisms, and continues identically. A full archive encodes to
|
||||
2,543,648 of at most 3,219,496 bytes.
|
||||
|
||||
While qualifying, WSL2 was measured stepping the wall clock back about 0.5 s every
|
||||
~32 s. Four database-time tests now wait on the database's own expiry judgement
|
||||
instead of assuming a local sleep equals database time; ADR-0060 records the
|
||||
deployment requirement. Windows ran 57/57. Linux GCC Debug/Release, clang-tidy
|
||||
and ASan/UBSan CTest each ran 57/57. The PostgreSQL suites passed 2/2/5 times.
|
||||
|
||||
Next: M7.5 durable decisions, history and outbox rows, and commit-ID resolution.
|
||||
Retained ancestry still needs M7.5 history and M7.7/M7.9 kill-recovery evidence.
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
## Status
|
||||
|
||||
Adopted for M3.5. It does not settle NPC mutation, which the roadmap lists as a
|
||||
Adopted for M3.5. ADR-0062 (M7.4b) adds the whole-archive codec, which
|
||||
decodes by replaying these same admission, birth and death rules and never
|
||||
overwrites an existing archive. It does not settle NPC mutation, which the roadmap lists as a
|
||||
dependency and OPEN_QUESTIONS leaves open until M12: the archive records an origin
|
||||
event and never an actor, so whether an NPC may diverge changes nothing here.
|
||||
Durable storage, retention tiers and cross-version migration remain M7 and M8.
|
||||
|
||||
@@ -89,7 +89,19 @@ pre-created by an operator. This adds a second migration and a chain of prefix c
|
||||
|
||||
## Risks
|
||||
Grants made after the last migrate run are not audited until it runs again, so
|
||||
operators must rerun it after role changes. A frozen VM/hypervisor can stop all
|
||||
operators must rerun it after role changes.
|
||||
|
||||
Database time is the server's wall clock (`clock_timestamp()`), so steps count.
|
||||
M7.4b measured WSL2 stepping it back about 0.5 s every ~32 s: 3.66 s over 240 s,
|
||||
roughly a 1.5% rate difference from CLOCK_MONOTONIC.
|
||||
- Backward steps only lengthen leases in database time. Exclusivity holds, and a
|
||||
holder's monotonic deadline still fences it early.
|
||||
- Forward steps shorten leases. A successor can then acquire while the old holder
|
||||
still believes it has authority. Commits are still rejected, but provisional
|
||||
output could overlap.
|
||||
Database hosts therefore need slewed, not stepped, time. A forward step larger
|
||||
than the safety margin is outside the declared failure model. Tests wait on
|
||||
database-time expiry instead of assuming a local sleep equals database time. A frozen VM/hypervisor can stop all
|
||||
guest clocks. Database commits stay fenced, but
|
||||
provisional output could then be late. Gateways still need bounded-validity
|
||||
registration (ADR-0058). The bare head kernel accepts any 32-byte digest until M7.4
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
# ADR-0062: Lineage archive codec and history-capable checkpoint profile
|
||||
|
||||
## Status
|
||||
Accepted for M7.4b, 2026-09-28. With ADR-0061 this completes M7.4. Durable
|
||||
decision and history rows are still M7.5. The M7 retained-ancestry exit claim
|
||||
still needs M7.7/M7.9 process-kill recovery evidence.
|
||||
|
||||
## Context
|
||||
REGION_DURABILITY.md requires a history-capable profile before retained ancestry
|
||||
can be claimed. That profile must validate:
|
||||
- organism-to-genome bindings and parent closure;
|
||||
- lifecycle agreement between the simulation and the archive;
|
||||
- identity high-waters and archived dead members.
|
||||
|
||||
`LineageArchive` (ADR-0034) had no codec and deliberately has no assignment, so
|
||||
an older copy cannot be written back. The M6 environmental fixture has two
|
||||
organism accounts but no genome linkage, and it uses no randomness.
|
||||
|
||||
## Decision
|
||||
**Archive codec v1** (`LineageArchive::encode/decode`, schemas/lineage-archive-v1.md):
|
||||
- Genome v1 records are ordered by (ancestry depth, id), so parents precede
|
||||
children. Organisms follow in id order, as 33-byte records.
|
||||
- A trailing FNV-1a64 detects corruption. The caps are the existing 4096 genomes
|
||||
and 65536 organisms, at most 3,219,496 bytes, checked before allocation.
|
||||
- Decoding constructs a new archive by replaying `admit`, `record_birth` and
|
||||
`record_death`, so retention, parent presence, death finality and tick order are
|
||||
re-established by the archive's own rules. A byte-identical re-encoding is
|
||||
required, so equal content has exactly one encoding.
|
||||
- Decoding never overwrites an existing archive. Refusing an older checkpoint is
|
||||
the committed chain's job (ADR-0061): recovery only ever loads the head.
|
||||
|
||||
**`LineageRegion`** (processes, headless) holds the environmental community, its
|
||||
archive, the binding of the producer and consumer accounts to archived organisms,
|
||||
and the next organism and genome identities.
|
||||
|
||||
*Closure* requires:
|
||||
- both bound organisms are recorded;
|
||||
- each terminal flag equals the archive's death state;
|
||||
- no birth or death is after the current tick;
|
||||
- the archive's living organisms are exactly the bound, non-terminal ones, so
|
||||
there is no unowned living state;
|
||||
- both high-waters are above every recorded identity.
|
||||
|
||||
`advance` records each terminal transition as an archive death at that tick. A
|
||||
checkpoint therefore cannot split a death from the state that caused it. The
|
||||
container is schemas/environmental-lineage-v1.md: header, binding, envelope,
|
||||
archive and FNV.
|
||||
|
||||
**Checkpoint profile 2** is `environmental-lineage-v1`, with history profile 1
|
||||
(archive 1 + genome schema 1). It uses the same nine version slots and zero RNG
|
||||
streams, and its payload cap is 72 + 1 MiB + 3,219,496 bytes. The manifest
|
||||
structure is unchanged, and profile and history profile must agree.
|
||||
|
||||
Schema v4 replaces the SQL parser to admit profile 2 with its cap. Publication
|
||||
additionally requires the predecessor's exact profile, RNG, history and durability
|
||||
declarations. Grants are unchanged. Linux adds
|
||||
`provision/publish/recover_lineage_checkpoint` on the same publication and
|
||||
recovery path as profile 1; each path refuses the other profile.
|
||||
|
||||
## Alternatives Considered
|
||||
- **Extending environmental-v1 with optional sections**: forbidden, because empty
|
||||
sections cannot stand in for missing history.
|
||||
- **Admission-order encoding**: it is not recoverable from the archive and would
|
||||
make equal content encode differently.
|
||||
- **Deaths recorded by the host after the tick** could be split from the state
|
||||
across a checkpoint.
|
||||
- **A separate history object** would need its own atomic publication; one
|
||||
container shares the manifest's digest and fence.
|
||||
|
||||
## Benefits
|
||||
- Retained genomes, ancestry and dead organisms now survive checkpoint recovery
|
||||
and must agree with the simulation state that references them.
|
||||
- Decode reuses the archive's invariants rather than duplicating them.
|
||||
- Existing environmental-v1 regions are unaffected.
|
||||
|
||||
## Costs
|
||||
- The whole archive is rewritten with every batch. A full archive is about 2.5 MiB
|
||||
in the measured test, up to about 3.2 MiB.
|
||||
- Decode is O(genomes²) in the worst case because of sorted insertion.
|
||||
- Archive memory is bounded by the fixed caps rather than charged to a budget.
|
||||
|
||||
## Risks
|
||||
- The environmental fixture has no reproduction, so births within a batch are not
|
||||
exercised. A reproducing profile needs its own closure rules.
|
||||
- Durable history rows (M7.5) must agree with the checkpoint's archive. Only the
|
||||
checkpoint side exists yet.
|
||||
- FNV is not authentication; SHA-256 at the manifest covers the payload.
|
||||
|
||||
## Consequences
|
||||
- M7.5 adds death decisions, history rows and outbox rows derived from these
|
||||
transitions, in the same transaction.
|
||||
- M7.7 recovers this profile after a process kill.
|
||||
- M7.9 retains builds that can decode archive 1 and genome schema 1, and
|
||||
qualifies retained ancestry end to end.
|
||||
- A new genome schema or archive format needs a new history profile value and a
|
||||
migration.
|
||||
@@ -72,3 +72,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0059: PostgreSQL bootstrap](0059-postgresql-bootstrap.md) - accepted for M7.2 optional adapter, pinned server/client, migrations and Secret boundary.
|
||||
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
|
||||
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
|
||||
- [ADR-0062: Lineage checkpoint profile](0062-lineage-checkpoint-profile.md) - accepted for M7.4b archive codec, LineageRegion closure and history-capable profile 2.
|
||||
|
||||
@@ -179,6 +179,10 @@ a local filesystem object store:
|
||||
- Unsupported builds are refused by both the database and recovery.
|
||||
- Recovery never creates genesis or picks an older head.
|
||||
|
||||
Two parts remain open. M7.4b adds the lineage archive codec and a history-capable
|
||||
profile; until then the history rows of this audit stay open. No process-kill
|
||||
test was run yet (7.7).
|
||||
M7.4b (ADR-0062) closes the checkpoint side of the history audit. The lineage
|
||||
archive codec and the `environmental-lineage-v1` profile validate:
|
||||
- organism-to-genome bindings and parent closure (by archive replay);
|
||||
- lifecycle agreement, with deaths recorded in the causing tick;
|
||||
- the living set, identity high-waters and retained dead members.
|
||||
Recovery restores all of these. Durable history rows (M7.5) and process-kill
|
||||
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
|
||||
|
||||
@@ -259,7 +259,8 @@ API. Absence is also reversible: a lineage with no carrier can gain one again.
|
||||
|
||||
Capacity is bounded and admission fails loudly at the limit. Refusing to record is
|
||||
not deleting; a silent drop would be. Durable storage, retention tiers and
|
||||
cross-version migration are M7 and M8 work, and nothing here is serialized. NPC
|
||||
cross-version migration are M7 and M8 work. M7.4b (ADR-0062) serializes the
|
||||
whole archive canonically for history-capable region checkpoints. NPC
|
||||
mutation stays undecided, because the archive records an origin event and never an
|
||||
actor.
|
||||
|
||||
|
||||
@@ -2158,3 +2158,58 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
These are local socket and local disk only.
|
||||
- Not executed: Docker/Gitea CI, a remote object store or PVC, power-loss fsync
|
||||
semantics, read-back verification fault injection, process kill, and TSan.
|
||||
|
||||
## M7.4b lineage checkpoint profile - 2026-09-28
|
||||
|
||||
- Windows MSVC Release CTest passed 57/57, including the new `lineage-region-contracts`
|
||||
and extended lineage and region-checkpoint contracts. The repository clang-format
|
||||
19 check passed.
|
||||
- WSL Ubuntu (GCC 14.2.0, clang 19.1.1, PostgreSQL 18.6, OpenSSL 3.0.13):
|
||||
- The Linux `checks` build set (separate build dirs) passed CTest 57/57 in each
|
||||
of GCC14 Debug, GCC14 Release, clang19 analysis and clang19 ASan/UBSan.
|
||||
- The PostgreSQL integration suite passed 2× Release, 2× analysis and 5×
|
||||
ASan/UBSan.
|
||||
- Archive codec coverage:
|
||||
- canonical encoding regardless of admission order, including a child with a
|
||||
smaller ID than its parent;
|
||||
- checksum, magic, version, count cap, length and organism-count framing;
|
||||
- genome damage, unknown genome, invalid alive/died forms, death before birth,
|
||||
reordered organisms (non-canonical) and a child encoded before its parent;
|
||||
- a full 4096-genome/65536-organism round trip at 2,543,648 bytes.
|
||||
- `LineageRegion` coverage:
|
||||
- closure refusals: missing, identical or dead binding; low high-waters; stray
|
||||
living organism; future birth; archive death while the community is active;
|
||||
- the death recorded at tick 20 in the same advance; a refused tick changes
|
||||
nothing;
|
||||
- restore plus 36 continuation ticks identical to the original;
|
||||
- container, binding, envelope and archive damage refused.
|
||||
- Profile 2 coverage:
|
||||
- the 4,268,144-byte cap is asserted in C++ and enforced by SQL (the maximum is
|
||||
accepted, +1 refused); the profile-1 1 MiB+1 is refused;
|
||||
- mismatched profile/history is EVL06;
|
||||
- lineage publish (death included), recover and continue, and a second
|
||||
publication;
|
||||
- the environmental path refuses a lineage predecessor before upload;
|
||||
- the database refuses a profile switch;
|
||||
- a host of the other profile reports unsupported.
|
||||
- Migrations: clean→v4; v1→v4; v3 with a committed genesis upgraded to v4 with
|
||||
the commit retained; v2 heads still block. Tampered v2/v3/v4 checksums and an
|
||||
unknown v5 are refused.
|
||||
- Mutations (restored afterwards), each failing at the expected line: removing
|
||||
the SQL profile/history continuity check (store test 392), the living-set
|
||||
closure check (lineage region 58), or death recording (lineage region 78).
|
||||
- Flake found and fixed. The ASan PostgreSQL suite failed intermittently: 1 of 3,
|
||||
then 1 of 5 runs, at the queued-writer and expired-renewal assertions.
|
||||
- Cause: a 4-minute probe measured WSL2 stepping CLOCK_REALTIME back 470-596 ms
|
||||
every ~32 s (3.66 s total, roughly 1.5% versus CLOCK_MONOTONIC).
|
||||
- PostgreSQL `clock_timestamp()` follows that wall clock, so a 1.2 s local sleep
|
||||
could be under 1 s of database time.
|
||||
- Tests now poll `clock_timestamp() >= expires_at`, holding the row lock where
|
||||
relevant. Afterwards the suite ran 5/5 under ASan.
|
||||
- The fencing logic itself was unchanged; the requirement is recorded in
|
||||
ADR-0060.
|
||||
- Local measurements, Release:
|
||||
- manifest publish p50 0.72 ms and p99 1.24 ms;
|
||||
- checkpoint publication 3.5-5.0 ms for 23-37 KiB payloads.
|
||||
- Not executed: Docker/Gitea CI, remote storage, process kill, reproduction within
|
||||
a batch (the fixture has none), durable history rows (M7.5) and TSan.
|
||||
|
||||
@@ -107,6 +107,11 @@ Operator steps:
|
||||
connection. It uploads and verifies the tick-0 payload, then calls the
|
||||
owner-only `provision_region_checkpoint(manifest)`.
|
||||
|
||||
Schema v4 (ADR-0062) additionally admits profile 2 (`environmental-lineage-v1`,
|
||||
payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
|
||||
profile, history, RNG and durability declarations. Provision history-capable
|
||||
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
|
||||
|
||||
There is no operator CLI for provisioning yet, and no real world has been
|
||||
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
have no payload.
|
||||
|
||||
Reference in New Issue
Block a user