Implement M5.6 canonical world snapshots and edit journals

This commit is contained in:
dingo
2026-09-26 19:05:29 -04:00
parent 8b5bf48795
commit 207f9e6953
22 changed files with 1262 additions and 9 deletions
+3 -1
View File
@@ -266,7 +266,9 @@ M5.4 is complete under [ADR-0045](adr/0045-bounded-chunk-ownership.md):
[bounded ownership](world/CHUNK_OWNERSHIP.md), compact storage, validated revision tokens
and dirty eviction refusal. M5.5 now implements [atomic excavation](world/TERRAIN_EDITS.md)
with per-material volume conservation under [ADR-0046](adr/0046-transactional-excavation.md).
Next is 5.6: canonical snapshots and edit journals.
M5.6 is complete under [ADR-0047](adr/0047-world-codec.md):
[canonical snapshots and edit journals](world/WORLD_CODEC.md) validate geometry, debris
and retry state together. Next is 5.7: local publication, compaction and recovery.
**Decisions still open:** final spherical layout, region dimensions, adaptive LOD,
field quantization beyond the initial experiment, hydrology and surface extraction.
+10 -2
View File
@@ -7,7 +7,7 @@ persistent simulation server. The founding vision remains unchanged.
## Completed implementation
- Vision, architectural pillars, risks, open-question register, architecture
diagrams, biological contracts, engineering constitution and 46 ADRs.
diagrams, biological contracts, engineering constitution and 47 ADRs.
Proposed/deferred choices remain distinct from accepted proof decisions.
- M0–M4 task breakdowns with dependencies, acceptance evidence and risks,
plus M5–M14 gates in [ROADMAP.md](ROADMAP.md).
@@ -61,7 +61,7 @@ for the bounded proof; the general production ECS remains a later measured choic
| Priority | Work | Completion evidence |
|---|---|---|
| 1 | M5.6 snapshots and edit journals | Bounded canonical format, geometry/debris/retry-state consistency and malformed-input rejection |
| 1 | M5.7 local publication and compaction | Verified snapshot/journal replacement, recovery and injected filesystem failures |
| Research | Improve aggregate fidelity and measure Linux/target-host costs | M4.6 found no qualifying width; see the aggregate budget report |
| 2 | Revisit the ADR-0031/0032 reproduction cost model | Reproduction is affordable only at the genome schema's floor; decide whether that is intended before M4 builds cohorts on it |
| Delivery gate | Add/update the dev workload when a deployable increment needs it; Argo is ready | Observe Job/pod completion separately from image publication; no World0 changes |
@@ -576,3 +576,11 @@ volume into finite per-material debris. Revision/digest/full-payload retry check
64-command retention and peak allocation backpressure are explicit. The
[edit contract](world/TERRAIN_EDITS.md) records excavation-only scope and persistence
obligations. No mass conversion or durable acknowledgement is claimed. Next: M5.6.
## M5.6 snapshots and edit journals
Added bounded canonical world snapshot v1 and edit journal v1, world/frame and base
version checks, full geometry/debris/retry restore, semantic validation and detached
all-or-nothing replay. [Codec contract](world/WORLD_CODEC.md) and schemas define
memory/trust limits. Dirty flags remain set; no file publication or durable ACK is
implemented. Next: M5.7 local publication, compaction and recovery.
+61
View File
@@ -0,0 +1,61 @@
# ADR-0047: Bounded world snapshot and edit-journal codecs
## Status
Accepted for M5.6 local binary buffers. File publication/crash consistency is M5.7.
## Context
Geometry, debris and retry state form one consistency boundary. Host allocation
budgets are runtime policy, not saved physical laws. Unknown versions fail closed.
## Decision
Implement explicit little-endian snapshot v1 and journal v1; no native object dumps.
Bind nonzero world/frame IDs supplied by the host and retain all base versions.
Snapshots sort chunks by address and successful replay records by command ID, record
slot capacity, revision high-water, command high-water, successful edit count,
debris capacities/balances, dirty state, logical samples and full retained requests
and receipts. Covered sequence is the command high-water (IDs may have gaps).
Use logical cells for the initial codec; resident compression is independent.
Bounds: 4096 slots, 64 replay records, 70 MiB snapshot, 256 KiB journal, at most 64
journal commands. Total length and FNV-1a corruption checksum precede semantic
acceptance. FNV is not authenticity or a security primitive. The host must provide
trusted provenance for durable data; future production manifests need strong digests.
Input/output buffers and bounded codec workspace are separate from resident budget.
No unchecked length controls allocation or iteration. Restore takes an explicit host
byte budget, builds an unpublished store and validates all state before return.
Validate cell structure/materials against the recorded procedural base: excavation
cannot change material, add fill, or alter clean chunks. Removed volume across resident
chunks must equal debris exactly. All changed chunks remain dirty/resident. Validate
replay shape/digest/receipt amounts, chronology, current revision linkage and reverse
volume feasibility for retained commands. Persist successful edit count to distinguish
a full bounded replay window from accidentally dropped retry records.
Journals contain predecessor snapshot checksum, base command high-water, ordered full
fresh edit requests and expected final snapshot checksum. Replay restores a detached
base then applies ordinary atomic excavation, verifies the final canonical snapshot,
and publishes only success. No partial replay escapes. Journal construction requires
already-loaded chunks and no intervening load/evict/pin operations; those require a
new snapshot. Empty journals are valid. Retry requests are not fresh journal events.
## Alternatives Considered
Native structs are ABI-dependent. Geometry-only snapshots lose conservation/retry
state. Reconstructing everything from 64 retained requests loses older edits. A full
database/event-sourcing backend is outside this gate. Compression is deferred.
## Benefits
Canonical round trips, strict version/identity boundaries and real snapshot+edit replay.
## Costs
Dense logical wire cells and full exact retry payloads trade bytes for simple validation.
Restore recomputes base geometry and validates history; this is not a latency SLA.
## Risks
Checksums do not stop malicious forgery. This codec does not fsync, atomically publish,
clear dirty flags, authenticate a client or acknowledge permanent outcomes durably.
Allocation exceptions propagate to the host; failed candidates are discarded.
## Consequences
Test canonical round trips, post-restore continuation/retries, journal chaining,
truncation/corruption/oversize/version/identity/budget and semantic rejections with
recomputed checksums. Add a bounded fuzz entry point; preserve existing goldens.
+2
View File
@@ -51,3 +51,5 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0045: Bounded chunk ownership](0045-bounded-chunk-ownership.md) - accepted for M5.4 local memory, revision tokens and dirty eviction guards.
- [ADR-0046: Transactional excavation](0046-transactional-excavation.md) - accepted for M5.5 atomic local volume transfers and bounded exact-request replay.
- [ADR-0047: World codecs](0047-world-codec.md) - accepted for M5.6 bounded canonical snapshots and detached edit-journal replay.
@@ -142,3 +142,11 @@ host loss. Exact guarantees for acknowledged outcomes require an explicitly
verified durable failure domain containing the complete causal recovery chain
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
permadeath. That admission/DR requirement is a World0 launch gate.
## Implemented M5.6 local world codec boundary
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
command high-water, resident cells, debris and exact retry state. Detached replay
verifies predecessor and final state. This is in-memory codec validation, not the
production durability contract above. File publication/crash recovery is M5.7.
+8
View File
@@ -141,3 +141,11 @@ request digest plus full-payload retry identity, and bounded history with a comm
high-water mark. [The contract](../world/TERRAIN_EDITS.md) specifies failure semantics.
No mass/density model or persistence is implied. Debris/retry state must accompany
geometry in M5.6/5.7 recovery, and M6 must reconcile volume with chemical inventories.
## Implemented M5.6 snapshot/replay boundary
The [world codec](../world/WORLD_CODEC.md) preserves loaded membership, dirty cells,
versions, revisions, finite debris and retry history in one validated snapshot.
Journal replay uses the existing excavation transaction on a detached store. Dirty
chunks remain pinned; encoding does not acknowledge durability. M5.7 supplies local
file publication and compaction; no production persistence dependency was added.
+35
View File
@@ -1532,3 +1532,38 @@ claimed. The source/replacement commit operations have compile-time nonthrowing
Linux CI and UBSan were not run locally. These are typed volume transfers, not a
validated density/mass/chemistry model. No deployment or World0 modification occurred.
Next: M5.6 canonical snapshots/edit journals preserving geometry, debris and retry state.
## M5.6 canonical world snapshots and edit journals - 2026-09-26
Added ADR-0047, byte schemas, canonical snapshot/restore and detached atomic journal
replay. State includes base versions, identity, membership, revisions, geometry,
debris, successful edit count and exact bounded retry records. No existing golden
or biological simulation format changed. New snapshot v1 has an independently encoded
132-byte empty fixture from tools/world_snapshot_reference.py.
Executed on Windows x64 / MSVC 19.44:
- Release configure/build and 46/46 CTest passed on the final source/tests.
- Analysis configure/build (clang-tidy 19) and 46/46 CTest passed after explicit
size/iterator conversion fixes. Final history-corruption additions rebuilt and
passed both tests selected by `ctest --test-dir build/analysis -R world-codec
--output-on-failure`.
- ASan configure/build and 46/46 CTest passed. After final size/iterator/golden and
history-corruption additions, rebuilt ASan and reran both world-codec tests: passed.
- `tools/windows-check.ps1 -Preset release -Action format`: passed.
- `python tools/world_snapshot_reference.py`: generated independent golden bytes;
encoder equality and decoder acceptance pass in CTest.
- `git diff --check`: passed.
New coverage includes 65-edit history rollover, deterministic continuation, historical
retry receipts, exact re-encoding, snapshot+journal replay, wrong predecessor/final
state, every fixture truncation, oversized/trailing data, version/identity/budget
rejection, recomputed-checksum geometry/debris/receipt/revision/history corruption,
and a deterministic 128-input semantic fuzz smoke. The fuzz harness mutates valid
fixtures and repairs checksums so validation below the envelope is exercised.
Linux libFuzzer/UBSan jobs are wired for 10000 cases but were not executed locally.
Native Windows ASan is not UBSan. Full-size throughput, allocation-failure injection
and crash-atomic files remain unverified/later gates. Codecs write no files, clear no
dirty flags and issue no durable acknowledgements. No Argo/World0 operation occurred.
Next: M5.7 local snapshot/journal publication, compaction and recovery.
+82
View File
@@ -0,0 +1,82 @@
# M5.6 world snapshot and edit-journal contract
[ADR-0047](../adr/0047-world-codec.md) accepts bounded local byte codecs. Public API
is in evo/world/chunks.hpp; evo::world remains free of database, networking, rendering
and Kubernetes dependencies. No existing biology/particle snapshot format changed.
## API and ownership
- `store.snapshot(identity)` returns canonical bytes for the current coherent store.
Caller supplies nonzero world/frame IDs and owns their association with this store.
- `ChunkStore::restore(bytes, expected_identity, byte_budget)` returns a new owning
store only on full validation. Never replaces an existing live store in-place.
- `make_edit_journal(base, identity, requests, byte_budget)` validates/restores a base,
executes up to 64 fresh requests privately and encodes their expected result.
- `replay_edit_journal(base, journal, identity, byte_budget)` restores the base, applies
the whole bounded journal and verifies the final state before returning a new store.
All are single-owner operations; input spans remain immutable/readable through return.
Use typed WorldCodecError for invalid input/budget/state. Allocation exceptions reach
the host. Views of the source store never become valid views of the restored store.
Budget is host policy, supplied explicitly and excluded from canonical state. Restore
preserves slot capacity, versions, revisions, successful edit count, debris and exact
bounded replay history. The snapshot covered sequence is command high-water, not a
tick or contiguous ordinal. Journal requests cannot contain retries or cache operations.
```mermaid
flowchart LR
S[Canonical snapshot] --> V[Envelope and semantic validation]
V --> C[Detached compact store]
J[Ordered journal] --> R[Predecessor and command validation]
C --> R
R --> H[Final canonical checksum verification]
H --> O[Return owning candidate]
```
The snapshot carries all loaded chunks including clean ones because loaded membership
and revisions affect admitted edits. Unloaded clean regions remain procedural. Dirty
chunks cannot be evicted; all removed material therefore remains accountable against
resident geometry. No implicit clean/regenerate operation discards edits.
## Version and conservation rules
[Snapshot schema](../../schemas/world-snapshot.md) and
[journal schema](../../schemas/world-journal.md) specify every byte and bound.
Unknown versions, identities, materials and malformed/trailing data fail closed.
Restore compares cells against deterministic generation: no new fill or changed
material; clean chunks must match base. Summed removed volume equals stored debris
per material. Replay receipts and revision chains are checked; no digest-only retry.
All retained history can be undone without exceeding original base fill.
FNV is an accidental-corruption check, not security. Untrusted/unauthenticated world
state is not authorized by a correct checksum. M5.7 and production infrastructure
must establish trusted manifests and complete journal coverage. Creating bytes alone
is neither durable acknowledgement nor permission to clear dirty flags.
## Resource bounds and limitations
Snapshot <=70 MiB, journal <=256 KiB, slots <=4096, retained/journal commands <=64,
cuts <=64 each, chunks per edit <=4. Restore validates framing/count bounds before
allocation and enforces the candidate's resident budget for metadata and payloads.
Codec input/output buffers are outside that budget: the host must separately admit
up to one input and one output snapshot plus journal and fixed codec workspace.
Largest explicit encoding workspace is a 4096-entry index array plus a 64-entry
history order array; restoration uses one 4096-word chunk and bounded history scans.
Reserve 64 KiB for codec workspace in addition to ordinary runtime/allocator overhead.
No compression or performance SLA is claimed; full restore recomputes base geometry
and checks at most 4096 retained cuts against later retained cuts.
## Evidence and next step
Tests cover independently encoded empty v1 bytes, canonical round trips, 65 edits
and history wrap, post-restore continuation/retries, seam journal replay, empty
segments, every truncation of fixtures, checksums, recomputed-checksum semantic
corruption, unknown versions, wrong identity, counts, trailing/oversized input and
insufficient memory. A 128-input deterministic fuzz smoke is in CTest; Linux CI adds
10000 libFuzzer cases. The harness repairs mutated checksums to exercise semantic
validation instead of stopping only at the envelope. Local Windows smoke is not a
claim that Linux coverage-guided fuzzing has executed.
Next M5.7: publish/compact local files with verified replacement and failure injection.
M5.6 does not write files, fsync, delete journals, clear dirty state or recover a live
server process. M7 supplies the production durability/ownership path.