Implement M5.6 canonical world snapshots and edit journals
This commit is contained in:
+3
-1
@@ -266,7 +266,9 @@ M5.4 is complete under [ADR-0045](adr/0045-bounded-chunk-ownership.md):
|
||||
[bounded ownership](world/CHUNK_OWNERSHIP.md), compact storage, validated revision tokens
|
||||
and dirty eviction refusal. M5.5 now implements [atomic excavation](world/TERRAIN_EDITS.md)
|
||||
with per-material volume conservation under [ADR-0046](adr/0046-transactional-excavation.md).
|
||||
Next is 5.6: canonical snapshots and edit journals.
|
||||
M5.6 is complete under [ADR-0047](adr/0047-world-codec.md):
|
||||
[canonical snapshots and edit journals](world/WORLD_CODEC.md) validate geometry, debris
|
||||
and retry state together. Next is 5.7: local publication, compaction and recovery.
|
||||
|
||||
**Decisions still open:** final spherical layout, region dimensions, adaptive LOD,
|
||||
field quantization beyond the initial experiment, hydrology and surface extraction.
|
||||
|
||||
+10
-2
@@ -7,7 +7,7 @@ persistent simulation server. The founding vision remains unchanged.
|
||||
## Completed implementation
|
||||
|
||||
- Vision, architectural pillars, risks, open-question register, architecture
|
||||
diagrams, biological contracts, engineering constitution and 46 ADRs.
|
||||
diagrams, biological contracts, engineering constitution and 47 ADRs.
|
||||
Proposed/deferred choices remain distinct from accepted proof decisions.
|
||||
- M0–M4 task breakdowns with dependencies, acceptance evidence and risks,
|
||||
plus M5–M14 gates in [ROADMAP.md](ROADMAP.md).
|
||||
@@ -61,7 +61,7 @@ for the bounded proof; the general production ECS remains a later measured choic
|
||||
|
||||
| Priority | Work | Completion evidence |
|
||||
|---|---|---|
|
||||
| 1 | M5.6 snapshots and edit journals | Bounded canonical format, geometry/debris/retry-state consistency and malformed-input rejection |
|
||||
| 1 | M5.7 local publication and compaction | Verified snapshot/journal replacement, recovery and injected filesystem failures |
|
||||
| Research | Improve aggregate fidelity and measure Linux/target-host costs | M4.6 found no qualifying width; see the aggregate budget report |
|
||||
| 2 | Revisit the ADR-0031/0032 reproduction cost model | Reproduction is affordable only at the genome schema's floor; decide whether that is intended before M4 builds cohorts on it |
|
||||
| Delivery gate | Add/update the dev workload when a deployable increment needs it; Argo is ready | Observe Job/pod completion separately from image publication; no World0 changes |
|
||||
@@ -576,3 +576,11 @@ volume into finite per-material debris. Revision/digest/full-payload retry check
|
||||
64-command retention and peak allocation backpressure are explicit. The
|
||||
[edit contract](world/TERRAIN_EDITS.md) records excavation-only scope and persistence
|
||||
obligations. No mass conversion or durable acknowledgement is claimed. Next: M5.6.
|
||||
|
||||
## M5.6 snapshots and edit journals
|
||||
|
||||
Added bounded canonical world snapshot v1 and edit journal v1, world/frame and base
|
||||
version checks, full geometry/debris/retry restore, semantic validation and detached
|
||||
all-or-nothing replay. [Codec contract](world/WORLD_CODEC.md) and schemas define
|
||||
memory/trust limits. Dirty flags remain set; no file publication or durable ACK is
|
||||
implemented. Next: M5.7 local publication, compaction and recovery.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# ADR-0047: Bounded world snapshot and edit-journal codecs
|
||||
|
||||
## Status
|
||||
Accepted for M5.6 local binary buffers. File publication/crash consistency is M5.7.
|
||||
|
||||
## Context
|
||||
Geometry, debris and retry state form one consistency boundary. Host allocation
|
||||
budgets are runtime policy, not saved physical laws. Unknown versions fail closed.
|
||||
|
||||
## Decision
|
||||
Implement explicit little-endian snapshot v1 and journal v1; no native object dumps.
|
||||
Bind nonzero world/frame IDs supplied by the host and retain all base versions.
|
||||
Snapshots sort chunks by address and successful replay records by command ID, record
|
||||
slot capacity, revision high-water, command high-water, successful edit count,
|
||||
debris capacities/balances, dirty state, logical samples and full retained requests
|
||||
and receipts. Covered sequence is the command high-water (IDs may have gaps).
|
||||
Use logical cells for the initial codec; resident compression is independent.
|
||||
|
||||
Bounds: 4096 slots, 64 replay records, 70 MiB snapshot, 256 KiB journal, at most 64
|
||||
journal commands. Total length and FNV-1a corruption checksum precede semantic
|
||||
acceptance. FNV is not authenticity or a security primitive. The host must provide
|
||||
trusted provenance for durable data; future production manifests need strong digests.
|
||||
Input/output buffers and bounded codec workspace are separate from resident budget.
|
||||
No unchecked length controls allocation or iteration. Restore takes an explicit host
|
||||
byte budget, builds an unpublished store and validates all state before return.
|
||||
|
||||
Validate cell structure/materials against the recorded procedural base: excavation
|
||||
cannot change material, add fill, or alter clean chunks. Removed volume across resident
|
||||
chunks must equal debris exactly. All changed chunks remain dirty/resident. Validate
|
||||
replay shape/digest/receipt amounts, chronology, current revision linkage and reverse
|
||||
volume feasibility for retained commands. Persist successful edit count to distinguish
|
||||
a full bounded replay window from accidentally dropped retry records.
|
||||
|
||||
Journals contain predecessor snapshot checksum, base command high-water, ordered full
|
||||
fresh edit requests and expected final snapshot checksum. Replay restores a detached
|
||||
base then applies ordinary atomic excavation, verifies the final canonical snapshot,
|
||||
and publishes only success. No partial replay escapes. Journal construction requires
|
||||
already-loaded chunks and no intervening load/evict/pin operations; those require a
|
||||
new snapshot. Empty journals are valid. Retry requests are not fresh journal events.
|
||||
|
||||
## Alternatives Considered
|
||||
Native structs are ABI-dependent. Geometry-only snapshots lose conservation/retry
|
||||
state. Reconstructing everything from 64 retained requests loses older edits. A full
|
||||
database/event-sourcing backend is outside this gate. Compression is deferred.
|
||||
|
||||
## Benefits
|
||||
Canonical round trips, strict version/identity boundaries and real snapshot+edit replay.
|
||||
|
||||
## Costs
|
||||
Dense logical wire cells and full exact retry payloads trade bytes for simple validation.
|
||||
Restore recomputes base geometry and validates history; this is not a latency SLA.
|
||||
|
||||
## Risks
|
||||
Checksums do not stop malicious forgery. This codec does not fsync, atomically publish,
|
||||
clear dirty flags, authenticate a client or acknowledge permanent outcomes durably.
|
||||
Allocation exceptions propagate to the host; failed candidates are discarded.
|
||||
|
||||
## Consequences
|
||||
Test canonical round trips, post-restore continuation/retries, journal chaining,
|
||||
truncation/corruption/oversize/version/identity/budget and semantic rejections with
|
||||
recomputed checksums. Add a bounded fuzz entry point; preserve existing goldens.
|
||||
@@ -51,3 +51,5 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0045: Bounded chunk ownership](0045-bounded-chunk-ownership.md) - accepted for M5.4 local memory, revision tokens and dirty eviction guards.
|
||||
|
||||
- [ADR-0046: Transactional excavation](0046-transactional-excavation.md) - accepted for M5.5 atomic local volume transfers and bounded exact-request replay.
|
||||
|
||||
- [ADR-0047: World codecs](0047-world-codec.md) - accepted for M5.6 bounded canonical snapshots and detached edit-journal replay.
|
||||
|
||||
@@ -142,3 +142,11 @@ host loss. Exact guarantees for acknowledged outcomes require an explicitly
|
||||
verified durable failure domain containing the complete causal recovery chain
|
||||
and referenced payloads before acknowledgment; restoring an older backup cannot silently undo
|
||||
permadeath. That admission/DR requirement is a World0 launch gate.
|
||||
|
||||
## Implemented M5.6 local world codec boundary
|
||||
|
||||
[ADR-0047](../adr/0047-world-codec.md) and [the codec contract](../world/WORLD_CODEC.md)
|
||||
implement bounded canonical buffers carrying world/frame IDs, base versions, covered
|
||||
command high-water, resident cells, debris and exact retry state. Detached replay
|
||||
verifies predecessor and final state. This is in-memory codec validation, not the
|
||||
production durability contract above. File publication/crash recovery is M5.7.
|
||||
|
||||
@@ -141,3 +141,11 @@ request digest plus full-payload retry identity, and bounded history with a comm
|
||||
high-water mark. [The contract](../world/TERRAIN_EDITS.md) specifies failure semantics.
|
||||
No mass/density model or persistence is implied. Debris/retry state must accompany
|
||||
geometry in M5.6/5.7 recovery, and M6 must reconcile volume with chemical inventories.
|
||||
|
||||
## Implemented M5.6 snapshot/replay boundary
|
||||
|
||||
The [world codec](../world/WORLD_CODEC.md) preserves loaded membership, dirty cells,
|
||||
versions, revisions, finite debris and retry history in one validated snapshot.
|
||||
Journal replay uses the existing excavation transaction on a detached store. Dirty
|
||||
chunks remain pinned; encoding does not acknowledge durability. M5.7 supplies local
|
||||
file publication and compaction; no production persistence dependency was added.
|
||||
|
||||
@@ -1532,3 +1532,38 @@ claimed. The source/replacement commit operations have compile-time nonthrowing
|
||||
Linux CI and UBSan were not run locally. These are typed volume transfers, not a
|
||||
validated density/mass/chemistry model. No deployment or World0 modification occurred.
|
||||
Next: M5.6 canonical snapshots/edit journals preserving geometry, debris and retry state.
|
||||
|
||||
## M5.6 canonical world snapshots and edit journals - 2026-09-26
|
||||
|
||||
Added ADR-0047, byte schemas, canonical snapshot/restore and detached atomic journal
|
||||
replay. State includes base versions, identity, membership, revisions, geometry,
|
||||
debris, successful edit count and exact bounded retry records. No existing golden
|
||||
or biological simulation format changed. New snapshot v1 has an independently encoded
|
||||
132-byte empty fixture from tools/world_snapshot_reference.py.
|
||||
|
||||
Executed on Windows x64 / MSVC 19.44:
|
||||
|
||||
- Release configure/build and 46/46 CTest passed on the final source/tests.
|
||||
- Analysis configure/build (clang-tidy 19) and 46/46 CTest passed after explicit
|
||||
size/iterator conversion fixes. Final history-corruption additions rebuilt and
|
||||
passed both tests selected by `ctest --test-dir build/analysis -R world-codec
|
||||
--output-on-failure`.
|
||||
- ASan configure/build and 46/46 CTest passed. After final size/iterator/golden and
|
||||
history-corruption additions, rebuilt ASan and reran both world-codec tests: passed.
|
||||
- `tools/windows-check.ps1 -Preset release -Action format`: passed.
|
||||
- `python tools/world_snapshot_reference.py`: generated independent golden bytes;
|
||||
encoder equality and decoder acceptance pass in CTest.
|
||||
- `git diff --check`: passed.
|
||||
|
||||
New coverage includes 65-edit history rollover, deterministic continuation, historical
|
||||
retry receipts, exact re-encoding, snapshot+journal replay, wrong predecessor/final
|
||||
state, every fixture truncation, oversized/trailing data, version/identity/budget
|
||||
rejection, recomputed-checksum geometry/debris/receipt/revision/history corruption,
|
||||
and a deterministic 128-input semantic fuzz smoke. The fuzz harness mutates valid
|
||||
fixtures and repairs checksums so validation below the envelope is exercised.
|
||||
|
||||
Linux libFuzzer/UBSan jobs are wired for 10000 cases but were not executed locally.
|
||||
Native Windows ASan is not UBSan. Full-size throughput, allocation-failure injection
|
||||
and crash-atomic files remain unverified/later gates. Codecs write no files, clear no
|
||||
dirty flags and issue no durable acknowledgements. No Argo/World0 operation occurred.
|
||||
Next: M5.7 local snapshot/journal publication, compaction and recovery.
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
# M5.6 world snapshot and edit-journal contract
|
||||
|
||||
[ADR-0047](../adr/0047-world-codec.md) accepts bounded local byte codecs. Public API
|
||||
is in evo/world/chunks.hpp; evo::world remains free of database, networking, rendering
|
||||
and Kubernetes dependencies. No existing biology/particle snapshot format changed.
|
||||
|
||||
## API and ownership
|
||||
|
||||
- `store.snapshot(identity)` returns canonical bytes for the current coherent store.
|
||||
Caller supplies nonzero world/frame IDs and owns their association with this store.
|
||||
- `ChunkStore::restore(bytes, expected_identity, byte_budget)` returns a new owning
|
||||
store only on full validation. Never replaces an existing live store in-place.
|
||||
- `make_edit_journal(base, identity, requests, byte_budget)` validates/restores a base,
|
||||
executes up to 64 fresh requests privately and encodes their expected result.
|
||||
- `replay_edit_journal(base, journal, identity, byte_budget)` restores the base, applies
|
||||
the whole bounded journal and verifies the final state before returning a new store.
|
||||
|
||||
All are single-owner operations; input spans remain immutable/readable through return.
|
||||
Use typed WorldCodecError for invalid input/budget/state. Allocation exceptions reach
|
||||
the host. Views of the source store never become valid views of the restored store.
|
||||
Budget is host policy, supplied explicitly and excluded from canonical state. Restore
|
||||
preserves slot capacity, versions, revisions, successful edit count, debris and exact
|
||||
bounded replay history. The snapshot covered sequence is command high-water, not a
|
||||
tick or contiguous ordinal. Journal requests cannot contain retries or cache operations.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
S[Canonical snapshot] --> V[Envelope and semantic validation]
|
||||
V --> C[Detached compact store]
|
||||
J[Ordered journal] --> R[Predecessor and command validation]
|
||||
C --> R
|
||||
R --> H[Final canonical checksum verification]
|
||||
H --> O[Return owning candidate]
|
||||
```
|
||||
|
||||
The snapshot carries all loaded chunks including clean ones because loaded membership
|
||||
and revisions affect admitted edits. Unloaded clean regions remain procedural. Dirty
|
||||
chunks cannot be evicted; all removed material therefore remains accountable against
|
||||
resident geometry. No implicit clean/regenerate operation discards edits.
|
||||
|
||||
## Version and conservation rules
|
||||
|
||||
[Snapshot schema](../../schemas/world-snapshot.md) and
|
||||
[journal schema](../../schemas/world-journal.md) specify every byte and bound.
|
||||
Unknown versions, identities, materials and malformed/trailing data fail closed.
|
||||
Restore compares cells against deterministic generation: no new fill or changed
|
||||
material; clean chunks must match base. Summed removed volume equals stored debris
|
||||
per material. Replay receipts and revision chains are checked; no digest-only retry.
|
||||
All retained history can be undone without exceeding original base fill.
|
||||
|
||||
FNV is an accidental-corruption check, not security. Untrusted/unauthenticated world
|
||||
state is not authorized by a correct checksum. M5.7 and production infrastructure
|
||||
must establish trusted manifests and complete journal coverage. Creating bytes alone
|
||||
is neither durable acknowledgement nor permission to clear dirty flags.
|
||||
|
||||
## Resource bounds and limitations
|
||||
|
||||
Snapshot <=70 MiB, journal <=256 KiB, slots <=4096, retained/journal commands <=64,
|
||||
cuts <=64 each, chunks per edit <=4. Restore validates framing/count bounds before
|
||||
allocation and enforces the candidate's resident budget for metadata and payloads.
|
||||
Codec input/output buffers are outside that budget: the host must separately admit
|
||||
up to one input and one output snapshot plus journal and fixed codec workspace.
|
||||
Largest explicit encoding workspace is a 4096-entry index array plus a 64-entry
|
||||
history order array; restoration uses one 4096-word chunk and bounded history scans.
|
||||
Reserve 64 KiB for codec workspace in addition to ordinary runtime/allocator overhead.
|
||||
No compression or performance SLA is claimed; full restore recomputes base geometry
|
||||
and checks at most 4096 retained cuts against later retained cuts.
|
||||
|
||||
## Evidence and next step
|
||||
|
||||
Tests cover independently encoded empty v1 bytes, canonical round trips, 65 edits
|
||||
and history wrap, post-restore continuation/retries, seam journal replay, empty
|
||||
segments, every truncation of fixtures, checksums, recomputed-checksum semantic
|
||||
corruption, unknown versions, wrong identity, counts, trailing/oversized input and
|
||||
insufficient memory. A 128-input deterministic fuzz smoke is in CTest; Linux CI adds
|
||||
10000 libFuzzer cases. The harness repairs mutated checksums to exercise semantic
|
||||
validation instead of stopping only at the envelope. Local Windows smoke is not a
|
||||
claim that Linux coverage-guided fuzzing has executed.
|
||||
|
||||
Next M5.7: publish/compact local files with verified replacement and failure injection.
|
||||
M5.6 does not write files, fsync, delete journals, clear dirty state or recover a live
|
||||
server process. M7 supplies the production durability/ownership path.
|
||||
Reference in New Issue
Block a user