Implement M5.5 atomic terrain excavation and debris accounting

This commit is contained in:
dingo
2026-09-26 14:56:44 -04:00
parent 25ac646f93
commit 8b5bf48795
19 changed files with 770 additions and 30 deletions
+3 -1
View File
@@ -264,7 +264,9 @@ M5.3 is complete under [ADR-0044](adr/0044-abiotic-generation.md):
[versioned abiotic generation](world/BASE_GENERATION.md) with frozen reference fixtures.
M5.4 is complete under [ADR-0045](adr/0045-bounded-chunk-ownership.md):
[bounded ownership](world/CHUNK_OWNERSHIP.md), compact storage, validated revision tokens
and dirty eviction refusal. Next is 5.5: accounted transactional terrain edits.
and dirty eviction refusal. M5.5 now implements [atomic excavation](world/TERRAIN_EDITS.md)
with per-material volume conservation under [ADR-0046](adr/0046-transactional-excavation.md).
Next is 5.6: canonical snapshots and edit journals.
**Decisions still open:** final spherical layout, region dimensions, adaptive LOD,
field quantization beyond the initial experiment, hydrology and surface extraction.
+10 -2
View File
@@ -7,7 +7,7 @@ persistent simulation server. The founding vision remains unchanged.
## Completed implementation
- Vision, architectural pillars, risks, open-question register, architecture
diagrams, biological contracts, engineering constitution and 45 ADRs.
diagrams, biological contracts, engineering constitution and 46 ADRs.
Proposed/deferred choices remain distinct from accepted proof decisions.
- M0–M4 task breakdowns with dependencies, acceptance evidence and risks,
plus M5–M14 gates in [ROADMAP.md](ROADMAP.md).
@@ -61,7 +61,7 @@ for the bounded proof; the general production ECS remains a later measured choic
| Priority | Work | Completion evidence |
|---|---|---|
| 1 | M5.5 accounted terrain edits | Transaction identity, revisions, matter destinations, atomic changes and retry validation |
| 1 | M5.6 snapshots and edit journals | Bounded canonical format, geometry/debris/retry-state consistency and malformed-input rejection |
| Research | Improve aggregate fidelity and measure Linux/target-host costs | M4.6 found no qualifying width; see the aggregate budget report |
| 2 | Revisit the ADR-0031/0032 reproduction cost model | Reproduction is affordable only at the genome schema's floor; decide whether that is intended before M4 builds cohorts on it |
| Delivery gate | Add/update the dev workload when a deployable increment needs it; Argo is ready | Observe Job/pod completion separately from image publication; no World0 changes |
@@ -568,3 +568,11 @@ Added fixed-slot, budgeted ChunkStore and immutable uniform/palette/dense Compac
Revision tokens reject stale and foreign reads. Clean chunks regenerate; dirty pins
refuse eviction. [Ownership contract](world/CHUNK_OWNERSHIP.md) documents memory/lifetime
limits and the absence of cell edits or durability. Next: M5.5 accounted terrain edits.
## M5.5 transactional terrain excavation
Added up to 64 cuts across four loaded chunks, atomically transferring occupied
volume into finite per-material debris. Revision/digest/full-payload retry checks,
64-command retention and peak allocation backpressure are explicit. The
[edit contract](world/TERRAIN_EDITS.md) records excavation-only scope and persistence
obligations. No mass conversion or durable acknowledgement is claimed. Next: M5.6.
+5
View File
@@ -61,3 +61,8 @@ apply backpressure rather than dropping state. Larger workloads require measurem
Test far-apart chunks, uniform/palette/dense equality, exact budget boundaries,
slot exhaustion, invalid input, stale/foreign views and dirty eviction refusal.
Retain existing generator goldens and simulation formats. Next is M5.5.
## M5.5 follow-up
ADR-0046 adds atomic excavation and fixed replay metadata. Its 32 KiB scratch
reservation supersedes the original 17 KiB allowance; dirty clearing remains absent.
+67
View File
@@ -0,0 +1,67 @@
# ADR-0046: Atomic local excavation and material volume accounting
## Status
Accepted for M5.5 bounded excavation only. Not mass, energy, chemistry or durability.
## Context
Cell fill denotes volume, not kilograms. Excavation must not erase occupied material.
M5.4 owns immutable chunks and rejects dirty eviction; transactions can prepare new
payloads before publishing. Distributed edits and persistence are later boundaries.
## Decision
Implement excavation only, with 1..64 unique ordered cuts touching at most four loaded
chunks in one store. Each cut names chunk, local cell, expected local revision and
positive removed fill numerator. Require lexicographic chunk order then X-fast cell
ordinal, no duplicates. Coordinates, fill bounds, material membership, revision and
source supply are checked. No implicit loading, deposits, conversions or unbounded edits.
One named destination (debris=1) is owned by each store. Creation specifies separate
finite rock/regolith capacities, initially empty; defaults are zero. Credit exactly
removed fill units by material. One unit is 1/65535 cubic metre at the v1 1m pitch.
No density/mass conversion is assumed, and debris is not a second copy of terrain.
Invariant per material: occupied volume + debris volume unchanged by every commit.
M6 must explicitly reconcile these inventories with its material/density/composition
model; no biological ledger femtogram values are fabricated from volume.
Request carries nonzero monotonically increasing command ID, destination, cuts and
server-verified FNV-1a request digest over explicit fields (see contract). This is not
cryptography/authentication. Retain 64 successful requests and receipts, including
full payload equality so hash collisions cannot authorize changed-payload retries.
Same retained ID/exact request returns the original receipt before checking now-stale
revisions; changed payload is rejected. IDs <= high-water mark outside retention
are stale, never executed again. Failed requests consume no ID; no out-of-order commits.
Prepare all compact replacement chunks and validate destination capacity and peak
budget while originals remain owned. Publish geometry, debris, revisions, dirty flags,
high-water mark and replay record in one nonthrowing single-owner phase. Allocation
failure before publish leaves state intact. Every touched chunk gets a new revision
and becomes dirty. No durable acknowledgement or dirty clearing is introduced.
Charge the fixed replay directory plus a conservative 32 KiB scratch allowance
(replacing 17 KiB), and all old plus new payloads at preparation peak. No automatic
cache eviction on memory failure. Caller buffers and allocator overhead remain excluded.
Retain current tokens only by re-query; receipts contain historical revisions.
## Alternatives Considered
A sink destroys matter. A client-owned destination cannot be atomically credited with
private chunk state. Reusing the biological mass ledger would invent a density model.
Generic deposit/replacement edits enlarge the first mutation surface without evidence.
Digest-only replay is unsafe even with a strong hash; exact payloads are bounded here.
## Benefits
Real multi-chunk excavation, conserved typed volume and atomic failure/retry behavior.
No persistence fiction or cross-domain unit conversion.
## Costs
Bounded fixed replay metadata, conservative peak memory, ordered commands and only one
destination. Same-store single owner only. Debris cannot be withdrawn in this gate.
## Risks
Crashes lose this entire in-memory state; M5.6/5.7 must persist debris, revisions,
command high-water/retention and geometry consistently. FNV is not authentication.
Allocation exceptions still require host handling; budgets are not RSS limits.
## Consequences
Test seam-crossing cuts, partial fill, capacity/source/stale failure with no partial
publication, same/changed retries, retention expiry, unknown destination/material,
invalid/duplicate/order/size requests and peak-memory rejection. Keep existing goldens.
+2
View File
@@ -49,3 +49,5 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0044: Abiotic generation](0044-abiotic-generation.md) - accepted for M5.3 synthetic integer base, version retention and material table.
- [ADR-0045: Bounded chunk ownership](0045-bounded-chunk-ownership.md) - accepted for M5.4 local memory, revision tokens and dirty eviction guards.
- [ADR-0046: Transactional excavation](0046-transactional-excavation.md) - accepted for M5.5 atomic local volume transfers and bounded exact-request replay.
+9
View File
@@ -132,3 +132,12 @@ fixed-slot, byte-budgeted local store. [The contract](../world/CHUNK_OWNERSHIP.m
specifies construction scratch accounting, current-revision read tokens, regeneration
and dirty eviction refusal. No database, distributed lease or durable acknowledgement
is implied. Cell edits remain M5.5 and persistence M5.6/5.7.
## Implemented M5.5 edit boundary
[ADR-0046](../adr/0046-transactional-excavation.md) implements bounded single-store
atomic excavation, per-material occupied-volume transfer to debris, current revisions,
request digest plus full-payload retry identity, and bounded history with a command
high-water mark. [The contract](../world/TERRAIN_EDITS.md) specifies failure semantics.
No mass/density model or persistence is implied. Debris/retry state must accompany
geometry in M5.6/5.7 recovery, and M6 must reconcile volume with chemical inventories.
+8
View File
@@ -276,3 +276,11 @@ fingerprint. Conservation holding over a million ticks demonstrates arithmetic,
not ecological plausibility. Strategy constants, lifespans and the targeting rule
are chosen inputs, not derived biology, and must not become gameplay by having
been measured. Real populations still require M4 aggregation.
## M5.5 terrain volume boundary
[Transactional excavation](../world/TERRAIN_EDITS.md) conserves occupied volume by
material between terrain fill and a finite debris inventory. VolumeUnits uses
1/65535 m^3 at the experimental 1m pitch, not femtograms. No conversion into the
biological ledger is allowed until M6 specifies density/composition and reconciles
both representations. Debris is the destination of removed terrain, not new matter.
+30
View File
@@ -1502,3 +1502,33 @@ accounting excludes allocator/runtime overhead and is not an RSS cap.
Linux CI/UBSan and target-host performance have not been run locally. No deployment
or World0 change occurred. Next: M5.5 accounted transactional edits.
## M5.5 transactional excavation - 2026-09-26
Added bounded atomic excavation, finite per-material debris volume, full-payload
retry identity and a 64-command replay ring under ADR-0046. Existing biology/world
base goldens and persisted formats were not changed. Formatter coverage now includes
private headers under src, including the shared chunk-state definitions.
Executed on Windows x64 / MSVC 19.44:
- `tools/windows-check.ps1 -Preset release -Action check`: 44/44 passed, including
the final late-source-failure test and nonthrowing commit static assertions.
- `tools/windows-check.ps1 -Preset analysis -Action check`: clang-tidy 19 build and
44/44 passed. After the final targeted additions, rebuilt analysis and reran
`ctest --test-dir build/analysis -R world-edit-contracts --output-on-failure`: passed.
- `tools/windows-check.ps1 -Preset asan -Action check`: 44/44 passed. After those
additions, rebuilt ASan and reran the same targeted test in build/asan: passed.
- `tools/windows-check.ps1 -Preset release -Action format`: passed.
- `git diff --check`: passed.
Coverage includes seam-crossing excavation, four chunks/64 cuts, partial fills,
rock/regolith volume conservation, all-or-nothing capacity/memory/source failures,
missing/stale revisions, duplicate/changed requests, malformed bounds and retention
expiry. Late source failure verifies an earlier prepared chunk is not published.
No allocator failure injection, revision exhaustion injection or crash recovery is
claimed. The source/replacement commit operations have compile-time nonthrowing guards.
Linux CI and UBSan were not run locally. These are typed volume transfers, not a
validated density/mass/chemistry model. No deployment or World0 modification occurred.
Next: M5.6 canonical snapshots/edit journals preserving geometry, debris and retry state.
+7 -7
View File
@@ -14,7 +14,7 @@ The owning unique_ptr may move, but the store itself cannot move or copy.
| Operation | Success | Refusal / invariant |
|---|---|---|
| minimum_bytes(slots) | Store object + fixed directory + 17 KiB scratch reservation | Invalid slot count rejected before multiplication |
| minimum_bytes(slots) | Store object + fixed directory + replay records + 32 KiB scratch reservation | Invalid slot count rejected before multiplication |
| create | Empty store with validated descriptor | Invalid config/version/budget before directory allocation |
| load(address) | Return current token or regenerate and publish one base chunk | Invalid address, slot or byte exhaustion, revision exhaustion; no existing state changes |
| view(address) | Token for loaded address | Missing is explicit; no implicit load |
@@ -29,7 +29,7 @@ an object later allocated at the same address. They cannot authorize persistence
cross-process operations. Epoch exhaustion is checked before increment; no wraparound.
Dirty marking currently changes only lifecycle metadata, not geometry. There is no
clear-dirty or cell-edit API. M5.5 must introduce accounted edit transactions; M5.6/5.7
clear-dirty API. M5.5 now adds [accounted excavation](TERRAIN_EDITS.md); M5.6/5.7
must establish durable revision coverage before permitting cleanup. Destruction at
host shutdown frees memory and is not a save operation. No production host should
use this in-memory store as its sole durable copy of changes.
@@ -62,8 +62,8 @@ not a wire format. M5.6 must specify canonical serialization independently.
Fixed directory bytes include optional payload objects and lifecycle metadata for
all configured slots, occupied or not. Accounted bytes add exact allocated payload
array element sizes plus a permanent 17 KiB allowance for load construction scratch:
4096 logical words and 256 palette keys. Load generates through point queries to avoid
array element sizes plus a permanent 32 KiB conservative scratch allowance and fixed replay records
under ADR-0046. Loads use 4096 logical words and 256 palette keys. Load generates through point queries to avoid
an extra dense vector, computes required payload before allocation, then publishes
without throwing. On allocator failure existing state is unchanged; allocation
exceptions propagate to the host. Clean eviction subtracts payload only.
@@ -84,6 +84,6 @@ tokens, conservative dirty pinning and rejection without changing resident state
Existing generator goldens remain active. Counter exhaustion is guarded in code but
not forced via a public test-only state injection API.
M5.5 adds transaction IDs, payload identity, expected revisions, material destinations
and atomic edits. No transactional edit, snapshot, crash recovery, threading, automatic
LRU, renderer or Kubernetes integration is provided by M5.4.
M5.5 now adds transaction IDs, full request identity, expected revisions, a finite debris
destination and atomic excavation; see TERRAIN_EDITS.md. Snapshot/crash recovery,
threading, automatic LRU, renderer and Kubernetes integration remain later work.
+96
View File
@@ -0,0 +1,96 @@
# M5.5 transactional excavation
[ADR-0046](../adr/0046-transactional-excavation.md) scopes the first terrain edit to
excavation. Public contracts are in `evo/world/chunks.hpp`; implementation is in
`src/world/edits.cpp`. All calls require one sequential owner and stable request memory
through return. Inputs are intentions, never trusted geometry or authoritative balances.
## Request and atomic scope
An ExcavationRequest names a nonzero command ID, destination 1 (this store's debris),
a server-verified digest and 1..64 CellCuts. Cuts touch at most four loaded chunks in
one store. Each names chunk/local address, expected revision and removed fill in
[1,65535]. Sort lexicographically by chunk x/y/z, then X-fast local ordinal; duplicate
cells and inconsistent revisions within one chunk are rejected. No implicit load or
cross-store/world/region transaction occurs. No deposit, arbitrary replacement,
material transmutation, biological cost or external inventory is implemented.
```mermaid
sequenceDiagram
participant Host
participant Store
participant Prepared
Host->>Store: command + digest + ordered cuts + expected revisions
Store->>Store: Validate shape, digest, replay identity, ownership and revisions
Store->>Prepared: Copy logical cells and stage compact replacements
Store->>Store: Check source volume, debris capacity, old+new peak bytes
alt all checks and allocations succeed
Store->>Store: Publish all chunks + debris + revisions + replay receipt
Store-->>Host: receipt
else any refusal or allocation failure
Store-->>Host: error; original state remains intact
end
```
Published chunks get distinct increasing local revisions and dirty flags. Previous
views become stale; query `view` for current read tokens. The returned receipt lists
historical committed revisions and removed volume by material, not current live views.
Dirty chunks remain unevictable. Mutation of already-dirty chunks is allowed with
current revisions. Geometry and debris commit together without throwing; all fallible
allocations precede publication. Host must handle allocation exceptions. This is
in-process atomicity, not crash consistency, client authentication or a durable ACK.
## Volume conservation and explicit destination
`VolumeUnits{n}` means n/65535 cubic metres at the v1 1m pitch. A CellSample's fill
is already in these units. DebrisCapacity specifies independent uint64 capacities for
rock and regolith at store creation; defaults are zero and balances start empty.
Unknown destination IDs are rejected. No public capacity change, withdrawal or balance
setter exists. Material keys alone never grant debris supply.
For each material m and committed command:
`sum(terrain fill[m])_after + debris[m]_after = sum(terrain fill[m])_before + debris[m]_before`.
Partial excavation reduces fill with the same material ID. Removing the full fill
produces structurally valid void. Rock cannot credit regolith or vice versa. Compare
credits to capacity minus balance before adding, avoiding unsigned overflow. Bounded
commands remove at most 64*65535 units. Geometry of the removed sub-cell portion is
unknown; no collision shape or smooth surface is claimed.
These inventories are volume, not kg/femtograms. M6 must define density/composition
and reconcile them without double-counting before coupling to biology. No value in
the existing biological ledger is reinterpreted or modified by this gate.
## Retry identity and compatibility
The [request digest schema](../../schemas/world-edits.md) defines the explicit byte
order. FNV-1a is an accidental-corruption/identity aid, never security. Retained command
IDs compare all original fields as well as digest, so collisions cannot authorize a
changed request. For valid request shape and digest:
- Same retained ID and exact payload returns the original receipt without reapplying,
even after newer edits made its expected revisions stale.
- Same retained ID with different payload returns command_conflict.
- An ID <= the high-water mark with no retained record returns stale_command.
- A fresh ID must exceed the high-water mark; gaps are permitted. Failures consume no
ID and mutate no cells, debris, revision, dirty flag or history record.
Only the last 64 successful commands are retained in a fixed ring. This is bounded
local retry history, not event retention. M5.6/5.7 must persist geometry, debris,
capacities, revisions, command high-water and replay records consistently. A restart
that retains only geometry would duplicate or lose material and is not supported.
## Memory and tests
minimum_bytes now includes the fixed 64-record replay array and 32 KiB conservative
scratch reservation, superseding M5.4's 17 KiB allowance. New replacement payloads
must fit beside all old payloads; old storage is released only during commit. Unused
replay fields remain bounded metadata. No cache eviction occurs to accommodate edits.
Allocator/runtime overhead and caller buffers remain outside accounted bytes.
CTest world-edit-contracts covers a seam crossing, all four chunks/64 cuts, partial
fill, rock/regolith conservation, destination and second-replacement memory failures,
missing/stale/source failures, dirty eviction refusal, malformed request bounds,
same/changed retries and expiry after 65 commands. Existing generator and simulation
goldens remain active. Allocator fault injection and crash recovery remain future work.