Complete M7.5 durable batch outcomes and commit resolution

ADR-0063: manifest v2 (360 bytes) binds a per-tick validation trace
object and a canonical decision block by SHA-256. Publication replays the
trace from the committed predecessor's verified payload and requires
every report and the candidate bytes, derives deaths deterministically,
uploads payload and trace with verified read-back, then sends manifest
and decisions to one fenced statement. RegionBatch seals at a terminal
transition or 64 ticks and refuses ticks until durable confirmation.

Schema v5 records the checkpoint, its decisions and outbox rows
atomically; a unique index keeps one terminal outcome per actor
(EVL08), triggers make commits/decisions immutable (EVL09), an exact
retry of a committed ID returns the original receipt, changed bytes
conflict, and read-only resolve_region_commit answers committed/absent/
conflict. Outbox pending/acknowledge give idempotent at-least-once
delivery. Recovery reconciles durable deaths with restored lifecycle
(and archive deaths for the lineage profile).

Validated: Windows CTest 58/58; Linux GCC Debug/Release, clang-tidy and
ASan/UBSan CTest 58/58 each; PostgreSQL suites Release, analysis, ASan
(lost response, committed-after-timeout, terminated orphan retry,
changed bytes, repeated/partial/reordered decisions, gap, outbox
redelivery, history quarantine); five mutations each caught. No
Docker/Gitea CI, process kill, broker or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 17:30:30 -04:00
co-authored by Claude Opus 5.5
parent a2c4543df2
commit dd7e8028b0
27 changed files with 2075 additions and 216 deletions
+12
View File
@@ -542,3 +542,15 @@ adds:
Recovery restores retained genomes, ancestry and dead organisms, and continues
identically. Next: M7.5 durable decisions, history and outbox rows in the same
transaction, and commit-ID resolution.
M7.5 implemented and locally qualified (2026-09-28), under ADR-0063.
- Manifest v2 binds a per-tick validation trace and the decision block.
- Staging replays the trace from the committed predecessor payload.
- `RegionBatch` seals at a death or at 64 ticks and pauses until the commit is
confirmed.
- Schema v5 stores decisions and outbox rows atomically with each checkpoint,
with one terminal outcome per actor and immutable history.
- Exact retries return the original receipt; read-only resolution covers
ambiguous outcomes.
- Recovery reconciles decisions with the restored state.
Next: M7.6 hosts bounded authoritative ticks (apps/evo-server, region_host).
+24
View File
@@ -812,3 +812,27 @@ and ASan/UBSan CTest each ran 57/57. The PostgreSQL suites passed 2/2/5 times.
Next: M7.5 durable decisions, history and outbox rows, and commit-ID resolution.
Retained ancestry still needs M7.5 history and M7.7/M7.9 kill-recovery evidence.
## M7.5 durable batch outcomes (2026-09-28)
ADR-0063 introduces manifest v2, which binds a per-tick validation trace object
and a decision block.
- Staging replays the trace from the committed predecessor's verified payload and
must reproduce every report and the candidate bytes.
- `RegionBatch` seals at a death or at 64 ticks and refuses ticks until the commit
is confirmed.
- Schema v5 records each checkpoint, its deaths and their outbox rows in one
statement:
- one terminal outcome per actor;
- committed history is immutable;
- exact retries return the original receipt;
- read-only resolution answers committed, absent or conflict.
- Recovery reconciles durable deaths with the restored lifecycle.
Real-PostgreSQL tests cover a lost response, a timeout that later committed, a
terminated orphan that was retried, changed bytes, repeated or partial or
reordered decisions, gaps, outbox redelivery, and history disagreement. Five
mutations were each caught. Windows and Linux ran 58/58; PostgreSQL suites passed
under Release, analysis and ASan.
Next: M7.6 region host. No host, broker, deployment or process-kill test exists yet.
+119
View File
@@ -0,0 +1,119 @@
# ADR-0063: Batch evidence, durable outcomes and commit resolution
## Status
Accepted for M7.5, 2026-09-28. Builds on ADR-0058 (full checkpoint batches),
ADR-0060 (fencing) and ADR-0061/0062 (publication, profiles). No region host,
broker or deployment is added; M7.6 hosts drive these APIs.
## Context
ADR-0058 requires irreversible outcomes to become final only through the same
fenced transaction as the state that produced them, with exact reconciliation of
ambiguous commits. M7.4 published full checkpoints but recorded no decisions, no
outbox and no batch trace. Its retries could only compare the head, and the
database had no record of why a death happened.
The environmental profiles have exactly two actors, whose only permanent outcome
is death, and deaths are terminal. The PostgreSQL adapter bounds each parameter
to 4 KiB and each call to 16 parameters (ADR-0059).
## Decision
**Manifest v2** (360 bytes) appends to v1 the trace length and SHA-256, the
decision count and the decision-block SHA-256; one reserved field must be zero.
The commit digest therefore binds the whole batch. v1 manifests from M7.4 commits
still decode, but only v2 is written.
**Validation trace** (schemas/region-batch.md) records the ordered
input/report pair for every tick. Its 96-byte header binds the profile, region,
predecessor commit and digest, and the tick range, so a trace cannot be replayed
elsewhere. Each tick is a 304-byte step, up to 19,552 bytes total. It is stored as
an immutable object beside the payload.
**Staging replays** the trace from the committed predecessor's verified payload
and must reproduce every report and the candidate bytes exactly. A missing or
corrupt predecessor stops publication (`predecessor`).
**Decisions** are 24-byte records (tick, actor, outcome, archived organism),
derived deterministically from terminal transitions. They are ordered by
(tick, actor), with at most one per actor and ticks inside the batch. Their
organism is 0 for environmental-v1 and the bound organism for the lineage
profile. The block is capped at 8 records, which keeps it within one adapter
parameter; these profiles can produce at most 2. The decision identity is
(region, outcome, actor, tick), with no hashing needed.
`RegionBatch` records at most one pending batch per state:
- It seals itself at the first terminal transition, so the whole region pauses
at an irreversible outcome. It also seals at 64 ticks, or when the host flushes.
- A sealed batch refuses ticks until `confirm(end_tick)`.
- Outcomes are provisional until a receipt arrives.
**Schema v5**:
- `publish_region_checkpoint(manifest, decisions)` validates the decision block
against the manifest digest, then checks for an existing commit ID. Identical
bytes return the original receipt without mutation or authority; anything else
is EVL07.
- A new commit then takes the fenced path. In the same statement it inserts the
commit (with its decision block), the head, the decisions and the outbox rows.
A repeated death is EVL08, backed by a unique index on terminal outcomes.
- `resolve_region_commit` is read-only and answers committed (with the receipt),
absent or conflict.
- `load_region_decisions` supports recovery reconciliation.
- `pending_region_outbox` and `acknowledge_region_outbox` give at-least-once
delivery; acknowledgement is idempotent.
- Triggers forbid UPDATE, DELETE and TRUNCATE of commits and decisions, and allow
outbox rows only a first delivery timestamp (EVL09).
- The runtime role's audited function set grows to ten.
**Recovery** loads decisions and requires lifecycle agreement:
- one death per terminal actor, none for an active actor;
- no death after the head tick, and organisms match the profile;
- for the lineage profile, each death equals the archive's death at the same tick.
Disagreement quarantines the region with `history`.
**Ambiguity**: after `unknown`, the host keeps its batch sealed and resolves the
exact prepared bytes.
- Committed: adopt the receipt.
- Absent: retry the same bytes, and only while the lease is valid.
- Conflict: fail closed. A timeout is never treated as a rollback.
## Alternatives Considered
- **The whole request in the database**: exceeds adapter bounds and duplicates
object storage; the trace digest binds it instead.
- **Hashed event IDs**: need SHA-256 in portable code; the structured identity is
unique by construction.
- **Retry by reloading the head**: cannot tell a lost response from a concurrent
successor, and cannot detect changed bytes.
- **Idempotency after the lease check**: would refuse a truthful receipt to a
writer whose lease expired after its commit.
## Benefits
- Deaths are durable exactly once, with their causing state.
- Lost responses resolve without guessing.
- The chain carries verifiable per-tick evidence.
- Delivery can be repeated without touching the simulation.
## Costs
- Every batch restores and replays its predecessor: about 7 ms in local Release
for these payloads, versus about 4 ms without replay in M7.4.
- Two objects are written per batch, and orphans double.
- History tables only grow.
## Risks
- The database still cannot validate the trace or the payload contents; a
compromised host with a valid lease can publish a self-consistent fiction
(ADR-0058).
- Owner roles can disable the triggers.
- The outbox has no consumer yet.
- Profiles with births or more outcome types need their own identity and closure
rules.
- An absent resolution does not prove a delayed original can never arrive; the
unique ID and head guard make the two converge.
## Consequences
- M7.6 hosts must use `RegionBatch`, flush by deadline, pause while sealed, and
resolve before any further tick after `unknown`.
- M7.7 exercises kills at each boundary.
- A broker or projection consumer is later operations work and must deduplicate
by the decision identity.
- Manifest v3, or new decision kinds, require new profile or history values and a
migration.
+1
View File
@@ -73,3 +73,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
- [ADR-0062: Lineage checkpoint profile](0062-lineage-checkpoint-profile.md) - accepted for M7.4b archive codec, LineageRegion closure and history-capable profile 2.
- [ADR-0063: Batch outcomes](0063-batch-outcomes.md) - accepted for M7.5 manifest v2, validation trace replay, durable decisions/outbox and commit resolution.
+15
View File
@@ -186,3 +186,18 @@ archive codec and the `environmental-lineage-v1` profile validate:
- the living set, identity high-waters and retained dead members.
Recovery restores all of these. Durable history rows (M7.5) and process-kill
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
M7.5 (ADR-0063) executed its rows against real PostgreSQL:
- A rejected publication (repeated death, partial or reordered decision block,
gap) leaves the head, commits and decisions unchanged. Publication is a single
statement; an injected failure between insert and commit is not yet tested.
- A lost response resolves to the original receipt, and an exact retry adds no
second death.
- A client timeout while queued resolved as committed after the lock cleared,
and was adopted.
- A terminated orphan resolved as absent and was retried with the same bytes.
- Changed bytes conflict.
- Outbox redelivery is idempotent and never touches the simulation.
Recovery now reconciles durable deaths with the restored lifecycle, so the
checkpoint and durable history must agree. Kill tests at each boundary remain
M7.7.
+52
View File
@@ -2213,3 +2213,55 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
- checkpoint publication 3.5-5.0 ms for 23-37 KiB payloads.
- Not executed: Docker/Gitea CI, remote storage, process kill, reproduction within
a batch (the fixture has none), durable history rows (M7.5) and TSan.
## M7.5 durable batch outcomes - 2026-09-28
- Windows MSVC Release CTest passed 58/58, including the new `region-batch-contracts`:
- recorder sealing at a death or at 64 ticks, early flush, confirm matching, and
a refused tick recording nothing;
- trace round trip, size (19,552 bytes at 64 ticks), range, contiguity and
header binding;
- decision order, uniqueness, range, and refusal of unknown values.
The manifest v2 golden layout, v1 decoding and batch-evidence structural rules
were also covered.
- WSL Linux: the `checks` build set (GCC14 Debug and Release, clang19 analysis,
clang19 ASan/UBSan) passed CTest 58/58 each, and the repository format check
passed.
- The first PostgreSQL clang-tidy build flagged a returned const-reference
parameter and two unchecked optional accesses; all three were fixed. Final
PostgreSQL suites: Release 2 runs, analysis 2, ASan/UBSan 4, all passing.
- Real-database coverage:
- Durable death at tick 20, with decisions and outbox rows in the same commit.
- Outbox: redelivered, acknowledged true then false, then empty; invalid limit
and unknown rows refused.
- A lost response retried with identical bytes returns the same receipt and
records no second death; resolution says committed.
- A changed manifest or changed decisions conflict (EVL07/22023). Other regions
and unknown IDs resolve as conflict and absent.
- A repeated death is EVL08. A forged same-size block, a partial block and a
reordered block are 22023. A gap is EVL03. None of these moved the head or
decisions.
- A 300 ms client timeout while queued on the lease row became unknown; once the
lock cleared the server committed, resolution returned committed and the retry
was idempotent.
- A terminated orphan request resolved as absent, and the same bytes were
retried and committed.
- A missing predecessor payload is `predecessor`. State advanced outside the
batch fails replay. Batches from the wrong predecessor and unsealed batches are
refused before upload.
- Recovery reconciles decisions. An owner-inserted death the state does not
show quarantines with `history`, and cannot be deleted (EVL09).
- Commit and decision UPDATE and DELETE are EVL09 even for the owner; the
defense-in-depth test disables and restores the trigger explicitly.
- Lineage profile: a death is recorded with the archived organism id and
reconciled against the archive.
- Migrations: clean→v5, v1→v5 and v3→v5 with an existing v1-manifest commit;
tampered checksums for v2 through v5 and an unknown v6 are refused.
- Mutations (scratch SQL copies; host source restored and rerun green), each
caught: removing idempotent retry, outcome finality, the decision-digest check,
decision recording, or recovery reconciliation.
- Local measurements, including predecessor replay: Release 6.6-8.1 ms per batch;
analysis about 20-23 ms; ASan about 19-21 ms. Manifest publish p50 about
0.75-0.86 ms.
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
consumer, an injected failure between commit insert and statement end, and TSan.
+11 -1
View File
@@ -68,7 +68,7 @@ every role transitively in it, and rolls back on any violation:
`pg_write_all_data`);
- any table privilege in `evolution`, including via PUBLIC;
- CREATE on the schema, the database or `public`;
- EXECUTE outside the six fenced functions (schema v3).
- EXECUTE outside the ten fenced functions (schema v5).
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
@@ -112,6 +112,16 @@ payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
profile, history, RNG and durability declarations. Provision history-capable
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
- `region_decisions` and `region_outbox`, with one terminal outcome per actor;
- `resolve_region_commit`, `load_region_decisions`, `pending_region_outbox` and
`acknowledge_region_outbox`, bringing the runtime function set to ten.
Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
a first delivery time. Owners can disable triggers, so treat that as an audited
break-glass action. EVL08 means an outcome was already final. After an
ambiguous publication, resolve the exact prepared bytes before doing anything
else.
There is no operator CLI for provisioning yet, and no real world has been
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
have no payload.