Complete M7.5 durable batch outcomes and commit resolution
ADR-0063: manifest v2 (360 bytes) binds a per-tick validation trace object and a canonical decision block by SHA-256. Publication replays the trace from the committed predecessor's verified payload and requires every report and the candidate bytes, derives deaths deterministically, uploads payload and trace with verified read-back, then sends manifest and decisions to one fenced statement. RegionBatch seals at a terminal transition or 64 ticks and refuses ticks until durable confirmation. Schema v5 records the checkpoint, its decisions and outbox rows atomically; a unique index keeps one terminal outcome per actor (EVL08), triggers make commits/decisions immutable (EVL09), an exact retry of a committed ID returns the original receipt, changed bytes conflict, and read-only resolve_region_commit answers committed/absent/ conflict. Outbox pending/acknowledge give idempotent at-least-once delivery. Recovery reconciles durable deaths with restored lifecycle (and archive deaths for the lineage profile). Validated: Windows CTest 58/58; Linux GCC Debug/Release, clang-tidy and ASan/UBSan CTest 58/58 each; PostgreSQL suites Release, analysis, ASan (lost response, committed-after-timeout, terminated orphan retry, changed bytes, repeated/partial/reordered decisions, gap, outbox redelivery, history quarantine); five mutations each caught. No Docker/Gitea CI, process kill, broker or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -542,3 +542,15 @@ adds:
|
||||
Recovery restores retained genomes, ancestry and dead organisms, and continues
|
||||
identically. Next: M7.5 durable decisions, history and outbox rows in the same
|
||||
transaction, and commit-ID resolution.
|
||||
|
||||
M7.5 implemented and locally qualified (2026-09-28), under ADR-0063.
|
||||
- Manifest v2 binds a per-tick validation trace and the decision block.
|
||||
- Staging replays the trace from the committed predecessor payload.
|
||||
- `RegionBatch` seals at a death or at 64 ticks and pauses until the commit is
|
||||
confirmed.
|
||||
- Schema v5 stores decisions and outbox rows atomically with each checkpoint,
|
||||
with one terminal outcome per actor and immutable history.
|
||||
- Exact retries return the original receipt; read-only resolution covers
|
||||
ambiguous outcomes.
|
||||
- Recovery reconciles decisions with the restored state.
|
||||
Next: M7.6 hosts bounded authoritative ticks (apps/evo-server, region_host).
|
||||
|
||||
@@ -812,3 +812,27 @@ and ASan/UBSan CTest each ran 57/57. The PostgreSQL suites passed 2/2/5 times.
|
||||
|
||||
Next: M7.5 durable decisions, history and outbox rows, and commit-ID resolution.
|
||||
Retained ancestry still needs M7.5 history and M7.7/M7.9 kill-recovery evidence.
|
||||
|
||||
## M7.5 durable batch outcomes (2026-09-28)
|
||||
|
||||
ADR-0063 introduces manifest v2, which binds a per-tick validation trace object
|
||||
and a decision block.
|
||||
- Staging replays the trace from the committed predecessor's verified payload and
|
||||
must reproduce every report and the candidate bytes.
|
||||
- `RegionBatch` seals at a death or at 64 ticks and refuses ticks until the commit
|
||||
is confirmed.
|
||||
- Schema v5 records each checkpoint, its deaths and their outbox rows in one
|
||||
statement:
|
||||
- one terminal outcome per actor;
|
||||
- committed history is immutable;
|
||||
- exact retries return the original receipt;
|
||||
- read-only resolution answers committed, absent or conflict.
|
||||
- Recovery reconciles durable deaths with the restored lifecycle.
|
||||
|
||||
Real-PostgreSQL tests cover a lost response, a timeout that later committed, a
|
||||
terminated orphan that was retried, changed bytes, repeated or partial or
|
||||
reordered decisions, gaps, outbox redelivery, and history disagreement. Five
|
||||
mutations were each caught. Windows and Linux ran 58/58; PostgreSQL suites passed
|
||||
under Release, analysis and ASan.
|
||||
|
||||
Next: M7.6 region host. No host, broker, deployment or process-kill test exists yet.
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# ADR-0063: Batch evidence, durable outcomes and commit resolution
|
||||
|
||||
## Status
|
||||
Accepted for M7.5, 2026-09-28. Builds on ADR-0058 (full checkpoint batches),
|
||||
ADR-0060 (fencing) and ADR-0061/0062 (publication, profiles). No region host,
|
||||
broker or deployment is added; M7.6 hosts drive these APIs.
|
||||
|
||||
## Context
|
||||
ADR-0058 requires irreversible outcomes to become final only through the same
|
||||
fenced transaction as the state that produced them, with exact reconciliation of
|
||||
ambiguous commits. M7.4 published full checkpoints but recorded no decisions, no
|
||||
outbox and no batch trace. Its retries could only compare the head, and the
|
||||
database had no record of why a death happened.
|
||||
|
||||
The environmental profiles have exactly two actors, whose only permanent outcome
|
||||
is death, and deaths are terminal. The PostgreSQL adapter bounds each parameter
|
||||
to 4 KiB and each call to 16 parameters (ADR-0059).
|
||||
|
||||
## Decision
|
||||
**Manifest v2** (360 bytes) appends to v1 the trace length and SHA-256, the
|
||||
decision count and the decision-block SHA-256; one reserved field must be zero.
|
||||
The commit digest therefore binds the whole batch. v1 manifests from M7.4 commits
|
||||
still decode, but only v2 is written.
|
||||
|
||||
**Validation trace** (schemas/region-batch.md) records the ordered
|
||||
input/report pair for every tick. Its 96-byte header binds the profile, region,
|
||||
predecessor commit and digest, and the tick range, so a trace cannot be replayed
|
||||
elsewhere. Each tick is a 304-byte step, up to 19,552 bytes total. It is stored as
|
||||
an immutable object beside the payload.
|
||||
|
||||
**Staging replays** the trace from the committed predecessor's verified payload
|
||||
and must reproduce every report and the candidate bytes exactly. A missing or
|
||||
corrupt predecessor stops publication (`predecessor`).
|
||||
|
||||
**Decisions** are 24-byte records (tick, actor, outcome, archived organism),
|
||||
derived deterministically from terminal transitions. They are ordered by
|
||||
(tick, actor), with at most one per actor and ticks inside the batch. Their
|
||||
organism is 0 for environmental-v1 and the bound organism for the lineage
|
||||
profile. The block is capped at 8 records, which keeps it within one adapter
|
||||
parameter; these profiles can produce at most 2. The decision identity is
|
||||
(region, outcome, actor, tick), with no hashing needed.
|
||||
|
||||
`RegionBatch` records at most one pending batch per state:
|
||||
- It seals itself at the first terminal transition, so the whole region pauses
|
||||
at an irreversible outcome. It also seals at 64 ticks, or when the host flushes.
|
||||
- A sealed batch refuses ticks until `confirm(end_tick)`.
|
||||
- Outcomes are provisional until a receipt arrives.
|
||||
|
||||
**Schema v5**:
|
||||
- `publish_region_checkpoint(manifest, decisions)` validates the decision block
|
||||
against the manifest digest, then checks for an existing commit ID. Identical
|
||||
bytes return the original receipt without mutation or authority; anything else
|
||||
is EVL07.
|
||||
- A new commit then takes the fenced path. In the same statement it inserts the
|
||||
commit (with its decision block), the head, the decisions and the outbox rows.
|
||||
A repeated death is EVL08, backed by a unique index on terminal outcomes.
|
||||
- `resolve_region_commit` is read-only and answers committed (with the receipt),
|
||||
absent or conflict.
|
||||
- `load_region_decisions` supports recovery reconciliation.
|
||||
- `pending_region_outbox` and `acknowledge_region_outbox` give at-least-once
|
||||
delivery; acknowledgement is idempotent.
|
||||
- Triggers forbid UPDATE, DELETE and TRUNCATE of commits and decisions, and allow
|
||||
outbox rows only a first delivery timestamp (EVL09).
|
||||
- The runtime role's audited function set grows to ten.
|
||||
|
||||
**Recovery** loads decisions and requires lifecycle agreement:
|
||||
- one death per terminal actor, none for an active actor;
|
||||
- no death after the head tick, and organisms match the profile;
|
||||
- for the lineage profile, each death equals the archive's death at the same tick.
|
||||
|
||||
Disagreement quarantines the region with `history`.
|
||||
|
||||
**Ambiguity**: after `unknown`, the host keeps its batch sealed and resolves the
|
||||
exact prepared bytes.
|
||||
- Committed: adopt the receipt.
|
||||
- Absent: retry the same bytes, and only while the lease is valid.
|
||||
- Conflict: fail closed. A timeout is never treated as a rollback.
|
||||
|
||||
## Alternatives Considered
|
||||
- **The whole request in the database**: exceeds adapter bounds and duplicates
|
||||
object storage; the trace digest binds it instead.
|
||||
- **Hashed event IDs**: need SHA-256 in portable code; the structured identity is
|
||||
unique by construction.
|
||||
- **Retry by reloading the head**: cannot tell a lost response from a concurrent
|
||||
successor, and cannot detect changed bytes.
|
||||
- **Idempotency after the lease check**: would refuse a truthful receipt to a
|
||||
writer whose lease expired after its commit.
|
||||
|
||||
## Benefits
|
||||
- Deaths are durable exactly once, with their causing state.
|
||||
- Lost responses resolve without guessing.
|
||||
- The chain carries verifiable per-tick evidence.
|
||||
- Delivery can be repeated without touching the simulation.
|
||||
|
||||
## Costs
|
||||
- Every batch restores and replays its predecessor: about 7 ms in local Release
|
||||
for these payloads, versus about 4 ms without replay in M7.4.
|
||||
- Two objects are written per batch, and orphans double.
|
||||
- History tables only grow.
|
||||
|
||||
## Risks
|
||||
- The database still cannot validate the trace or the payload contents; a
|
||||
compromised host with a valid lease can publish a self-consistent fiction
|
||||
(ADR-0058).
|
||||
- Owner roles can disable the triggers.
|
||||
- The outbox has no consumer yet.
|
||||
- Profiles with births or more outcome types need their own identity and closure
|
||||
rules.
|
||||
- An absent resolution does not prove a delayed original can never arrive; the
|
||||
unique ID and head guard make the two converge.
|
||||
|
||||
## Consequences
|
||||
- M7.6 hosts must use `RegionBatch`, flush by deadline, pause while sealed, and
|
||||
resolve before any further tick after `unknown`.
|
||||
- M7.7 exercises kills at each boundary.
|
||||
- A broker or projection consumer is later operations work and must deduplicate
|
||||
by the decision identity.
|
||||
- Manifest v3, or new decision kinds, require new profile or history values and a
|
||||
migration.
|
||||
@@ -73,3 +73,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
|
||||
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
|
||||
- [ADR-0062: Lineage checkpoint profile](0062-lineage-checkpoint-profile.md) - accepted for M7.4b archive codec, LineageRegion closure and history-capable profile 2.
|
||||
- [ADR-0063: Batch outcomes](0063-batch-outcomes.md) - accepted for M7.5 manifest v2, validation trace replay, durable decisions/outbox and commit resolution.
|
||||
|
||||
@@ -186,3 +186,18 @@ archive codec and the `environmental-lineage-v1` profile validate:
|
||||
- the living set, identity high-waters and retained dead members.
|
||||
Recovery restores all of these. Durable history rows (M7.5) and process-kill
|
||||
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
|
||||
|
||||
M7.5 (ADR-0063) executed its rows against real PostgreSQL:
|
||||
- A rejected publication (repeated death, partial or reordered decision block,
|
||||
gap) leaves the head, commits and decisions unchanged. Publication is a single
|
||||
statement; an injected failure between insert and commit is not yet tested.
|
||||
- A lost response resolves to the original receipt, and an exact retry adds no
|
||||
second death.
|
||||
- A client timeout while queued resolved as committed after the lock cleared,
|
||||
and was adopted.
|
||||
- A terminated orphan resolved as absent and was retried with the same bytes.
|
||||
- Changed bytes conflict.
|
||||
- Outbox redelivery is idempotent and never touches the simulation.
|
||||
Recovery now reconciles durable deaths with the restored lifecycle, so the
|
||||
checkpoint and durable history must agree. Kill tests at each boundary remain
|
||||
M7.7.
|
||||
|
||||
@@ -2213,3 +2213,55 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
- checkpoint publication 3.5-5.0 ms for 23-37 KiB payloads.
|
||||
- Not executed: Docker/Gitea CI, remote storage, process kill, reproduction within
|
||||
a batch (the fixture has none), durable history rows (M7.5) and TSan.
|
||||
|
||||
## M7.5 durable batch outcomes - 2026-09-28
|
||||
|
||||
- Windows MSVC Release CTest passed 58/58, including the new `region-batch-contracts`:
|
||||
- recorder sealing at a death or at 64 ticks, early flush, confirm matching, and
|
||||
a refused tick recording nothing;
|
||||
- trace round trip, size (19,552 bytes at 64 ticks), range, contiguity and
|
||||
header binding;
|
||||
- decision order, uniqueness, range, and refusal of unknown values.
|
||||
The manifest v2 golden layout, v1 decoding and batch-evidence structural rules
|
||||
were also covered.
|
||||
- WSL Linux: the `checks` build set (GCC14 Debug and Release, clang19 analysis,
|
||||
clang19 ASan/UBSan) passed CTest 58/58 each, and the repository format check
|
||||
passed.
|
||||
- The first PostgreSQL clang-tidy build flagged a returned const-reference
|
||||
parameter and two unchecked optional accesses; all three were fixed. Final
|
||||
PostgreSQL suites: Release 2 runs, analysis 2, ASan/UBSan 4, all passing.
|
||||
- Real-database coverage:
|
||||
- Durable death at tick 20, with decisions and outbox rows in the same commit.
|
||||
- Outbox: redelivered, acknowledged true then false, then empty; invalid limit
|
||||
and unknown rows refused.
|
||||
- A lost response retried with identical bytes returns the same receipt and
|
||||
records no second death; resolution says committed.
|
||||
- A changed manifest or changed decisions conflict (EVL07/22023). Other regions
|
||||
and unknown IDs resolve as conflict and absent.
|
||||
- A repeated death is EVL08. A forged same-size block, a partial block and a
|
||||
reordered block are 22023. A gap is EVL03. None of these moved the head or
|
||||
decisions.
|
||||
- A 300 ms client timeout while queued on the lease row became unknown; once the
|
||||
lock cleared the server committed, resolution returned committed and the retry
|
||||
was idempotent.
|
||||
- A terminated orphan request resolved as absent, and the same bytes were
|
||||
retried and committed.
|
||||
- A missing predecessor payload is `predecessor`. State advanced outside the
|
||||
batch fails replay. Batches from the wrong predecessor and unsealed batches are
|
||||
refused before upload.
|
||||
- Recovery reconciles decisions. An owner-inserted death the state does not
|
||||
show quarantines with `history`, and cannot be deleted (EVL09).
|
||||
- Commit and decision UPDATE and DELETE are EVL09 even for the owner; the
|
||||
defense-in-depth test disables and restores the trigger explicitly.
|
||||
- Lineage profile: a death is recorded with the archived organism id and
|
||||
reconciled against the archive.
|
||||
- Migrations: clean→v5, v1→v5 and v3→v5 with an existing v1-manifest commit;
|
||||
tampered checksums for v2 through v5 and an unknown v6 are refused.
|
||||
- Mutations (scratch SQL copies; host source restored and rerun green), each
|
||||
caught: removing idempotent retry, outcome finality, the decision-digest check,
|
||||
decision recording, or recovery reconciliation.
|
||||
- Local measurements, including predecessor replay: Release 6.6-8.1 ms per batch;
|
||||
analysis about 20-23 ms; ASan about 19-21 ms. Manifest publish p50 about
|
||||
0.75-0.86 ms.
|
||||
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
|
||||
consumer, an injected failure between commit insert and statement end, and TSan.
|
||||
|
||||
@@ -68,7 +68,7 @@ every role transitively in it, and rolls back on any violation:
|
||||
`pg_write_all_data`);
|
||||
- any table privilege in `evolution`, including via PUBLIC;
|
||||
- CREATE on the schema, the database or `public`;
|
||||
- EXECUTE outside the six fenced functions (schema v3).
|
||||
- EXECUTE outside the ten fenced functions (schema v5).
|
||||
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
||||
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
||||
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
||||
@@ -112,6 +112,16 @@ payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
|
||||
profile, history, RNG and durability declarations. Provision history-capable
|
||||
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
|
||||
|
||||
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
|
||||
- `region_decisions` and `region_outbox`, with one terminal outcome per actor;
|
||||
- `resolve_region_commit`, `load_region_decisions`, `pending_region_outbox` and
|
||||
`acknowledge_region_outbox`, bringing the runtime function set to ten.
|
||||
Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
|
||||
a first delivery time. Owners can disable triggers, so treat that as an audited
|
||||
break-glass action. EVL08 means an outcome was already final. After an
|
||||
ambiguous publication, resolve the exact prepared bytes before doing anything
|
||||
else.
|
||||
|
||||
There is no operator CLI for provisioning yet, and no real world has been
|
||||
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
have no payload.
|
||||
|
||||
Reference in New Issue
Block a user