Files
aevolution/docs/adr/0063-batch-outcomes.md
T
dingoandClaude Opus 5.5 ab4bf5355f Fix object reuse durability and refuse unsafe pre-v5 upgrades
Review P1 (object durability): a name installed by a put whose directory
fsync failed was later reused as success without syncing again, so a
publication could acknowledge an object that might vanish on power
loss. Reuse (existing and lost-link-race paths) now fsyncs the verified
file and the directory before succeeding, and namespace opens always
sync their parent. A test-only ObjectStoreFaults hook injects directory
sync failures; tests cover failed sync then retries, namespace creation
syncs, and publication refusing to proceed until durability holds.

Review P1 (pre-v5 history): v5 recovery requires one durable decision
per terminal actor, but v3/v4 published checkpoints carry none, so an
accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now
runs before 0005 in every upgrade path and refuses (EVL10) when any
published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is
unchanged. CI publishes a real v4 checkpoint and proves the upgrade is
refused with nothing changed; a committed death without a decision is
shown to recover as quarantined history.

Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2;
reverting each fix is caught by its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 19:20:35 -04:00

127 lines
6.3 KiB
Markdown

# ADR-0063: Batch evidence, durable outcomes and commit resolution
## Status
Accepted for M7.5, 2026-09-28. Builds on ADR-0058 (full checkpoint batches),
ADR-0060 (fencing) and ADR-0061/0062 (publication, profiles). No region host,
broker or deployment is added; M7.6 hosts drive these APIs.
## Context
ADR-0058 requires irreversible outcomes to become final only through the same
fenced transaction as the state that produced them, with exact reconciliation of
ambiguous commits. M7.4 published full checkpoints but recorded no decisions, no
outbox and no batch trace. Its retries could only compare the head, and the
database had no record of why a death happened.
The environmental profiles have exactly two actors, whose only permanent outcome
is death, and deaths are terminal. The PostgreSQL adapter bounds each parameter
to 4 KiB and each call to 16 parameters (ADR-0059).
## Decision
**Manifest v2** (360 bytes) appends to v1 the trace length and SHA-256, the
decision count and the decision-block SHA-256; one reserved field must be zero.
The commit digest therefore binds the whole batch. v1 manifests from M7.4 commits
still decode, but only v2 is written.
**Validation trace** (schemas/region-batch.md) records the ordered
input/report pair for every tick. Its 96-byte header binds the profile, region,
predecessor commit and digest, and the tick range, so a trace cannot be replayed
elsewhere. Each tick is a 304-byte step, up to 19,552 bytes total. It is stored as
an immutable object beside the payload.
**Staging replays** the trace from the committed predecessor's verified payload
and must reproduce every report and the candidate bytes exactly. A missing or
corrupt predecessor stops publication (`predecessor`).
**Decisions** are 24-byte records (tick, actor, outcome, archived organism),
derived deterministically from terminal transitions. They are ordered by
(tick, actor), with at most one per actor and ticks inside the batch. Their
organism is 0 for environmental-v1 and the bound organism for the lineage
profile. The block is capped at 8 records, which keeps it within one adapter
parameter; these profiles can produce at most 2. The decision identity is
(region, outcome, actor, tick), with no hashing needed.
`RegionBatch` records at most one pending batch per state:
- It seals itself at the first terminal transition, so the whole region pauses
at an irreversible outcome. It also seals at 64 ticks, or when the host flushes.
- A sealed batch refuses ticks until `confirm(end_tick)`.
- Outcomes are provisional until a receipt arrives.
**Schema v5**:
- `publish_region_checkpoint(manifest, decisions)` validates the decision block
against the manifest digest, then checks for an existing commit ID. Identical
bytes return the original receipt without mutation or authority; anything else
is EVL07.
- A new commit then takes the fenced path. In the same statement it inserts the
commit (with its decision block), the head, the decisions and the outbox rows.
A repeated death is EVL08, backed by a unique index on terminal outcomes.
- `resolve_region_commit` is read-only and answers committed (with the receipt),
absent or conflict.
- `load_region_decisions` supports recovery reconciliation.
- `pending_region_outbox` and `acknowledge_region_outbox` give at-least-once
delivery; acknowledgement is idempotent.
- Triggers forbid UPDATE, DELETE and TRUNCATE of commits and decisions, and allow
outbox rows only a first delivery timestamp (EVL09).
- The runtime role's audited function set grows to ten.
**Upgrades** from v3 or v4 run `migrations/pre_v5_guard.sql` inside the same
transaction, before 0005. The guard refuses (EVL10), changing nothing, if any
published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold
terminal organisms without decisions, and v5 recovery would quarantine it.
Genesis-only regions pass, because tick 0 has no terminal organisms. A verified
legacy-history conversion is future work; there is none now.
**Recovery** loads decisions and requires lifecycle agreement:
- one death per terminal actor, none for an active actor;
- no death after the head tick, and organisms match the profile;
- for the lineage profile, each death equals the archive's death at the same tick.
Disagreement quarantines the region with `history`.
**Ambiguity**: after `unknown`, the host keeps its batch sealed and resolves the
exact prepared bytes.
- Committed: adopt the receipt.
- Absent: retry the same bytes, and only while the lease is valid.
- Conflict: fail closed. A timeout is never treated as a rollback.
## Alternatives Considered
- **The whole request in the database**: exceeds adapter bounds and duplicates
object storage; the trace digest binds it instead.
- **Hashed event IDs**: need SHA-256 in portable code; the structured identity is
unique by construction.
- **Retry by reloading the head**: cannot tell a lost response from a concurrent
successor, and cannot detect changed bytes.
- **Idempotency after the lease check**: would refuse a truthful receipt to a
writer whose lease expired after its commit.
## Benefits
- Deaths are durable exactly once, with their causing state.
- Lost responses resolve without guessing.
- The chain carries verifiable per-tick evidence.
- Delivery can be repeated without touching the simulation.
## Costs
- Every batch restores and replays its predecessor: about 7 ms in local Release
for these payloads, versus about 4 ms without replay in M7.4.
- Two objects are written per batch, and orphans double.
- History tables only grow.
## Risks
- The database still cannot validate the trace or the payload contents; a
compromised host with a valid lease can publish a self-consistent fiction
(ADR-0058).
- Owner roles can disable the triggers.
- The outbox has no consumer yet.
- Profiles with births or more outcome types need their own identity and closure
rules.
- An absent resolution does not prove a delayed original can never arrive; the
unique ID and head guard make the two converge.
## Consequences
- M7.6 hosts must use `RegionBatch`, flush by deadline, pause while sealed, and
resolve before any further tick after `unknown`.
- M7.7 exercises kills at each boundary.
- A broker or projection consumer is later operations work and must deduplicate
by the decision identity.
- Manifest v3, or new decision kinds, require new profile or history values and a
migration.