Files
aevolution/docs/adr/0063-batch-outcomes.md
T
dingoandClaude Opus 5.5 ab4bf5355f Fix object reuse durability and refuse unsafe pre-v5 upgrades
Review P1 (object durability): a name installed by a put whose directory
fsync failed was later reused as success without syncing again, so a
publication could acknowledge an object that might vanish on power
loss. Reuse (existing and lost-link-race paths) now fsyncs the verified
file and the directory before succeeding, and namespace opens always
sync their parent. A test-only ObjectStoreFaults hook injects directory
sync failures; tests cover failed sync then retries, namespace creation
syncs, and publication refusing to proceed until durability holds.

Review P1 (pre-v5 history): v5 recovery requires one durable decision
per terminal actor, but v3/v4 published checkpoints carry none, so an
accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now
runs before 0005 in every upgrade path and refuses (EVL10) when any
published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is
unchanged. CI publishes a real v4 checkpoint and proves the upgrade is
refused with nothing changed; a committed death without a decision is
shown to recover as quarantined history.

Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2;
reverting each fix is caught by its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-28 19:20:35 -04:00

6.3 KiB

ADR-0063: Batch evidence, durable outcomes and commit resolution

Status

Accepted for M7.5, 2026-09-28. Builds on ADR-0058 (full checkpoint batches), ADR-0060 (fencing) and ADR-0061/0062 (publication, profiles). No region host, broker or deployment is added; M7.6 hosts drive these APIs.

Context

ADR-0058 requires irreversible outcomes to become final only through the same fenced transaction as the state that produced them, with exact reconciliation of ambiguous commits. M7.4 published full checkpoints but recorded no decisions, no outbox and no batch trace. Its retries could only compare the head, and the database had no record of why a death happened.

The environmental profiles have exactly two actors, whose only permanent outcome is death, and deaths are terminal. The PostgreSQL adapter bounds each parameter to 4 KiB and each call to 16 parameters (ADR-0059).

Decision

Manifest v2 (360 bytes) appends to v1 the trace length and SHA-256, the decision count and the decision-block SHA-256; one reserved field must be zero. The commit digest therefore binds the whole batch. v1 manifests from M7.4 commits still decode, but only v2 is written.

Validation trace (schemas/region-batch.md) records the ordered input/report pair for every tick. Its 96-byte header binds the profile, region, predecessor commit and digest, and the tick range, so a trace cannot be replayed elsewhere. Each tick is a 304-byte step, up to 19,552 bytes total. It is stored as an immutable object beside the payload.

Staging replays the trace from the committed predecessor's verified payload and must reproduce every report and the candidate bytes exactly. A missing or corrupt predecessor stops publication (predecessor).

Decisions are 24-byte records (tick, actor, outcome, archived organism), derived deterministically from terminal transitions. They are ordered by (tick, actor), with at most one per actor and ticks inside the batch. Their organism is 0 for environmental-v1 and the bound organism for the lineage profile. The block is capped at 8 records, which keeps it within one adapter parameter; these profiles can produce at most 2. The decision identity is (region, outcome, actor, tick), with no hashing needed.

RegionBatch records at most one pending batch per state:

  • It seals itself at the first terminal transition, so the whole region pauses at an irreversible outcome. It also seals at 64 ticks, or when the host flushes.
  • A sealed batch refuses ticks until confirm(end_tick).
  • Outcomes are provisional until a receipt arrives.

Schema v5:

  • publish_region_checkpoint(manifest, decisions) validates the decision block against the manifest digest, then checks for an existing commit ID. Identical bytes return the original receipt without mutation or authority; anything else is EVL07.
  • A new commit then takes the fenced path. In the same statement it inserts the commit (with its decision block), the head, the decisions and the outbox rows. A repeated death is EVL08, backed by a unique index on terminal outcomes.
  • resolve_region_commit is read-only and answers committed (with the receipt), absent or conflict.
  • load_region_decisions supports recovery reconciliation.
  • pending_region_outbox and acknowledge_region_outbox give at-least-once delivery; acknowledgement is idempotent.
  • Triggers forbid UPDATE, DELETE and TRUNCATE of commits and decisions, and allow outbox rows only a first delivery timestamp (EVL09).
  • The runtime role's audited function set grows to ten.

Upgrades from v3 or v4 run migrations/pre_v5_guard.sql inside the same transaction, before 0005. The guard refuses (EVL10), changing nothing, if any published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold terminal organisms without decisions, and v5 recovery would quarantine it. Genesis-only regions pass, because tick 0 has no terminal organisms. A verified legacy-history conversion is future work; there is none now.

Recovery loads decisions and requires lifecycle agreement:

  • one death per terminal actor, none for an active actor;
  • no death after the head tick, and organisms match the profile;
  • for the lineage profile, each death equals the archive's death at the same tick.

Disagreement quarantines the region with history.

Ambiguity: after unknown, the host keeps its batch sealed and resolves the exact prepared bytes.

  • Committed: adopt the receipt.
  • Absent: retry the same bytes, and only while the lease is valid.
  • Conflict: fail closed. A timeout is never treated as a rollback.

Alternatives Considered

  • The whole request in the database: exceeds adapter bounds and duplicates object storage; the trace digest binds it instead.
  • Hashed event IDs: need SHA-256 in portable code; the structured identity is unique by construction.
  • Retry by reloading the head: cannot tell a lost response from a concurrent successor, and cannot detect changed bytes.
  • Idempotency after the lease check: would refuse a truthful receipt to a writer whose lease expired after its commit.

Benefits

  • Deaths are durable exactly once, with their causing state.
  • Lost responses resolve without guessing.
  • The chain carries verifiable per-tick evidence.
  • Delivery can be repeated without touching the simulation.

Costs

  • Every batch restores and replays its predecessor: about 7 ms in local Release for these payloads, versus about 4 ms without replay in M7.4.
  • Two objects are written per batch, and orphans double.
  • History tables only grow.

Risks

  • The database still cannot validate the trace or the payload contents; a compromised host with a valid lease can publish a self-consistent fiction (ADR-0058).
  • Owner roles can disable the triggers.
  • The outbox has no consumer yet.
  • Profiles with births or more outcome types need their own identity and closure rules.
  • An absent resolution does not prove a delayed original can never arrive; the unique ID and head guard make the two converge.

Consequences

  • M7.6 hosts must use RegionBatch, flush by deadline, pause while sealed, and resolve before any further tick after unknown.
  • M7.7 exercises kills at each boundary.
  • A broker or projection consumer is later operations work and must deduplicate by the decision identity.
  • Manifest v3, or new decision kinds, require new profile or history values and a migration.