Complete M7.5 durable batch outcomes and commit resolution

ADR-0063: manifest v2 (360 bytes) binds a per-tick validation trace
object and a canonical decision block by SHA-256. Publication replays the
trace from the committed predecessor's verified payload and requires
every report and the candidate bytes, derives deaths deterministically,
uploads payload and trace with verified read-back, then sends manifest
and decisions to one fenced statement. RegionBatch seals at a terminal
transition or 64 ticks and refuses ticks until durable confirmation.

Schema v5 records the checkpoint, its decisions and outbox rows
atomically; a unique index keeps one terminal outcome per actor
(EVL08), triggers make commits/decisions immutable (EVL09), an exact
retry of a committed ID returns the original receipt, changed bytes
conflict, and read-only resolve_region_commit answers committed/absent/
conflict. Outbox pending/acknowledge give idempotent at-least-once
delivery. Recovery reconciles durable deaths with restored lifecycle
(and archive deaths for the lineage profile).

Validated: Windows CTest 58/58; Linux GCC Debug/Release, clang-tidy and
ASan/UBSan CTest 58/58 each; PostgreSQL suites Release, analysis, ASan
(lost response, committed-after-timeout, terminated orphan retry,
changed bytes, repeated/partial/reordered decisions, gap, outbox
redelivery, history quarantine); five mutations each caught. No
Docker/Gitea CI, process kill, broker or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 17:30:30 -04:00
co-authored by Claude Opus 5.5
parent a2c4543df2
commit dd7e8028b0
27 changed files with 2075 additions and 216 deletions
+59 -27
View File
@@ -1,30 +1,49 @@
#pragma once
#include "evo/persistence/object_store.hpp"
#include "evo/persistence/region_batch.hpp"
#include "evo/persistence/region_checkpoint.hpp"
#include "evo/persistence/region_lease_store.hpp"
#include <chrono>
#include <expected>
#include <memory>
#include <span>
#include <vector>
// Linux publication and recovery of environmental-v1 checkpoints (ADR-0061).
// Order: validate candidate -> immutable upload -> verified read-back -> fenced
// manifest transaction. Nothing before the transaction has authority; a failure
// there leaves the previous head and at most an unreferenced orphan object.
// Linux publication and recovery of region checkpoints (ADR-0061/0062/0063).
// Batch order: replay the recorded trace from the committed predecessor payload
// and require the candidate bytes -> derive decisions -> immutable payload and
// trace uploads with verified read-back -> one fenced transaction recording
// checkpoint, decisions and outbox. Nothing before the transaction has authority:
// a failure there leaves the previous head and at most unreferenced orphan objects.
// Permanent outcomes are final only when a receipt (publish or resolve) confirms them.
namespace evo::persistence {
enum class PublicationError : std::uint8_t {
candidate, // State/manifest not a valid successor; nothing uploaded.
digest, // Digest provider failure; nothing uploaded.
upload, // Object write failed; no manifest was sent.
verify, // Read-back differs from the candidate; no manifest was sent.
rejected, // Database refused (see store); the upload is an orphan.
unknown, // Outcome unknown (see store); stay paused and reconcile the head.
receipt, // Committed receipt disagrees with the local manifest: fail closed.
candidate, // State, batch or manifest is not a valid successor; nothing uploaded.
predecessor, // Committed predecessor payload missing/corrupt/invalid: quarantine.
digest, // Digest provider failure; nothing uploaded.
upload, // Object write failed; no manifest was sent.
verify, // Read-back differs from the candidate; no manifest was sent.
rejected, // Database refused (see store); uploads are orphans.
unknown, // Outcome unknown (see store): stay paused; resolve `prepared`.
receipt, // Committed receipt disagrees with the local manifest: fail closed.
};
// The exact bytes sent to the database. After `unknown`, resolve or retry only these.
struct PreparedCommit {
RegionKey key{};
Uuid commit{};
std::vector<std::uint8_t> manifest;
std::vector<std::uint8_t> decisions;
};
struct PublicationFailure {
PublicationError kind{};
LeaseStoreFailure store{}; // Meaningful for rejected/unknown.
ObjectError object{}; // Meaningful for upload/verify.
ObjectError object{}; // Meaningful for upload/verify/predecessor.
PreparedCommit prepared; // Present for rejected/unknown.
};
// Durably confirmed batch: its decisions are now final and may be acknowledged.
struct PublishedBatch {
CommittedCheckpoint committed;
std::vector<BatchDecision> decisions;
};
enum class RecoveryError : std::uint8_t {
store, // Database unavailable/failed (see store).
@@ -34,6 +53,7 @@ enum class RecoveryError : std::uint8_t {
missing_payload, // Referenced object absent: quarantine, no older head.
corrupt_payload, // Length/digest mismatch: quarantine, no older head.
payload, // Verified bytes fail full state validation: quarantine.
history, // Durable decisions disagree with the restored lifecycle: quarantine.
};
struct RecoveryFailure {
RecoveryError kind{};
@@ -43,11 +63,13 @@ struct RecoveredCheckpoint {
CommittedCheckpoint committed;
CheckpointManifest manifest;
std::unique_ptr<processes::EnvironmentalCommunity> state;
std::vector<CommittedDecision> decisions;
};
struct RecoveredLineageCheckpoint {
CommittedCheckpoint committed;
CheckpointManifest manifest;
std::unique_ptr<processes::LineageRegion> state;
std::vector<CommittedDecision> decisions;
};
// Owner/operator connection only: uploads a tick-0 state and records genesis.
@@ -55,17 +77,17 @@ struct RecoveredLineageCheckpoint {
Postgres& owner, const FilesystemObjectStore& objects, RegionKey key, std::uint64_t frame,
const processes::EnvironmentalCommunity& opening, Uuid commit, const Sha256Digest& build,
std::size_t restore_budget, std::chrono::milliseconds timeout);
// Publishes `state` as the successor of `predecessor` under `lease`. The caller
// must hold the committed predecessor it simulated from and a lease that may
// begin a commit (LeaseAuthority::may_begin_commit).
[[nodiscard]] std::expected<CommittedCheckpoint, PublicationFailure>
publish_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects,
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
const processes::EnvironmentalCommunity& state, Uuid commit,
const Sha256Digest& build, std::size_t restore_budget,
std::chrono::milliseconds timeout);
// Loads and fully verifies the committed head. Readiness additionally requires a
// fresh valid lease acquired before this call (M7.7); this does not grant one.
// Publishes a sealed batch that began at `predecessor`, under `lease` (which the
// caller has checked with LeaseAuthority::may_begin_commit). On success call
// batch.confirm(end_tick); only then are its decisions final.
[[nodiscard]] std::expected<PublishedBatch, PublicationFailure>
publish_environmental_batch(Postgres& db, const FilesystemObjectStore& objects,
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
const RegionBatch<processes::EnvironmentalCommunity>& batch,
Uuid commit, const Sha256Digest& build, std::size_t restore_budget,
std::chrono::milliseconds timeout);
// Loads and fully verifies the committed head and reconciles durable decisions.
// Readiness additionally requires a fresh valid lease acquired before this call.
[[nodiscard]] std::expected<RecoveredCheckpoint, RecoveryFailure>
recover_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
std::span<const Sha256Digest> retained_builds,
@@ -79,12 +101,22 @@ provision_lineage_region(Postgres& owner, const FilesystemObjectStore& objects,
std::uint64_t frame, const processes::LineageRegion& opening, Uuid commit,
const Sha256Digest& build, std::size_t restore_budget,
std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<CommittedCheckpoint, PublicationFailure> publish_lineage_checkpoint(
Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
const CommittedCheckpoint& predecessor, const processes::LineageRegion& state, Uuid commit,
const Sha256Digest& build, std::size_t restore_budget, std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<PublishedBatch, PublicationFailure>
publish_lineage_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
const CommittedCheckpoint& predecessor,
const RegionBatch<processes::LineageRegion>& batch, Uuid commit,
const Sha256Digest& build, std::size_t restore_budget,
std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<RecoveredLineageCheckpoint, RecoveryFailure>
recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
std::span<const Sha256Digest> retained_builds,
std::size_t restore_budget, std::chrono::milliseconds timeout);
// After an `unknown` publication: resolve exactly the prepared bytes. A committed
// result yields the final batch; absent may be retried with the same prepared bytes
// only while the lease is still valid; conflict fails closed.
[[nodiscard]] std::expected<ResolvedCommit, LeaseStoreFailure>
resolve_prepared(Postgres& db, const PreparedCommit& prepared, std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<RegionHead, LeaseStoreFailure>
retry_prepared(Postgres& db, const PreparedCommit& prepared, std::chrono::milliseconds timeout);
} // namespace evo::persistence
+145
View File
@@ -0,0 +1,145 @@
#pragma once
#include "evo/persistence/region_lease.hpp"
#include "evo/processes/environmental_community.hpp"
#include <array>
#include <cstddef>
#include <cstdint>
#include <expected>
#include <span>
#include <vector>
// M7.5 batch evidence (ADR-0063, schemas/region-batch.md): the ordered validation
// trace of host inputs and reports for one checkpoint batch, and the permanent
// outcomes (decisions) derived from it. Portable and storage-free.
namespace evo::persistence {
enum class BatchError : std::uint8_t {
format, // Malformed trace or decision bytes.
sealed, // The batch holds a pending permanent outcome or is full.
empty, // Nothing to commit.
state, // The simulation refused the tick; nothing was recorded.
mismatch, // Replay, receipt or reconciliation disagrees with the record.
};
// Encoded as u32; decoding refuses unknown values rather than truncating them.
enum class Actor : std::uint8_t { producer = 0, consumer = 1 };
enum class Outcome : std::uint8_t { death = 1 };
// One permanent outcome. Identity is (region, outcome, actor, tick); a death is
// terminal, so each actor has at most one. organism is the archived organism for
// the lineage profile and 0 for environmental-v1.
struct BatchDecision {
std::uint64_t tick{};
Actor actor{};
Outcome outcome{Outcome::death};
std::uint64_t organism{};
friend bool operator==(const BatchDecision&, const BatchDecision&) = default;
};
struct TraceStep {
processes::EnvironmentalInput input{};
processes::EnvironmentalReport report{};
friend bool operator==(const TraceStep&, const TraceStep&) = default;
};
// Binds a trace to its place in the chain so it cannot be replayed elsewhere.
struct TraceHeader {
std::uint32_t profile{};
RegionKey key{};
Uuid predecessor_commit{};
Sha256Digest predecessor_digest{};
std::uint64_t begin_tick{}, end_tick{};
friend bool operator==(const TraceHeader&, const TraceHeader&) = default;
};
inline constexpr std::size_t trace_header_bytes = 96;
inline constexpr std::size_t trace_step_bytes = 304;
inline constexpr std::size_t decision_bytes = 24;
// Two actors, each dies at most once; the wire cap leaves room for later profiles.
inline constexpr std::size_t max_batch_decisions = 8;
[[nodiscard]] constexpr std::size_t trace_bytes_for(std::uint64_t ticks) noexcept {
return trace_header_bytes + (static_cast<std::size_t>(ticks) * trace_step_bytes);
}
// One step per tick, begin_tick+1..end_tick, at most 64.
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
encode_trace(const TraceHeader& header, std::span<const TraceStep> steps);
struct DecodedTrace {
TraceHeader header;
std::vector<TraceStep> steps;
};
[[nodiscard]] std::expected<DecodedTrace, BatchError>
decode_trace(std::span<const std::uint8_t> bytes);
// Sorted by (tick, actor), unique actors, death only, ticks within the batch.
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
encode_decisions(std::span<const BatchDecision> decisions, std::uint64_t begin_tick,
std::uint64_t end_tick);
[[nodiscard]] std::expected<std::vector<BatchDecision>, BatchError>
decode_decisions(std::span<const std::uint8_t> bytes, std::uint64_t begin_tick,
std::uint64_t end_tick);
// Every terminal transition in the trace yields exactly one decision, in order.
[[nodiscard]] std::vector<BatchDecision> derive_decisions(std::span<const TraceStep> steps,
std::uint64_t producer_organism,
std::uint64_t consumer_organism);
// Records one pending batch over a single-owner state (EnvironmentalCommunity or
// LineageRegion). Seals itself after any tick with a terminal transition, or at
// 64 ticks; a sealed batch refuses further ticks until confirm() records durable
// confirmation. Nothing here is durable: outcomes are provisional until then.
template <typename State> class RegionBatch {
public:
explicit RegionBatch(State& state) : state_(&state), begin_(tick_of(state)) {}
[[nodiscard]] std::expected<processes::EnvironmentalReport, BatchError>
advance(const processes::EnvironmentalInput& input) {
if (sealed_)
return std::unexpected{BatchError::sealed};
const auto tick = tick_of(*state_) + 1;
auto report = state_->advance(tick, input);
if (!report)
return std::unexpected{BatchError::state};
steps_.push_back({input, *report});
if (report->terminal_transition == 1 || report->consumer_transition == 1 ||
steps_.size() == max_ticks)
sealed_ = true;
return *report;
}
// Host deadline or shutdown: stop and commit what exists.
void seal() noexcept {
sealed_ = true;
}
// Durable confirmation for exactly this batch: the next batch starts here.
[[nodiscard]] std::expected<void, BatchError> confirm(std::uint64_t committed_end_tick) {
if (!sealed_ || steps_.empty() || committed_end_tick != end_tick())
return std::unexpected{BatchError::mismatch};
begin_ = committed_end_tick;
steps_.clear();
sealed_ = false;
return {};
}
[[nodiscard]] bool sealed() const noexcept {
return sealed_;
}
[[nodiscard]] std::uint64_t begin_tick() const noexcept {
return begin_;
}
[[nodiscard]] std::uint64_t end_tick() const noexcept {
return begin_ + steps_.size();
}
[[nodiscard]] std::span<const TraceStep> steps() const noexcept {
return steps_;
}
[[nodiscard]] const State& state() const noexcept {
return *state_;
}
private:
static constexpr std::size_t max_ticks = 64;
static std::uint64_t tick_of(const State& s) {
if constexpr (requires { s.community(); })
return s.community().progress().tick;
else
return s.progress().tick;
}
State* state_;
std::uint64_t begin_;
std::vector<TraceStep> steps_;
bool sealed_{};
};
} // namespace evo::persistence
+16 -2
View File
@@ -1,4 +1,5 @@
#pragma once
#include "evo/persistence/region_batch.hpp"
#include "evo/persistence/region_lease.hpp"
#include "evo/processes/environmental_community.hpp"
#include "evo/processes/lineage_region.hpp"
@@ -34,7 +35,18 @@ struct VersionBundle {
};
inline constexpr VersionBundle environmental_v1_versions{};
// Batch evidence bound by manifest v2 (ADR-0063); all zero at genesis.
struct BatchDigests {
std::uint64_t trace_bytes{};
Sha256Digest trace_sha256{};
std::uint32_t decision_count{};
Sha256Digest decisions_sha256{}; // Zero exactly when there are no decisions.
friend bool operator==(const BatchDigests&, const BatchDigests&) = default;
};
struct CheckpointManifest {
// 2 is the only version written; 1 (M7.4, no batch evidence) still decodes.
std::uint32_t manifest_version{2};
CheckpointProfile profile{CheckpointProfile::environmental_v1};
RegionKey key{};
std::uint64_t frame{};
@@ -53,10 +65,12 @@ struct CheckpointManifest {
DurabilityPolicy durability{DurabilityPolicy::single_volume_development};
std::uint64_t payload_bytes{};
Sha256Digest payload_sha256{};
BatchDigests batch{};
friend bool operator==(const CheckpointManifest&, const CheckpointManifest&) = default;
};
inline constexpr std::size_t checkpoint_manifest_bytes = 280;
inline constexpr std::size_t checkpoint_manifest_v1_bytes = 280;
inline constexpr std::size_t checkpoint_manifest_bytes = 360;
inline constexpr std::size_t max_environmental_payload_bytes = std::size_t{1024} * 1024;
inline constexpr std::size_t max_lineage_payload_bytes = processes::max_lineage_region_bytes;
// Payload cap by profile; zero for an unknown profile.
@@ -95,7 +109,7 @@ genesis_manifest(RegionKey key, std::uint64_t frame, Uuid commit, std::uint64_t
successor_manifest(const CheckpointManifest& predecessor, const Sha256Digest& predecessor_digest,
Uuid commit, const LeaseGrant& lease, std::uint64_t end_tick,
const Sha256Digest& build, std::uint64_t payload_bytes,
const Sha256Digest& payload_sha256);
const Sha256Digest& payload_sha256, const BatchDigests& batch);
// Fully restores payload bytes (envelope checksum, bounds and replay audit), then
// requires manifest identity, end tick, tick quantum, envelope format 2 and a
+42 -5
View File
@@ -1,5 +1,6 @@
#pragma once
#include "evo/persistence/postgres.hpp"
#include "evo/persistence/region_batch.hpp"
#include "evo/persistence/region_lease.hpp"
#include <chrono>
#include <cstdint>
@@ -18,7 +19,9 @@ enum class LeaseStoreError : std::uint8_t {
not_provisioned, // No genesis; runtime can never create one.
exhausted, // Epoch or commit sequence would wrap: fail closed.
unsupported, // Profile/version/build not admitted, or rules/quantum changed.
commit_conflict, // Commit ID already used by another manifest (resolve at M7.5).
commit_conflict, // Commit ID already holds other bytes: never retry it.
outcome_final, // A decision repeats an already committed permanent outcome.
immutable, // Attempted rewrite of committed history (owner paths only).
invalid, // Rejected arguments; nothing changed.
store, // Other SQL/configuration failure; nothing committed.
unknown, // May have executed; reconnect and reconcile before reuse.
@@ -47,11 +50,45 @@ release_region_lease(Postgres& db, RegionKey key, const LeaseGrant& lease,
load_region_head(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<CommittedCheckpoint, LeaseStoreFailure>
load_region_checkpoint(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
// Canonical manifest v1 bytes; the database parses identity, lease and chain from
// them, locks the lease then the head and requires the exact committed predecessor.
// The payload must already be verified in the object store. head_mismatch after
// unknown: load the head and compare its commit ID (full resolution is M7.5).
// Canonical manifest v2 bytes plus the decision block they digest (ADR-0063). The
// database parses identity, lease and chain, locks the lease then the head, requires
// the exact committed predecessor and records decisions and outbox rows in the same
// statement. An exact retry of a committed ID returns its original receipt; changed
// bytes are commit_conflict. The payload and trace must already be verified objects.
[[nodiscard]] std::expected<RegionHead, LeaseStoreFailure>
publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
std::span<const std::uint8_t> decisions,
std::chrono::milliseconds timeout);
enum class Resolution : std::uint8_t {
committed, // Exact bytes are durable; receipt holds the head. Grants no authority.
absent, // Not committed now; proves nothing about a request still in flight.
conflict, // The ID holds other bytes or belongs to another region: fail closed.
};
struct ResolvedCommit {
Resolution status{};
RegionHead receipt{}; // Only for committed.
};
[[nodiscard]] std::expected<ResolvedCommit, LeaseStoreFailure>
resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
std::span<const std::uint8_t> manifest,
std::span<const std::uint8_t> decisions, std::chrono::milliseconds timeout);
struct CommittedDecision {
std::uint64_t commit_sequence{};
BatchDecision decision{};
friend bool operator==(const CommittedDecision&, const CommittedDecision&) = default;
};
// Every committed permanent outcome of the region, ordered by (tick, actor).
[[nodiscard]] std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
load_region_decisions(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
// Undelivered outbox rows (1..64), ordered by (commit, tick, actor). Delivery is
// at-least-once; consumers deduplicate by (region, outcome, actor, tick).
[[nodiscard]] std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
pending_region_outbox(Postgres& db, RegionKey key, std::uint32_t limit,
std::chrono::milliseconds timeout);
// True on first acknowledgement, false when already delivered.
[[nodiscard]] std::expected<bool, LeaseStoreFailure>
acknowledge_region_outbox(Postgres& db, RegionKey key, const BatchDecision& decision,
std::chrono::milliseconds timeout);
} // namespace evo::persistence