Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped

ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.

The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
  abort deadline was in memory): schema v14 overdue_probe_transfers and a
  session-layer sweep abort open outgoing transfers past the ticket
  lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
  took control back: admission of a committed transfer now requires the
  record as that commit left it.

M10.8 is complete. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-06 23:17:49 -04:00
co-authored by Claude Opus 5.5
parent dabeee8e8b
commit 265163a9a6
26 changed files with 856 additions and 20 deletions
+5
View File
@@ -87,4 +87,9 @@ struct IncomingTransfer {
[[nodiscard]] std::expected<std::vector<IncomingTransfer>, ProbeStoreFailure>
incoming_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::milliseconds timeout);
// Schema v14: the caller's open outgoing transfers prepared more than `age` (1 s .. 1 day)
// ago by the database clock, for recovery aborts after a restart (at most 256, oldest first).
[[nodiscard]] std::expected<std::vector<Uuid>, ProbeStoreFailure>
overdue_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::seconds age, std::chrono::milliseconds timeout);
} // namespace evo::persistence
@@ -33,7 +33,7 @@ class PostgresHandoffLedger final : public network::HandoffLedger {
std::chrono::milliseconds duration_, timeout_;
};
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-13), on the
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-14), on the
// host's lease and its own connection slot. Undecided or failed calls report `unavailable`
// (the session layer retries or refuses); a fenced lease reports `fenced`.
class PostgresProbeLedger final : public ProbeLedger {
@@ -55,6 +55,8 @@ class PostgresProbeLedger final : public ProbeLedger {
[[nodiscard]] std::expected<bool, ProbeLedgerError>
abort(const network::TransferId& transfer) override;
[[nodiscard]] std::expected<std::vector<IncomingProbe>, ProbeLedgerError> incoming() override;
[[nodiscard]] std::expected<std::vector<network::TransferId>, ProbeLedgerError>
overdue(std::chrono::seconds age) override;
private:
PgSlot* db_;
+5
View File
@@ -1,6 +1,7 @@
#pragma once
#include "evo/network/handoff_ticket.hpp"
#include "evo/network/session_control.hpp"
#include <chrono>
#include <cstdint>
#include <expected>
#include <vector>
@@ -43,6 +44,10 @@ class ProbeLedger {
// Open transfers addressed to this region (shown as ghosts).
[[nodiscard]] virtual std::expected<std::vector<IncomingProbe>, ProbeLedgerError>
incoming() = 0;
// M10.8d: this region's open outgoing transfers prepared more than `age` ago by the
// store's clock, whoever prepared them (an earlier process too): recovery aborts them.
[[nodiscard]] virtual std::expected<std::vector<network::TransferId>, ProbeLedgerError>
overdue(std::chrono::seconds age) = 0;
protected:
ProbeLedger(ProbeLedger&&) = default;
+3
View File
@@ -88,6 +88,9 @@ struct RegionSessionsConfig {
std::vector<Neighbour> neighbours;
std::uint64_t ticket_lifetime{30}; // Seconds a handoff ticket is valid.
std::chrono::milliseconds ghost_poll{1000}; // Incoming transfers read this often.
// M10.8d recovery: this often, open outgoing transfers prepared more than ticket_lifetime
// + 10 s ago (by the store's clock, e.g. by a process that died) are aborted.
std::chrono::milliseconds overdue_poll{5000};
};
// The region host's authority, read once per poll.
struct HostFacts {