Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and 7/162, adjacent areas) run crossings and crossings back, SIGKILL of the source after prepare and of the destination after stage and after commit, duplicate and stale ticket retries, a client that never arrives and a destination that is down. After every scenario SQL checks one owner per probe, the conserved inventory and frozen-iff-open-transfer. The matrix found two recovery gaps, now fixed: - a source restarted after a prepare left the probe frozen for good (its abort deadline was in memory): schema v14 overdue_probe_transfers and a session-layer sweep abort open outgoing transfers past the ticket lifetime plus 10 s by the database clock; - a ticket replayed at a restarted destination after the probe moved on took control back: admission of a committed transfer now requires the record as that commit left it. M10.8 is complete. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -33,7 +33,7 @@ class PostgresHandoffLedger final : public network::HandoffLedger {
|
||||
std::chrono::milliseconds duration_, timeout_;
|
||||
};
|
||||
|
||||
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-13), on the
|
||||
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-14), on the
|
||||
// host's lease and its own connection slot. Undecided or failed calls report `unavailable`
|
||||
// (the session layer retries or refuses); a fenced lease reports `fenced`.
|
||||
class PostgresProbeLedger final : public ProbeLedger {
|
||||
@@ -55,6 +55,8 @@ class PostgresProbeLedger final : public ProbeLedger {
|
||||
[[nodiscard]] std::expected<bool, ProbeLedgerError>
|
||||
abort(const network::TransferId& transfer) override;
|
||||
[[nodiscard]] std::expected<std::vector<IncomingProbe>, ProbeLedgerError> incoming() override;
|
||||
[[nodiscard]] std::expected<std::vector<network::TransferId>, ProbeLedgerError>
|
||||
overdue(std::chrono::seconds age) override;
|
||||
|
||||
private:
|
||||
PgSlot* db_;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#pragma once
|
||||
#include "evo/network/handoff_ticket.hpp"
|
||||
#include "evo/network/session_control.hpp"
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <vector>
|
||||
@@ -43,6 +44,10 @@ class ProbeLedger {
|
||||
// Open transfers addressed to this region (shown as ghosts).
|
||||
[[nodiscard]] virtual std::expected<std::vector<IncomingProbe>, ProbeLedgerError>
|
||||
incoming() = 0;
|
||||
// M10.8d: this region's open outgoing transfers prepared more than `age` ago by the
|
||||
// store's clock, whoever prepared them (an earlier process too): recovery aborts them.
|
||||
[[nodiscard]] virtual std::expected<std::vector<network::TransferId>, ProbeLedgerError>
|
||||
overdue(std::chrono::seconds age) = 0;
|
||||
|
||||
protected:
|
||||
ProbeLedger(ProbeLedger&&) = default;
|
||||
|
||||
@@ -88,6 +88,9 @@ struct RegionSessionsConfig {
|
||||
std::vector<Neighbour> neighbours;
|
||||
std::uint64_t ticket_lifetime{30}; // Seconds a handoff ticket is valid.
|
||||
std::chrono::milliseconds ghost_poll{1000}; // Incoming transfers read this often.
|
||||
// M10.8d recovery: this often, open outgoing transfers prepared more than ticket_lifetime
|
||||
// + 10 s ago (by the store's clock, e.g. by a process that died) are aborted.
|
||||
std::chrono::milliseconds overdue_poll{5000};
|
||||
};
|
||||
// The region host's authority, read once per poll.
|
||||
struct HostFacts {
|
||||
|
||||
Reference in New Issue
Block a user