Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped

ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.

The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
  abort deadline was in memory): schema v14 overdue_probe_transfers and a
  session-layer sweep abort open outgoing transfers past the ticket
  lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
  took control back: admission of a committed transfer now requires the
  record as that commit left it.

M10.8 is complete. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-06 23:17:49 -04:00
co-authored by Claude Opus 5.5
parent dabeee8e8b
commit 265163a9a6
26 changed files with 856 additions and 20 deletions
+23
View File
@@ -748,4 +748,27 @@ incoming_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
}
return out;
}
std::expected<std::vector<Uuid>, ProbeStoreFailure>
overdue_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::seconds age, milliseconds timeout) {
if (age.count() < 1 || age.count() > 86400)
return std::unexpected{ProbeStoreFailure{ProbeStoreError::invalid, {}}};
const std::array<std::string, 5> p{std::to_string(key.world), std::to_string(key.region),
uuid_text(lease.holder), std::to_string(lease.epoch),
std::to_string(age.count())};
auto rows = db.query("SELECT * FROM evolution.overdue_probe_transfers($1::evolution.u64, "
"$2::evolution.u64, $3::uuid, $4::evolution.u64, $5::integer)",
p, timeout);
if (!rows)
return std::unexpected{probe_failure(rows.error())};
std::vector<Uuid> out;
for (const auto& r : *rows) {
const auto transfer = r.size() == 1 ? parse_uuid(r[0]) : std::nullopt;
if (!transfer)
return std::unexpected{ProbeStoreFailure{ProbeStoreError::unknown, {}}};
out.push_back(*transfer);
}
return out;
}
} // namespace evo::persistence