Files
aevolution/include/evo/persistence/probe_store.hpp
T
dingoandClaude Opus 5.5 265163a9a6
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Add M10.8d two-process handoff fault matrix and recovery fixes
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.

The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
  abort deadline was in memory): schema v14 overdue_probe_transfers and a
  session-layer sweep abort open outgoing transfers past the ticket
  lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
  took control back: admission of a committed transfer now requires the
  record as that commit left it.

M10.8 is complete. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:17:49 -04:00

96 lines
5.4 KiB
C++

#pragma once
#include "evo/persistence/postgres.hpp"
#include "evo/persistence/region_lease.hpp"
#include <chrono>
#include <cstdint>
#include <expected>
#include <optional>
#include <vector>
// Linux-only calls to the probe record and handoff functions of schema v12 (ADR-0096,
// migrations/0012_probe_handoff.sql). EXPERIMENT: probes are non-biological engineering
// entities (ADR-0094). Each call is one statement under the caller's region lease: a fenced
// or expired lease refuses everything. No retries; every transfer call can be retried with
// the same transfer id and returns the recorded outcome.
namespace evo::persistence {
enum class ProbeStoreError : std::uint8_t {
fenced, // Lease, ownership, generation or frozen state no longer allows it.
unknown_transfer, // No transfer with this id in the world.
not_staged, // Commit before the caller's current epoch staged the transfer.
exhausted, // Generations would wrap: fail closed.
invalid, // Rejected arguments (or an unprovisioned region); nothing changed.
store, // Other SQL/configuration failure; nothing committed.
unknown, // May have executed; reconnect and reconcile (retry the same id).
};
struct ProbeStoreFailure {
ProbeStoreError kind{};
PgFailure cause{};
};
struct ProbeRecord {
std::uint64_t region{}, generation{}; // Owning region; generation of this ownership.
std::int64_t x{}, y{}, z{}; // Micrometres; z is the cell slice.
std::uint64_t inventory{}; // EXPERIMENT conserved test inventory.
std::optional<Uuid> transfer; // Frozen under this open transfer.
};
enum class TransferState : std::uint8_t { prepared, staged, committed, aborted };
struct ProbeTransfer {
TransferState state{};
Uuid account{};
std::uint64_t generation{}, source{}, destination{};
std::int64_t x{}, y{}, z{};
std::uint64_t inventory{};
friend bool operator==(const ProbeTransfer&, const ProbeTransfer&) = default;
};
// The account's probe, created owned by the caller's region (at the position, with the
// inventory) when the world has none. The caller acts for it only if it owns it and it is
// not frozen.
[[nodiscard]] std::expected<ProbeRecord, ProbeStoreFailure>
open_probe(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
std::int64_t x, std::int64_t y, std::int64_t z, std::uint64_t inventory,
std::chrono::milliseconds timeout);
// Records the active, owned probe's position (fenced otherwise).
[[nodiscard]] std::expected<void, ProbeStoreFailure>
save_probe(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
std::uint64_t generation, std::int64_t x, std::int64_t y, std::int64_t z,
std::chrono::milliseconds timeout);
// Source: freezes the probe at its final position under `transfer` for `destination`.
[[nodiscard]] std::expected<ProbeTransfer, ProbeStoreFailure>
prepare_probe_transfer(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
std::uint64_t generation, const Uuid& transfer, std::uint64_t destination,
std::int64_t x, std::int64_t y, std::int64_t z,
std::chrono::milliseconds timeout);
// Destination: durably accepts it under its current epoch.
[[nodiscard]] std::expected<ProbeTransfer, ProbeStoreFailure>
stage_probe_transfer(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& transfer,
std::chrono::milliseconds timeout);
// Destination: the authority switch (staged by this epoch only).
[[nodiscard]] std::expected<ProbeTransfer, ProbeStoreFailure>
commit_probe_transfer(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& transfer,
std::chrono::milliseconds timeout);
// Source or destination: abandons it before its commit; the source resumes.
[[nodiscard]] std::expected<ProbeTransfer, ProbeStoreFailure>
abort_probe_transfer(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& transfer,
std::chrono::milliseconds timeout);
// Any region of the world: the transfer as recorded, or nothing.
[[nodiscard]] std::expected<std::optional<ProbeTransfer>, ProbeStoreFailure>
find_probe_transfer(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& transfer,
std::chrono::milliseconds timeout);
// Schema v13: the open (prepared or staged) transfers addressed to the caller's region, for
// read-only ghosts (at most 256, oldest first).
struct IncomingTransfer {
Uuid transfer{}, account{};
std::uint64_t source{};
std::int64_t x{}, y{}, z{};
friend bool operator==(const IncomingTransfer&, const IncomingTransfer&) = default;
};
[[nodiscard]] std::expected<std::vector<IncomingTransfer>, ProbeStoreFailure>
incoming_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::milliseconds timeout);
// Schema v14: the caller's open outgoing transfers prepared more than `age` (1 s .. 1 day)
// ago by the database clock, for recovery aborts after a restart (at most 256, oldest first).
[[nodiscard]] std::expected<std::vector<Uuid>, ProbeStoreFailure>
overdue_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::seconds age, std::chrono::milliseconds timeout);
} // namespace evo::persistence