Files
aevolution/migrations/0014_probe_overdue.sql
dingoandClaude Opus 5.5 265163a9a6
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Add M10.8d two-process handoff fault matrix and recovery fixes
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and
7/162, adjacent areas) run crossings and crossings back, SIGKILL of the
source after prepare and of the destination after stage and after commit,
duplicate and stale ticket retries, a client that never arrives and a
destination that is down. After every scenario SQL checks one owner per
probe, the conserved inventory and frozen-iff-open-transfer.

The matrix found two recovery gaps, now fixed:
- a source restarted after a prepare left the probe frozen for good (its
  abort deadline was in memory): schema v14 overdue_probe_transfers and a
  session-layer sweep abort open outgoing transfers past the ticket
  lifetime plus 10 s by the database clock;
- a ticket replayed at a restarted destination after the probe moved on
  took control back: admission of a committed transfer now requires the
  record as that commit left it.

M10.8 is complete. Refreshes the dev session window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-06 23:17:49 -04:00

36 lines
1.8 KiB
PL/PgSQL

-- M10.8d overdue probe transfers (ADR-0096). A source region aborts the transfers whose
-- client never arrived once their ticket has expired; it kept that deadline only in memory,
-- so a source restarted after a prepare (SIGKILL, lease loss) left the probe frozen for good.
-- This lists the caller's open outgoing transfers prepared more than `p_age_seconds` ago by
-- the database clock, so any later process holding the region's lease can abort them (abort
-- is serialized with stage and commit: a transfer committed meanwhile stays committed).
-- Read-only, under the caller's exact unexpired region lease. At most 256 rows, oldest first.
CREATE FUNCTION evolution.overdue_probe_transfers(p_world evolution.u64, p_region evolution.u64,
p_holder uuid, p_epoch evolution.u64, p_age_seconds integer)
RETURNS TABLE (transfer_id uuid)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE
now_t timestamptz;
BEGIN
IF p_age_seconds IS NULL OR p_age_seconds NOT BETWEEN 1 AND 86400 THEN
RAISE EXCEPTION 'invalid probe request' USING ERRCODE = '22023';
END IF;
now_t := evolution.check_region_lease(p_world, p_region, p_holder, p_epoch);
RETURN QUERY
SELECT t.transfer_id
FROM evolution.probe_transfers t
WHERE t.world_id = p_world AND t.source_region = p_region
AND t.state IN ('prepared', 'staged')
AND t.prepared_at < now_t - p_age_seconds * interval '1 second'
ORDER BY t.prepared_at, t.transfer_id
LIMIT 256;
END $$;
REVOKE ALL ON FUNCTION
evolution.overdue_probe_transfers(evolution.u64, evolution.u64, uuid, evolution.u64, integer)
FROM PUBLIC;
GRANT EXECUTE ON FUNCTION
evolution.overdue_probe_transfers(evolution.u64, evolution.u64, uuid, evolution.u64, integer)
TO :"runtime_role";
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (14, :'migration_sha_14');