Add M10.8d two-process handoff fault matrix and recovery fixes
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Foundation / checks (deploy) (push) Successful in 3m20s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (client) (push) Failing after 10m4s
Foundation / checks (network) (push) Successful in 13m3s
Foundation / checks (tsan) (push) Successful in 7m53s
Foundation / checks (checks) (push) Successful in 32m18s
Foundation / checks (postgres) (push) Successful in 43m26s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
ADR-0096. Two real evo-server processes on PostgreSQL (regions 7/161 and 7/162, adjacent areas) run crossings and crossings back, SIGKILL of the source after prepare and of the destination after stage and after commit, duplicate and stale ticket retries, a client that never arrives and a destination that is down. After every scenario SQL checks one owner per probe, the conserved inventory and frozen-iff-open-transfer. The matrix found two recovery gaps, now fixed: - a source restarted after a prepare left the probe frozen for good (its abort deadline was in memory): schema v14 overdue_probe_transfers and a session-layer sweep abort open outgoing transfers past the ticket lifetime plus 10 s by the database clock; - a ticket replayed at a restarted destination after the probe moved on took control back: admission of a committed transfer now requires the record as that commit left it. M10.8 is complete. Refreshes the dev session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+3
-1
@@ -264,7 +264,9 @@ if(EVO_BUILD_POSTGRES)
|
||||
endif()
|
||||
if(EVO_BUILD_QUIC)
|
||||
# M10.4c: evo-server with the QUIC listener, real clients and the probe (ADR-0092).
|
||||
target_sources(evo-postgres-tests PRIVATE tests/region_m10_session_tests.cpp)
|
||||
# M10.8d: two evo-server processes handing probes over (ADR-0096).
|
||||
target_sources(evo-postgres-tests PRIVATE tests/region_m10_session_tests.cpp
|
||||
tests/region_handoff_process_tests.cpp)
|
||||
target_link_libraries(evo-postgres-tests PRIVATE evo_session_client)
|
||||
target_compile_definitions(evo-postgres-tests PRIVATE EVO_POSTGRES_M10=1)
|
||||
add_dependencies(evo-postgres-tests evo-session-probe)
|
||||
|
||||
@@ -12,7 +12,7 @@ spec:
|
||||
env:
|
||||
- {name: REGION_SESSION, value: "dev"}
|
||||
- {name: REGION_SESSION_LISTEN, value: "0.0.0.0:7400"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261007T015045-2140:1791339045"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261007T031149-a452:1791343909"}
|
||||
ports:
|
||||
# QUIC (ADR-0090): UDP only; nothing listens on TCP 7400.
|
||||
- {name: session, containerPort: 7400, protocol: UDP}
|
||||
|
||||
+3
-1
@@ -539,7 +539,9 @@ transfer state machine with a PostgreSQL fault matrix done; 10.8b signed handoff
|
||||
ticket admission (durable stage, commit and control move) done; 10.8c-1 session protocol v4,
|
||||
durable probes in sessions, edge crossing, ghosts (schema v13) and a two-region loopback test
|
||||
done; 10.8c-2 client switch (offers, redirects, multi-region grants) and evo-server/probe
|
||||
configuration done; 10.8d two-process kill tests follow. 10.9 planned.
|
||||
configuration done; 10.8d two evo-server processes on PostgreSQL with SIGKILL around each
|
||||
durable step, duplicate/stale retries and a failed destination done (schema v14 recovery
|
||||
sweep, replay guard). M10.8 complete. 10.9 planned.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+12
-1
@@ -87,7 +87,18 @@ the session. M10.8c-1 wires the regions:
|
||||
M10.8c-2 has the engineering client follow crossings (ticket, no loading screen) and
|
||||
redirects, with grants picked by region in rising serial order. evo-server gains
|
||||
`--m10-area`, `--m10-handoff-key` and `--m10-neighbour`; the probe tool writes multi-region
|
||||
bundles. Next: 10.8d, two evo-server processes on PostgreSQL with kills at each step.
|
||||
bundles. M10.8d runs two evo-server processes on PostgreSQL through crossings and
|
||||
crossings back, with SIGKILL of the source after prepare and of the destination after stage
|
||||
and after commit, duplicate and stale ticket retries, a client that never arrives and a
|
||||
destination that is down. After every scenario each probe has one owner, the inventory is
|
||||
conserved and a record is frozen only under its open transfer. It found and fixed two
|
||||
recovery gaps:
|
||||
- a source restarted after a prepare now aborts overdue transfers from the database
|
||||
(schema v14);
|
||||
- a ticket replayed at a restarted destination after the probe moved on is refused.
|
||||
|
||||
M10.8 is complete. Deploying needs protocol v4 and schemas v12-v14 (a rebuilt migrations
|
||||
image). Next: M10.9, load and abuse qualification.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -220,3 +220,50 @@ controllable before M11.
|
||||
It passed 3 times in a row on Linux and also passes on Windows.
|
||||
- The test fixture `tests/fixtures/fake_handoff_world.hpp` (the v12 state machine) is
|
||||
shared by both handoff tests.
|
||||
|
||||
## M10.8d two-process fault matrix (2026-10-06)
|
||||
Real evo-server processes on PostgreSQL found two recovery gaps. Both are fixed within this
|
||||
ADR's decisions, with no change to the protocol or the ticket format:
|
||||
- **A source restarted after a prepare left the probe frozen for good.** The abort deadline
|
||||
for a client that never arrives was kept only in the source process's memory. Schema v14
|
||||
adds `overdue_probe_transfers`: read-only, under the lease, it lists the caller's open
|
||||
outgoing transfers prepared more than a given age ago by the database clock. Every
|
||||
`overdue_poll` (5 s), sessions abort those older than the ticket lifetime plus 10 s,
|
||||
whichever process prepared them. The in-memory deadline (lifetime plus 5 s) stays as the
|
||||
fast path. Abort is serialized with the destination's stage and commit, so a transfer
|
||||
committed meanwhile stays committed: clock skew can only delay an abort.
|
||||
- **A ticket replayed at a restarted destination could take control back.** The
|
||||
destination's used-ticket list is in memory. After its restart, a ticket for a transfer
|
||||
committed earlier (still within its lifetime) was admitted durably even after the probe
|
||||
had moved on: it moved the account's control away from the live owner and opened a
|
||||
phantom probe. Ownership in the database stayed unique. `PostgresHandoffLedger::admit`
|
||||
now admits a committed transfer only while the record is as that commit left it: owned
|
||||
here, at the transfer's generation plus 1, and not frozen. Otherwise it refuses
|
||||
(`unauthenticated`). An undecided retry right after the commit still converges.
|
||||
|
||||
Evidence: `tests/region_handoff_process_tests.cpp` (in `evo-postgres-tests`, regions 7/161
|
||||
and 7/162 with adjacent areas, handoff keys and each other as neighbour), with these
|
||||
scenarios:
|
||||
1. A crossing and a crossing back (generation 1, 2, 3). A stale grant login at the old
|
||||
region is redirected (`moved`).
|
||||
2. Destination SIGKILL after stage (staged under its lease, then killed). The restarted
|
||||
destination stages again under its new epoch and commits on the ticket. After the probe
|
||||
crosses back, a replay of that ticket at a destination process that never saw it is
|
||||
refused, and control stays with the owner.
|
||||
3. Destination SIGKILL after commit (client admitted). A retry of the same ticket at the
|
||||
restarted destination converges: same commit, and control moves again. A duplicate in
|
||||
the same process is refused locally (`stale_grant`).
|
||||
4. Source SIGKILL after prepare. One client's ticket still commits at the destination. The
|
||||
other client never arrives, and its probe is refused as frozen (`capacity`) at the
|
||||
restarted source until the recovery sweep aborts it. Then it logs in where it stopped.
|
||||
5. Destination down. The ticket cannot be used, and the live source aborts after the
|
||||
ticket's lifetime. The late ticket at the restarted destination is refused (`expired`).
|
||||
|
||||
After every scenario, SQL checks that:
|
||||
- each probe has one owning region (A or B);
|
||||
- the inventory total is conserved, in records and in transfers;
|
||||
- each record is frozen exactly while its single open transfer exists.
|
||||
|
||||
Library tests add the v14 call (age bounds, source only, decided transfers excluded), the
|
||||
refused replay in the ledger test, and the sweep over the fake world (never before the
|
||||
store reports a transfer overdue).
|
||||
|
||||
@@ -4858,3 +4858,49 @@ Not done:
|
||||
- real two-process handoffs on PostgreSQL with kills (10.8d);
|
||||
- a GUI run across regions on evolution-dev (one dev region only; needs v4 and v13
|
||||
deployed).
|
||||
|
||||
## 2026-10-06 — M10.8d two-process handoff fault matrix (ADR-0096)
|
||||
|
||||
Implemented:
|
||||
- `tests/region_handoff_process_tests.cpp`: two evo-server processes (regions 7/161 and
|
||||
7/162, adjacent `--m10-area`s, handoff keys and each other as `--m10-neighbour`, fixed
|
||||
loopback ports) with in-process session clients that follow offers and tickets. It covers:
|
||||
- a crossing and a crossing back, plus a redirected stale login;
|
||||
- destination SIGKILL after stage, then a re-stage and commit under the new epoch;
|
||||
- destination SIGKILL after commit, then a converging ticket retry;
|
||||
- a refused ticket replay after the probe moved on;
|
||||
- a refused same-process duplicate;
|
||||
- source SIGKILL after prepare: one client still arrives, the other never does and is
|
||||
aborted by the restarted source's sweep;
|
||||
- a destination that is down, with the live source aborting and the late ticket expired.
|
||||
|
||||
After every scenario, SQL checks one owner per probe, the conserved inventory, and
|
||||
frozen-iff-open-transfer.
|
||||
- Schema v14 `overdue_probe_transfers` (apply chain, audit allowlist, upgrade and drift
|
||||
checks), `ProbeLedger::overdue`, and the session layer's recovery sweep
|
||||
(`overdue_poll`, ticket lifetime plus 10 s).
|
||||
- The replay guard in `PostgresHandoffLedger::admit`: a committed transfer admits only
|
||||
while the record is still owned here at generation plus 1 and not frozen.
|
||||
- Library tests:
|
||||
- the v14 store call (age bounds, source only, aborted transfers excluded);
|
||||
- the refused retry in the ledger test;
|
||||
- the sweep over the fake world in `two_region_handoff`.
|
||||
|
||||
Findings: the two recovery gaps above were found by this matrix, before any deployment of
|
||||
v4/v12/v13.
|
||||
|
||||
Executed:
|
||||
- **Linux GCC 14:** `postgres-check.sh` with QUIC, all suites including the new one (restart
|
||||
waits 0.03-4 s; the replay ran at 8 s of the 30 s ticket lifetime). It passed 3 times in a
|
||||
row, plus a final run after a tidy-only change. Presets `quic` 4/4 and `client-network`
|
||||
16/16.
|
||||
- **Linux Clang 19:** clang-tidy 19 on `region_sessions.cpp`, `postgres_handoff_ledger.cpp`,
|
||||
`region_lease_store.cpp`, the QUIC session, remote client, probe store, process,
|
||||
migration and Postgres runner tests (one finding fixed); clang-format 19.
|
||||
- **Windows MSVC 19.44:** presets `quic` 4/4 and `client-network` 14/14. The PostgreSQL
|
||||
process tests are Linux-only.
|
||||
|
||||
Not done:
|
||||
- deployment of protocol v4 and schemas v12-v14 (needs a rebuilt migrations image; not done
|
||||
from routine development);
|
||||
- a GUI run across two regions on evolution-dev (only one dev region exists).
|
||||
|
||||
@@ -218,8 +218,11 @@ Lock order: upgrade lock (shared), own region lease row, probe record, transfer;
|
||||
`handoff_account_control` reads the transfer, then locks the account control row. The
|
||||
privilege audit covers all eight functions, and the tables are owner-only. Schema v13 adds
|
||||
`incoming_probe_transfers`, a read-only function under the lease that lists the open
|
||||
transfers addressed to the caller's region (for ghosts). Deploying v12 and v13 needs a
|
||||
rebuilt migrations (tools) image.
|
||||
transfers addressed to the caller's region (for ghosts). Schema v14 adds
|
||||
`overdue_probe_transfers`: read-only, under the lease, it lists the caller's open outgoing
|
||||
transfers prepared more than the given age (1 s to 1 day) ago by the database clock. A
|
||||
restarted source uses it to abort transfers whose client never arrived (the dead process's
|
||||
deadlines were in memory). Deploying v12-v14 needs a rebuilt migrations (tools) image.
|
||||
|
||||
## Host lease discipline
|
||||
|
||||
|
||||
@@ -87,4 +87,9 @@ struct IncomingTransfer {
|
||||
[[nodiscard]] std::expected<std::vector<IncomingTransfer>, ProbeStoreFailure>
|
||||
incoming_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Schema v14: the caller's open outgoing transfers prepared more than `age` (1 s .. 1 day)
|
||||
// ago by the database clock, for recovery aborts after a restart (at most 256, oldest first).
|
||||
[[nodiscard]] std::expected<std::vector<Uuid>, ProbeStoreFailure>
|
||||
overdue_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
|
||||
std::chrono::seconds age, std::chrono::milliseconds timeout);
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -33,7 +33,7 @@ class PostgresHandoffLedger final : public network::HandoffLedger {
|
||||
std::chrono::milliseconds duration_, timeout_;
|
||||
};
|
||||
|
||||
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-13), on the
|
||||
// M10.8c the region's durable probe records over PostgreSQL (ADR-0096, schema v12-14), on the
|
||||
// host's lease and its own connection slot. Undecided or failed calls report `unavailable`
|
||||
// (the session layer retries or refuses); a fenced lease reports `fenced`.
|
||||
class PostgresProbeLedger final : public ProbeLedger {
|
||||
@@ -55,6 +55,8 @@ class PostgresProbeLedger final : public ProbeLedger {
|
||||
[[nodiscard]] std::expected<bool, ProbeLedgerError>
|
||||
abort(const network::TransferId& transfer) override;
|
||||
[[nodiscard]] std::expected<std::vector<IncomingProbe>, ProbeLedgerError> incoming() override;
|
||||
[[nodiscard]] std::expected<std::vector<network::TransferId>, ProbeLedgerError>
|
||||
overdue(std::chrono::seconds age) override;
|
||||
|
||||
private:
|
||||
PgSlot* db_;
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
#pragma once
|
||||
#include "evo/network/handoff_ticket.hpp"
|
||||
#include "evo/network/session_control.hpp"
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <vector>
|
||||
@@ -43,6 +44,10 @@ class ProbeLedger {
|
||||
// Open transfers addressed to this region (shown as ghosts).
|
||||
[[nodiscard]] virtual std::expected<std::vector<IncomingProbe>, ProbeLedgerError>
|
||||
incoming() = 0;
|
||||
// M10.8d: this region's open outgoing transfers prepared more than `age` ago by the
|
||||
// store's clock, whoever prepared them (an earlier process too): recovery aborts them.
|
||||
[[nodiscard]] virtual std::expected<std::vector<network::TransferId>, ProbeLedgerError>
|
||||
overdue(std::chrono::seconds age) = 0;
|
||||
|
||||
protected:
|
||||
ProbeLedger(ProbeLedger&&) = default;
|
||||
|
||||
@@ -88,6 +88,9 @@ struct RegionSessionsConfig {
|
||||
std::vector<Neighbour> neighbours;
|
||||
std::uint64_t ticket_lifetime{30}; // Seconds a handoff ticket is valid.
|
||||
std::chrono::milliseconds ghost_poll{1000}; // Incoming transfers read this often.
|
||||
// M10.8d recovery: this often, open outgoing transfers prepared more than ticket_lifetime
|
||||
// + 10 s ago (by the store's clock, e.g. by a process that died) are aborted.
|
||||
std::chrono::milliseconds overdue_poll{5000};
|
||||
};
|
||||
// The region host's authority, read once per poll.
|
||||
struct HostFacts {
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
-- M10.8d overdue probe transfers (ADR-0096). A source region aborts the transfers whose
|
||||
-- client never arrived once their ticket has expired; it kept that deadline only in memory,
|
||||
-- so a source restarted after a prepare (SIGKILL, lease loss) left the probe frozen for good.
|
||||
-- This lists the caller's open outgoing transfers prepared more than `p_age_seconds` ago by
|
||||
-- the database clock, so any later process holding the region's lease can abort them (abort
|
||||
-- is serialized with stage and commit: a transfer committed meanwhile stays committed).
|
||||
-- Read-only, under the caller's exact unexpired region lease. At most 256 rows, oldest first.
|
||||
CREATE FUNCTION evolution.overdue_probe_transfers(p_world evolution.u64, p_region evolution.u64,
|
||||
p_holder uuid, p_epoch evolution.u64, p_age_seconds integer)
|
||||
RETURNS TABLE (transfer_id uuid)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
now_t timestamptz;
|
||||
BEGIN
|
||||
IF p_age_seconds IS NULL OR p_age_seconds NOT BETWEEN 1 AND 86400 THEN
|
||||
RAISE EXCEPTION 'invalid probe request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
now_t := evolution.check_region_lease(p_world, p_region, p_holder, p_epoch);
|
||||
RETURN QUERY
|
||||
SELECT t.transfer_id
|
||||
FROM evolution.probe_transfers t
|
||||
WHERE t.world_id = p_world AND t.source_region = p_region
|
||||
AND t.state IN ('prepared', 'staged')
|
||||
AND t.prepared_at < now_t - p_age_seconds * interval '1 second'
|
||||
ORDER BY t.prepared_at, t.transfer_id
|
||||
LIMIT 256;
|
||||
END $$;
|
||||
|
||||
REVOKE ALL ON FUNCTION
|
||||
evolution.overdue_probe_transfers(evolution.u64, evolution.u64, uuid, evolution.u64, integer)
|
||||
FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION
|
||||
evolution.overdue_probe_transfers(evolution.u64, evolution.u64, uuid, evolution.u64, integer)
|
||||
TO :"runtime_role";
|
||||
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (14, :'migration_sha_14');
|
||||
+27
-3
@@ -65,28 +65,43 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|
||||
|| ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10'
|
||||
|| ',11:' || :'migration_sha_11' || ',12:' || :'migration_sha_12'
|
||||
|| ',13:' || :'migration_sha_13' AS at_v13 \gset
|
||||
\if :at_v13
|
||||
|| ',13:' || :'migration_sha_13' AS at_v13,
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|
||||
|| ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10'
|
||||
|| ',11:' || :'migration_sha_11' || ',12:' || :'migration_sha_12'
|
||||
|| ',13:' || :'migration_sha_13' || ',14:' || :'migration_sha_14'
|
||||
AS at_v14 \gset
|
||||
\if :at_v14
|
||||
\elif :at_v13
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v12
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v11
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v10
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v9
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v8
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v7
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
@@ -94,6 +109,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v6
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
@@ -102,6 +118,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v5
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
@@ -111,6 +128,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v4
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
@@ -122,6 +140,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v3
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
@@ -134,6 +153,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v2
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
@@ -147,6 +167,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\elif :at_v1
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
@@ -161,6 +182,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\else
|
||||
\echo 'Unknown or changed migration history'
|
||||
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
|
||||
@@ -180,6 +202,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0011_region_actions.sql
|
||||
\ir 0012_probe_handoff.sql
|
||||
\ir 0013_probe_incoming.sql
|
||||
\ir 0014_probe_overdue.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
|
||||
@@ -226,7 +249,8 @@ BEGIN
|
||||
('evolution.abort_probe_transfer(evolution.u64,evolution.u64,uuid,evolution.u64,uuid)'::regprocedure::oid),
|
||||
('evolution.find_probe_transfer(evolution.u64,evolution.u64,uuid,evolution.u64,uuid)'::regprocedure::oid),
|
||||
('evolution.handoff_account_control(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,uuid,evolution.u64,integer)'::regprocedure::oid),
|
||||
('evolution.incoming_probe_transfers(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid)
|
||||
('evolution.incoming_probe_transfers(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.overdue_probe_transfers(evolution.u64,evolution.u64,uuid,evolution.u64,integer)'::regprocedure::oid)
|
||||
), violations(role_oid, reason) AS (
|
||||
SELECT r.oid, 'privileged attribute or owner identity' FROM pg_catalog.pg_roles r
|
||||
JOIN closure c ON c.oid = r.oid
|
||||
|
||||
@@ -41,6 +41,8 @@ Each call is made under the caller's exact unexpired region lease.
|
||||
| abort_probe_transfer | source or destination | prepared or staged becomes aborted; the record is active at the source |
|
||||
| find_probe_transfer | any region | The transfer as recorded |
|
||||
| handoff_account_control | destination | Account control (v10) for a transfer committed to the caller |
|
||||
| incoming_probe_transfers (v13) | any region | Open transfers addressed to the caller (ghosts), at most 256 |
|
||||
| overdue_probe_transfers (v14) | source | Caller's open outgoing transfers prepared more than the given age (1 s to 1 day) ago by the database clock, at most 256 (recovery aborts) |
|
||||
|
||||
A retry with the same transfer id returns the recorded outcome. SQLSTATE: EVL13 fenced,
|
||||
EVL14 unknown transfer, EVL15 not staged by this epoch, EVL12 control handoff refused.
|
||||
|
||||
@@ -748,4 +748,27 @@ incoming_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::expected<std::vector<Uuid>, ProbeStoreFailure>
|
||||
overdue_probe_transfers(Postgres& db, RegionKey key, const LeaseGrant& lease,
|
||||
std::chrono::seconds age, milliseconds timeout) {
|
||||
if (age.count() < 1 || age.count() > 86400)
|
||||
return std::unexpected{ProbeStoreFailure{ProbeStoreError::invalid, {}}};
|
||||
const std::array<std::string, 5> p{std::to_string(key.world), std::to_string(key.region),
|
||||
uuid_text(lease.holder), std::to_string(lease.epoch),
|
||||
std::to_string(age.count())};
|
||||
auto rows = db.query("SELECT * FROM evolution.overdue_probe_transfers($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::uuid, $4::evolution.u64, $5::integer)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{probe_failure(rows.error())};
|
||||
std::vector<Uuid> out;
|
||||
for (const auto& r : *rows) {
|
||||
const auto transfer = r.size() == 1 ? parse_uuid(r[0]) : std::nullopt;
|
||||
if (!transfer)
|
||||
return std::unexpected{ProbeStoreFailure{ProbeStoreError::unknown, {}}};
|
||||
out.push_back(*transfer);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -53,6 +53,17 @@ PostgresHandoffLedger::admit(const network::TicketClaims& ticket, std::uint64_t
|
||||
return fail(committed.error());
|
||||
if (committed->state != persistence::TransferState::committed)
|
||||
return std::unexpected(HandoffLedgerError::refused); // Aborted meanwhile.
|
||||
// M10.8d: a retried ticket (e.g. after this region restarted and forgot it was used) is
|
||||
// honoured only while the probe is still as that commit left it: owned here at the next
|
||||
// generation and not frozen. Once it moved on, the ticket must not take control back.
|
||||
const auto record =
|
||||
persistence::open_probe(*db, key_, *lease, ticket.account.bytes, committed->x, committed->y,
|
||||
committed->z, PostgresProbeLedger::inventory, timeout_);
|
||||
if (!record)
|
||||
return fail(record.error());
|
||||
if (record->region != key_.region || record->generation != committed->generation + 1 ||
|
||||
record->transfer)
|
||||
return std::unexpected(HandoffLedgerError::refused);
|
||||
const auto control = persistence::handoff_account_control(
|
||||
*db, key_, *lease, ticket.account.bytes, ticket.transfer, session, duration_, timeout_);
|
||||
if (!control) {
|
||||
@@ -169,4 +180,21 @@ std::expected<std::vector<IncomingProbe>, ProbeLedgerError> PostgresProbeLedger:
|
||||
out.push_back({t.transfer, t.x, t.y});
|
||||
return out;
|
||||
}
|
||||
|
||||
std::expected<std::vector<network::TransferId>, ProbeLedgerError>
|
||||
PostgresProbeLedger::overdue(std::chrono::seconds age) {
|
||||
const auto lease = lease_();
|
||||
if (!lease)
|
||||
return std::unexpected(ProbeLedgerError::fenced);
|
||||
auto* db = db_->get();
|
||||
if (!db)
|
||||
return std::unexpected(ProbeLedgerError::unavailable);
|
||||
auto r = persistence::overdue_probe_transfers(*db, key_, *lease, age, timeout_);
|
||||
if (!r) {
|
||||
if (r.error().kind == ProbeStoreError::store || r.error().kind == ProbeStoreError::unknown)
|
||||
db_->broken();
|
||||
return std::unexpected(probe_error(r.error()));
|
||||
}
|
||||
return std::vector<network::TransferId>(r->begin(), r->end());
|
||||
}
|
||||
} // namespace evo::server
|
||||
|
||||
@@ -106,7 +106,7 @@ struct RegionSessions::Impl {
|
||||
std::optional<network::motion::Area> own_area; // This region's part of the marker area.
|
||||
std::int64_t spawn_x{}, spawn_y{}; // Where new probes start.
|
||||
std::vector<Outgoing> outgoing; // Handed off, not yet known committed.
|
||||
Clock::time_point next_ghosts{};
|
||||
Clock::time_point next_ghosts{}, next_overdue{};
|
||||
std::optional<network::SessionInterest> default_interest;
|
||||
HostFacts facts;
|
||||
std::vector<Connection> connections;
|
||||
@@ -471,6 +471,20 @@ struct RegionSessions::Impl {
|
||||
return config.probes->abort(o.transfer).has_value();
|
||||
});
|
||||
}
|
||||
// Recovery (M10.8d): the in-memory deadlines die with the process, so outgoing transfers
|
||||
// still open well after any ticket for them expired are aborted from the store's list.
|
||||
// Abort is serialized with the destination's commit: a committed one stays committed.
|
||||
void abort_overdue(Clock::time_point now) {
|
||||
if (now < next_overdue)
|
||||
return;
|
||||
next_overdue = now + config.overdue_poll;
|
||||
const auto overdue =
|
||||
config.probes->overdue(std::chrono::seconds{config.ticket_lifetime + 10});
|
||||
if (!overdue)
|
||||
return; // Retried next poll.
|
||||
for (const auto& t : *overdue)
|
||||
(void)config.probes->abort(t).has_value();
|
||||
}
|
||||
void refresh_ghosts(Clock::time_point now) {
|
||||
if (now < next_ghosts)
|
||||
return;
|
||||
@@ -549,7 +563,8 @@ RegionSessions::create(RegionSessionsConfig config) {
|
||||
config.ticket_lifetime == 0 || config.ticket_lifetime > network::max_ticket_lifetime ||
|
||||
config.transport.limits.max_connections < config.control.max_sessions ||
|
||||
config.hello_deadline.count() <= 0 || config.close_flush.count() <= 0 ||
|
||||
config.renew_interval.count() <= 0 || config.safety_margin.count() <= 0)
|
||||
config.renew_interval.count() <= 0 || config.safety_margin.count() <= 0 ||
|
||||
config.overdue_poll.count() <= 0)
|
||||
return std::unexpected(RegionSessionsError::configuration);
|
||||
auto control = network::SessionControl::create(config.control);
|
||||
if (!control)
|
||||
@@ -609,6 +624,7 @@ void RegionSessions::poll(const HostFacts& facts, std::uint64_t now_seconds,
|
||||
m.crossings(now_seconds, now);
|
||||
if (m.config.probes) {
|
||||
m.expire_outgoing(now_seconds);
|
||||
m.abort_overdue(now);
|
||||
m.refresh_ghosts(now);
|
||||
}
|
||||
}
|
||||
|
||||
+11
@@ -37,6 +37,8 @@ struct WorldDb {
|
||||
};
|
||||
std::map<AccountId, Probe> probes;
|
||||
std::map<TransferId, Transfer> transfers;
|
||||
bool all_overdue{}; // overdue() reports every open transfer (a restarted source's view).
|
||||
std::uint64_t overdue_reads{};
|
||||
};
|
||||
struct RegionDb final : server::ProbeLedger, HandoffLedger, ControlLedger {
|
||||
RegionDb(WorldDb& w, std::uint64_t r) : world{&w}, region{r} {}
|
||||
@@ -85,6 +87,15 @@ struct RegionDb final : server::ProbeLedger, HandoffLedger, ControlLedger {
|
||||
out.push_back({id, t.x, t.y});
|
||||
return out;
|
||||
}
|
||||
std::expected<std::vector<TransferId>, server::ProbeLedgerError>
|
||||
overdue(std::chrono::seconds) override {
|
||||
++world->overdue_reads;
|
||||
std::vector<TransferId> out;
|
||||
for (const auto& [id, t] : world->transfers)
|
||||
if (world->all_overdue && t.source == region && t.state == WorldDb::State::prepared)
|
||||
out.push_back(id);
|
||||
return out;
|
||||
}
|
||||
std::expected<HandoffAdmission, HandoffLedgerError> admit(const TicketClaims& ticket,
|
||||
std::uint64_t) override {
|
||||
const auto found = world->transfers.find(ticket.transfer);
|
||||
|
||||
@@ -15,6 +15,7 @@ void run_region_session_tests();
|
||||
#endif
|
||||
#ifdef EVO_POSTGRES_M10
|
||||
void run_region_m10_session_tests();
|
||||
void run_region_handoff_process_tests();
|
||||
#endif
|
||||
void run_region_recovery_tests();
|
||||
void run_region_ops_tests();
|
||||
@@ -138,6 +139,7 @@ int main() {
|
||||
#endif
|
||||
#ifdef EVO_POSTGRES_M10
|
||||
run_region_m10_session_tests();
|
||||
run_region_handoff_process_tests();
|
||||
#endif
|
||||
run_region_recovery_tests();
|
||||
std::puts("region recovery integration passed");
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
#ifdef EVO_POSTGRES_SESSIONS
|
||||
#include "evo/server/postgres_handoff_ledger.hpp"
|
||||
#endif
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
@@ -425,6 +426,9 @@ void handoff_ledger(World& w) {
|
||||
const auto t2 = id(0x76, 2);
|
||||
CHECK(state_of(prepare_probe_transfer(db, w.b, w.lb, frank, 2, t2, 1, 3, 4, -7, 5s)) ==
|
||||
TransferState::prepared);
|
||||
// M10.8d: once the probe moved on (here frozen for the next transfer), a retry of the
|
||||
// committed transfer's ticket no longer admits (nor takes control).
|
||||
CHECK(ledger.admit(ticket, 47).error() == evo::network::HandoffLedgerError::refused);
|
||||
CHECK(state_of(abort_probe_transfer(db, w.b, w.lb, t2, 5s)) == TransferState::aborted);
|
||||
auto aborted = ticket;
|
||||
aborted.transfer = t2;
|
||||
@@ -437,6 +441,29 @@ void handoff_ledger(World& w) {
|
||||
invariants(*w.admin, w.probes);
|
||||
}
|
||||
#endif
|
||||
// Schema v14: a source's open outgoing transfers older than the given age, by the database
|
||||
// clock, for recovery aborts; never another region's, never decided ones.
|
||||
void overdue(World& w) {
|
||||
auto& db = *w.db;
|
||||
const auto gina = id(0x49);
|
||||
w.open(w.a, w.la, gina);
|
||||
++w.probes;
|
||||
const auto t = id(0x77, 1);
|
||||
CHECK(state_of(prepare_probe_transfer(db, w.a, w.la, gina, 1, t, 2, 5, 6, -7, 5s)) ==
|
||||
TransferState::prepared);
|
||||
const auto has = [&](RegionKey key, const LeaseGrant& lease, std::chrono::seconds age) {
|
||||
const auto found = overdue_probe_transfers(db, key, lease, age, 5s);
|
||||
CHECK(found.has_value());
|
||||
return std::ranges::find(*found, t) != found->end();
|
||||
};
|
||||
CHECK(!has(w.a, w.la, 60s));
|
||||
std::this_thread::sleep_for(1100ms);
|
||||
CHECK(has(w.a, w.la, 1s) && !has(w.b, w.lb, 1s));
|
||||
CHECK(failed(overdue_probe_transfers(db, w.a, w.la, 0s, 5s), ProbeStoreError::invalid));
|
||||
CHECK(state_of(abort_probe_transfer(db, w.a, w.la, t, 5s)) == TransferState::aborted);
|
||||
CHECK(!has(w.a, w.la, 1s));
|
||||
invariants(*w.admin, w.probes);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_probe_handoff_store_tests() {
|
||||
@@ -445,6 +472,7 @@ void run_probe_handoff_store_tests() {
|
||||
restarts(w);
|
||||
aborts_and_races(w);
|
||||
control_handoff(w);
|
||||
overdue(w);
|
||||
#ifdef EVO_POSTGRES_SESSIONS
|
||||
handoff_ledger(w);
|
||||
#endif
|
||||
|
||||
@@ -667,6 +667,7 @@ struct Region {
|
||||
area_of(neighbour_x0)});
|
||||
c.ticket_lifetime = 20;
|
||||
c.ghost_poll = std::chrono::milliseconds{20};
|
||||
c.overdue_poll = std::chrono::milliseconds{20};
|
||||
auto made = RegionSessions::create(std::move(c));
|
||||
CHECK(made.has_value());
|
||||
sessions = std::move(*made);
|
||||
@@ -846,6 +847,21 @@ void two_region_handoff(const Files& f) {
|
||||
CHECK(pump([&] { return late->state() == RegionSessionState::closed; }));
|
||||
CHECK(late->close_reason() == SessionCloseReason::expired);
|
||||
CHECK(b->state() == RegionSessionState::established && r1.db.releases > 0);
|
||||
// M10.8d recovery: a transfer prepared by an earlier process (unknown to this one's
|
||||
// in-memory deadlines) is aborted once the store reports it overdue; never before.
|
||||
const auto dave = account(24);
|
||||
TransferId orphan{};
|
||||
orphan.fill(0x6D);
|
||||
world.probes[dave] = {1, 1, 9 * m_um, 5 * m_um, orphan};
|
||||
world.transfers[orphan] = {dave, 1, 2, 1, 9 * m_um, 5 * m_um, WorldDb::State::prepared};
|
||||
const auto reads = world.overdue_reads;
|
||||
CHECK(pump([&] { return world.overdue_reads > reads + 2; }));
|
||||
CHECK(world.transfers.at(orphan).state == WorldDb::State::prepared);
|
||||
world.all_overdue = true;
|
||||
CHECK(pump([&] { return !world.probes.at(dave).transfer.has_value(); }));
|
||||
CHECK(world.transfers.at(orphan).state == WorldDb::State::aborted &&
|
||||
world.probes.at(dave).owner == 1);
|
||||
world.all_overdue = false;
|
||||
}
|
||||
|
||||
void durable_control(const Files& f) {
|
||||
|
||||
@@ -0,0 +1,519 @@
|
||||
// Real process tests for M10.8d (ADR-0096): two evo-server processes on PostgreSQL with
|
||||
// adjacent marker areas, handoff keys and each other as neighbour; in-process
|
||||
// RegionSessionClients follow offers and tickets. Regions 7/161 (A, cells 0..1) and 7/162
|
||||
// (B, cells 2..3). Covers a crossing and a crossing back, SIGKILL of the source after
|
||||
// prepare and of the destination after stage and after commit, duplicate and stale ticket
|
||||
// retries, a client that never arrives (aborted by the restarted source's recovery sweep)
|
||||
// and a destination that is down (aborted by the live source). After every scenario: one
|
||||
// owning region per probe, the test inventory conserved, and a record frozen exactly while
|
||||
// its one open transfer exists. Run by tools/ci/postgres-check.sh when the build has
|
||||
// EVO_BUILD_QUIC.
|
||||
#include "evo/client/region_session.hpp"
|
||||
#include "evo/network/dev_credentials.hpp"
|
||||
#include "evo/network/handoff_ticket.hpp"
|
||||
#include "evo/network/session_grant.hpp"
|
||||
#include "fixtures/server_process.hpp"
|
||||
#include <algorithm>
|
||||
#include <arpa/inet.h>
|
||||
#include <array>
|
||||
#include <chrono>
|
||||
#include <fcntl.h>
|
||||
#include <functional>
|
||||
#include <netinet/in.h>
|
||||
#include <optional>
|
||||
#include <source_location>
|
||||
#include <string>
|
||||
#include <sys/socket.h>
|
||||
#include <unistd.h>
|
||||
#include <vector>
|
||||
|
||||
void run_region_handoff_process_tests();
|
||||
|
||||
namespace {
|
||||
using namespace evo::test::process;
|
||||
using namespace evo;
|
||||
using namespace evo::network;
|
||||
using client::RegionSessionClient;
|
||||
using client::RegionSessionState;
|
||||
using evo::test::region_identity;
|
||||
void check(bool value, std::source_location at = std::source_location::current()) {
|
||||
if (!value)
|
||||
throw std::runtime_error("region handoff process test line " + std::to_string(at.line()));
|
||||
}
|
||||
#define CHECK(...) check(static_cast<bool>(__VA_ARGS__))
|
||||
template <typename T>
|
||||
const T& must(const std::optional<T>& o,
|
||||
std::source_location at = std::source_location::current()) {
|
||||
if (!o)
|
||||
throw std::runtime_error("region handoff process missing value line " +
|
||||
std::to_string(at.line()));
|
||||
return *o;
|
||||
}
|
||||
constexpr const char* name = "dev-region.evolution.test";
|
||||
constexpr std::uint64_t region_a = 161, region_b = 162;
|
||||
|
||||
void write_file(const fs::path& path, std::span<const std::uint8_t> bytes) {
|
||||
const int fd = ::open(path.c_str(), O_WRONLY | O_CREAT | O_TRUNC | O_CLOEXEC, 0600);
|
||||
expect(fd >= 0 &&
|
||||
::write(fd, bytes.data(), bytes.size()) == static_cast<ssize_t>(bytes.size()) &&
|
||||
::close(fd) == 0,
|
||||
"write file");
|
||||
}
|
||||
void write_text(const fs::path& path, const std::string& text) {
|
||||
write_file(path, {reinterpret_cast<const std::uint8_t*>(text.data()), text.size()});
|
||||
}
|
||||
AccountId account(std::uint8_t tag) {
|
||||
AccountId a;
|
||||
a.bytes.fill(tag);
|
||||
return a;
|
||||
}
|
||||
template <std::size_t N> std::string hex(const std::array<std::uint8_t, N>& bytes) {
|
||||
constexpr char digits[] = "0123456789abcdef";
|
||||
std::string out;
|
||||
for (const auto b : bytes) {
|
||||
out.push_back(digits[b >> 4U]);
|
||||
out.push_back(digits[b & 0xFU]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::uint64_t wall_seconds() {
|
||||
return static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(
|
||||
std::chrono::system_clock::now().time_since_epoch())
|
||||
.count());
|
||||
}
|
||||
// A UDP port that was free a moment ago (each server must know its neighbour's port).
|
||||
std::uint16_t free_port() {
|
||||
const int fd = ::socket(AF_INET, SOCK_DGRAM | SOCK_CLOEXEC, 0);
|
||||
expect(fd >= 0, "socket");
|
||||
sockaddr_in a{};
|
||||
a.sin_family = AF_INET;
|
||||
a.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
||||
socklen_t size = sizeof a;
|
||||
const bool ok = ::bind(fd, reinterpret_cast<const sockaddr*>(&a), sizeof a) == 0 &&
|
||||
::getsockname(fd, reinterpret_cast<sockaddr*>(&a), &size) == 0;
|
||||
::close(fd);
|
||||
expect(ok, "free port");
|
||||
return ntohs(a.sin_port);
|
||||
}
|
||||
// A 32-byte scalar that imports as a key, written with its public key.
|
||||
template <typename Key, std::size_t N>
|
||||
void make_key(const fs::path& scalar_file, const fs::path& public_file,
|
||||
const std::function<std::optional<Key>(std::span<const std::uint8_t>)>& import) {
|
||||
for (int attempt = 0; attempt < 8; ++attempt) {
|
||||
std::array<std::uint8_t, N> scalar{};
|
||||
expect(random_bytes(scalar).has_value(), "random");
|
||||
if (auto key = import(scalar)) {
|
||||
write_file(scalar_file, scalar);
|
||||
write_file(public_file, *key->public_key());
|
||||
return;
|
||||
}
|
||||
}
|
||||
expect(false, "key");
|
||||
}
|
||||
|
||||
// A genesis commit byte for an all-equal commit id (ids are unique across the suite): one
|
||||
// no other test names (later tests provision theirs by literal) and none earlier produced.
|
||||
std::uint8_t unused_genesis() {
|
||||
for (const unsigned b : {0x4BU, 0x4CU, 0x4DU, 0x8BU, 0x8CU, 0x8DU, 0x8EU}) {
|
||||
std::string hex_id;
|
||||
for (int i = 0; i < 16; ++i) {
|
||||
hex_id.push_back("0123456789abcdef"[b >> 4U]);
|
||||
hex_id.push_back("0123456789abcdef"[b & 0xFU]);
|
||||
}
|
||||
const std::array<std::string, 1> p{hex_id};
|
||||
const auto r = owner()->query(
|
||||
"SELECT count(*) FROM evolution.region_commits WHERE commit_id = $1::uuid", p, 5s);
|
||||
expect(r.has_value() && r->size() == 1, "genesis lookup");
|
||||
if ((*r)[0][0] == "0")
|
||||
return static_cast<std::uint8_t>(b);
|
||||
}
|
||||
expect(false, "genesis byte");
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Both regions: one CA and server certificate, one grant issuer, a handoff key each.
|
||||
struct Pair {
|
||||
ServerRegion a{provision(region_a, unused_genesis())}, b{provision(region_b, unused_genesis())};
|
||||
fs::path files{a.root / "pair"};
|
||||
std::string ca_pem, window;
|
||||
std::optional<GrantSigningKey> issuer;
|
||||
std::uint16_t port_a{free_port()}, port_b{free_port()};
|
||||
Pair() {
|
||||
fs::create_directory(files);
|
||||
const auto made = generate_dev_credentials(DeploymentEnvironment::dev, name, 1,
|
||||
std::chrono::system_clock::now());
|
||||
expect(made.has_value(), "credentials");
|
||||
ca_pem = made->authority_pem;
|
||||
write_text(files / "ca.pem", made->authority_pem);
|
||||
write_text(files / "server.pem", made->server_certificate_pem);
|
||||
write_text(files / "server.key", made->server_key_pem);
|
||||
make_key<GrantSigningKey, grant_private_key_bytes>(
|
||||
files / "issuer.key", files / "issuer.pub", [](std::span<const std::uint8_t> s) {
|
||||
auto k = GrantSigningKey::import(DeploymentEnvironment::dev, 1, s);
|
||||
return k ? std::optional{std::move(*k)} : std::nullopt;
|
||||
});
|
||||
std::array<std::uint8_t, grant_private_key_bytes> scalar{};
|
||||
const int fd = ::open((files / "issuer.key").c_str(), O_RDONLY | O_CLOEXEC);
|
||||
expect(fd >= 0 &&
|
||||
::read(fd, scalar.data(), scalar.size()) ==
|
||||
static_cast<ssize_t>(scalar.size()) &&
|
||||
::close(fd) == 0,
|
||||
"issuer key");
|
||||
auto key = GrantSigningKey::import(DeploymentEnvironment::dev, 1, scalar);
|
||||
expect(key.has_value(), "issuer import");
|
||||
issuer.emplace(std::move(*key));
|
||||
for (const auto r : {region_a, region_b})
|
||||
make_key<HandoffSigningKey, grant_private_key_bytes>(
|
||||
files / ("handoff-" + std::to_string(r) + ".key"),
|
||||
files / ("handoff-" + std::to_string(r) + ".pub"),
|
||||
[r](std::span<const std::uint8_t> s) {
|
||||
auto k = HandoffSigningKey::import(DeploymentEnvironment::dev,
|
||||
static_cast<std::uint32_t>(r), s);
|
||||
return k ? std::optional{std::move(*k)} : std::nullopt;
|
||||
});
|
||||
const auto now = wall_seconds();
|
||||
window = "test-" + std::to_string(now) + ":" + std::to_string(now + 1200);
|
||||
}
|
||||
[[nodiscard]] std::vector<std::string> args(bool at_a) const {
|
||||
const auto& own = at_a ? a : b;
|
||||
const auto other = at_a ? region_b : region_a;
|
||||
const std::string other_cells = at_a ? "2,0,3,1" : "0,0,1,1";
|
||||
return server_args(
|
||||
own, {"--synthetic-inputs",
|
||||
"development-v1",
|
||||
"--m10-listen",
|
||||
"127.0.0.1:" + std::to_string(at_a ? port_a : port_b),
|
||||
"--m10-cert",
|
||||
(files / "server.pem").string(),
|
||||
"--m10-key",
|
||||
(files / "server.key").string(),
|
||||
"--m10-issuer",
|
||||
(files / "issuer.pub").string(),
|
||||
"--m10-environment",
|
||||
"dev",
|
||||
"--session-window",
|
||||
window,
|
||||
"--m10-area",
|
||||
at_a ? "0,0,1,1" : "2,0,3,1",
|
||||
"--m10-handoff-key",
|
||||
(files / ("handoff-" + std::to_string(own.key.region) + ".key")).string(),
|
||||
"--m10-neighbour",
|
||||
std::to_string(other) + "=" + other_cells +
|
||||
"=127.0.0.1:" + std::to_string(at_a ? port_b : port_a) + "=" + name + "=" +
|
||||
(files / ("handoff-" + std::to_string(other) + ".pub")).string()});
|
||||
}
|
||||
[[nodiscard]] WireGrant grant(const AccountId& who, std::uint64_t serial,
|
||||
std::uint64_t region) const {
|
||||
GrantClaims c;
|
||||
c.grant_id = (region * 100'000) + serial;
|
||||
c.account = who;
|
||||
c.serial = serial;
|
||||
c.world = {7, region_identity.frame};
|
||||
c.region = region;
|
||||
c.issued_at = wall_seconds();
|
||||
c.expires_at = c.issued_at + 120;
|
||||
const auto g = must(issuer).sign(c);
|
||||
expect(g.has_value(), "grant");
|
||||
return *g;
|
||||
}
|
||||
};
|
||||
|
||||
// A running evo-server, its output drained while waiting.
|
||||
struct Server {
|
||||
std::unique_ptr<Child> child;
|
||||
explicit Server(const Pair& p, bool at_a) : child{std::make_unique<Child>(p.args(at_a))} {
|
||||
const bool started =
|
||||
child->wait_line("m10 listening ", 30s) && child->wait_line("ready 1", 30s) &&
|
||||
child->wait_line("m10 motion area ", 30s) && child->printed("m10 handoff neighbours=1");
|
||||
if (!started)
|
||||
for (const auto& line : child->lines())
|
||||
if (starts(line, "m10") || starts(line, "error") || starts(line, "usage"))
|
||||
std::printf("server: %s\n", line.c_str());
|
||||
CHECK(started);
|
||||
}
|
||||
void kill() {
|
||||
child->signal(SIGKILL);
|
||||
CHECK(child->wait_exit(30s) == 128 + SIGKILL);
|
||||
}
|
||||
};
|
||||
|
||||
struct Bench {
|
||||
Pair pair;
|
||||
std::unique_ptr<Server> a, b;
|
||||
std::vector<AccountId> accounts; // Every probe this test created.
|
||||
std::array<std::uint64_t, 256> serials{};
|
||||
|
||||
void pump() {
|
||||
for (auto* s : {a.get(), b.get()})
|
||||
if (s)
|
||||
s->child->pump(1ms);
|
||||
}
|
||||
// Polls clients and live servers until `done`, at most `limit`.
|
||||
bool until(const std::vector<RegionSessionClient*>& clients, const std::function<bool()>& done,
|
||||
std::chrono::milliseconds limit = 20s) {
|
||||
const auto end = Clock::now() + limit;
|
||||
while (Clock::now() < end) {
|
||||
for (auto* c : clients)
|
||||
c->poll(Clock::now());
|
||||
pump();
|
||||
if (done())
|
||||
return true;
|
||||
}
|
||||
return done();
|
||||
}
|
||||
std::unique_ptr<RegionSessionClient> dial(std::uint16_t port, const WireGrant& grant) {
|
||||
client::RegionSessionClientConfig c;
|
||||
c.transport.port = port;
|
||||
c.transport.server_name = name;
|
||||
c.transport.authority_pem = pair.ca_pem;
|
||||
c.transport.authority_pem_file = (pair.files / "ca.pem").string();
|
||||
c.grant = grant;
|
||||
auto made = RegionSessionClient::connect(std::move(c), Clock::now());
|
||||
CHECK(made.has_value());
|
||||
return std::move(*made);
|
||||
}
|
||||
// A grant login (next serial for the account) at a region; waits for the outcome.
|
||||
std::unique_ptr<RegionSessionClient> login(const AccountId& who, std::uint64_t region) {
|
||||
if (std::ranges::find(accounts, who) == accounts.end())
|
||||
accounts.push_back(who);
|
||||
auto c = dial(region == region_a ? pair.port_a : pair.port_b,
|
||||
pair.grant(who, ++serials.at(who.bytes[0]), region));
|
||||
settle(*c);
|
||||
return c;
|
||||
}
|
||||
// A ticket login at the offer's destination; waits for the outcome.
|
||||
std::unique_ptr<RegionSessionClient> follow(const HandoffOffer& offer) {
|
||||
const auto& to = offer.destination;
|
||||
CHECK(to.server_name == name);
|
||||
const auto port = to.region == region_a ? pair.port_a : pair.port_b;
|
||||
CHECK(to.endpoint == "127.0.0.1:" + std::to_string(port));
|
||||
auto c = dial(port, offer.ticket);
|
||||
settle(*c);
|
||||
return c;
|
||||
}
|
||||
void settle(RegionSessionClient& c) {
|
||||
CHECK(until({&c}, [&] {
|
||||
return c.state() == RegionSessionState::established ||
|
||||
c.state() == RegionSessionState::closed;
|
||||
}));
|
||||
}
|
||||
// Walks the client's probe east (+1) or west (-1) until it is handed off; the offer.
|
||||
HandoffOffer cross(RegionSessionClient& c, int direction) {
|
||||
CHECK(c.state() == RegionSessionState::established);
|
||||
bool moving = false;
|
||||
std::optional<HandoffOffer> offer;
|
||||
CHECK(until({&c}, [&] {
|
||||
for (const auto& r : c.take_intention_receipts())
|
||||
moving = moving || r.status == AdmitStatus::admitted;
|
||||
if (!moving && c.state() == RegionSessionState::established)
|
||||
static_cast<void>(c.send_intention(c.server_tick() + 4,
|
||||
static_cast<std::int16_t>(direction * 1000), 0));
|
||||
if (!offer)
|
||||
offer = c.take_handoff();
|
||||
return offer.has_value() && c.state() == RegionSessionState::closed;
|
||||
}));
|
||||
CHECK(c.close_reason() == SessionCloseReason::handed_off);
|
||||
return must(offer);
|
||||
}
|
||||
|
||||
// SQL views of world 7.
|
||||
static persistence::PgRows rows(const std::string& sql,
|
||||
const std::vector<std::string>& params) {
|
||||
auto r = owner()->query(sql, params, 5s);
|
||||
CHECK(r.has_value());
|
||||
return *r;
|
||||
}
|
||||
static std::string one(const std::string& sql, const std::vector<std::string>& params) {
|
||||
const auto r = rows(sql, params);
|
||||
CHECK(r.size() == 1 && r[0].size() == 1);
|
||||
return r[0][0].value_or("");
|
||||
}
|
||||
// "owner:generation:frozen" of the account's probe.
|
||||
static std::string probe(const AccountId& who) {
|
||||
return one("SELECT region_id || ':' || generation || ':' || (transfer_id IS NOT NULL) "
|
||||
"FROM evolution.probe_records WHERE world_id = 7 AND account_id = $1::uuid",
|
||||
{hex(who.bytes)});
|
||||
}
|
||||
static std::string transfer(const TransferId& t) {
|
||||
return one("SELECT state FROM evolution.probe_transfers WHERE world_id = 7 AND "
|
||||
"transfer_id = $1::uuid",
|
||||
{hex(t)});
|
||||
}
|
||||
static std::uint64_t control_generation(const AccountId& who) {
|
||||
return std::stoull(one("SELECT generation FROM evolution.account_controls WHERE "
|
||||
"world_id = 7 AND account_id = $1::uuid",
|
||||
{hex(who.bytes)}));
|
||||
}
|
||||
// After every scenario: a single owning region (A or B) per probe, the inventory
|
||||
// conserved, at most one open transfer per probe and the record frozen exactly under it.
|
||||
void invariants() const {
|
||||
std::string list;
|
||||
for (const auto& who : accounts)
|
||||
list += (list.empty() ? "'" : ",'") + hex(who.bytes) + "'::uuid";
|
||||
const auto where = "r.world_id = 7 AND r.account_id IN (" + list + ")";
|
||||
CHECK(one("SELECT count(*) || ':' || sum(r.inventory) FROM evolution.probe_records r "
|
||||
"WHERE " +
|
||||
where + " AND r.region_id IN (161, 162)",
|
||||
{}) ==
|
||||
std::to_string(accounts.size()) + ":" + std::to_string(accounts.size() * 1000));
|
||||
CHECK(one("SELECT count(*) FROM evolution.probe_records r WHERE " + where +
|
||||
" AND r.transfer_id IS DISTINCT FROM (SELECT t.transfer_id FROM "
|
||||
"evolution.probe_transfers t WHERE t.world_id = 7 AND t.account_id = "
|
||||
"r.account_id AND t.state IN ('prepared', 'staged'))",
|
||||
{}) == "0");
|
||||
CHECK(one("SELECT count(*) FROM evolution.probe_transfers t WHERE t.world_id = 7 AND "
|
||||
"t.account_id IN (" +
|
||||
list + ") AND t.inventory <> 1000",
|
||||
{}) == "0");
|
||||
}
|
||||
// The region holding the account's durable control.
|
||||
static std::string control_holder(const AccountId& who) {
|
||||
return one("SELECT coalesce(region_id::text, '') FROM evolution.account_controls "
|
||||
"WHERE world_id = 7 AND account_id = $1::uuid",
|
||||
{hex(who.bytes)});
|
||||
}
|
||||
void kill(bool at_a) {
|
||||
auto& s = at_a ? a : b;
|
||||
s->kill();
|
||||
s.reset();
|
||||
}
|
||||
void restart(bool at_a) {
|
||||
const auto started = Clock::now();
|
||||
(at_a ? a : b) = std::make_unique<Server>(pair, at_a); // Waits out the old lease.
|
||||
std::printf("handoff process restart %c %lld ms\n", at_a ? 'A' : 'B',
|
||||
static_cast<long long>(std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
Clock::now() - started)
|
||||
.count()));
|
||||
}
|
||||
};
|
||||
|
||||
void handoffs() {
|
||||
Bench w;
|
||||
w.restart(true);
|
||||
w.restart(false);
|
||||
|
||||
// 1. A crossing and a crossing back (generation 1 -> 2 -> 3), then a stale grant login
|
||||
// at the old region is redirected.
|
||||
const auto alice = account(0xA1);
|
||||
auto a1 = w.login(alice, region_a);
|
||||
CHECK(a1->state() == RegionSessionState::established && w.probe(alice) == "161:1:false");
|
||||
const auto east = w.cross(*a1, 1);
|
||||
CHECK(east.destination.region == region_b);
|
||||
auto a2 = w.follow(east);
|
||||
CHECK(a2->state() == RegionSessionState::established && w.probe(alice) == "162:2:false" &&
|
||||
w.transfer(east.transfer) == "committed");
|
||||
const auto west = w.cross(*a2, -1);
|
||||
CHECK(west.destination.region == region_a);
|
||||
auto a3 = w.follow(west);
|
||||
CHECK(a3->state() == RegionSessionState::established && w.probe(alice) == "161:3:false");
|
||||
auto stale = w.login(alice, region_b);
|
||||
CHECK(stale->close_reason() == SessionCloseReason::moved && stale->redirect() &&
|
||||
must(stale->redirect()).region == region_a);
|
||||
w.invariants();
|
||||
|
||||
// 2. Destination SIGKILL after stage: the restarted destination stages again under its
|
||||
// new epoch and commits when the ticket arrives. Then the probe moves on, and a replay of
|
||||
// that ticket at a destination process that never saw it is refused durably.
|
||||
const auto bob = account(0xB2);
|
||||
auto b1 = w.login(bob, region_a);
|
||||
const auto b_east = w.cross(*b1, 1);
|
||||
const auto ticket_issued = wall_seconds();
|
||||
CHECK(w.transfer(b_east.transfer) == "prepared" && w.probe(bob) == "161:1:true");
|
||||
CHECK(w.one("SELECT transfer_state FROM evolution.stage_probe_transfer(7, 162, "
|
||||
"(SELECT holder FROM evolution.region_leases WHERE world_id = 7 AND "
|
||||
"region_id = 162), (SELECT epoch FROM evolution.region_leases WHERE "
|
||||
"world_id = 7 AND region_id = 162), $1::uuid)",
|
||||
{hex(b_east.transfer)}) == "staged"); // As B's process would, then B dies.
|
||||
w.kill(false);
|
||||
w.invariants();
|
||||
w.restart(false);
|
||||
auto b2 = w.follow(b_east);
|
||||
CHECK(b2->state() == RegionSessionState::established && w.probe(bob) == "162:2:false" &&
|
||||
w.transfer(b_east.transfer) == "committed");
|
||||
const auto b_west = w.cross(*b2, -1);
|
||||
auto b3 = w.follow(b_west);
|
||||
CHECK(b3->state() == RegionSessionState::established && w.probe(bob) == "161:3:false");
|
||||
|
||||
// 3. Destination SIGKILL after commit (the client was admitted): a retry of the same
|
||||
// ticket at the restarted destination converges (same commit, control moves again).
|
||||
const auto carol = account(0xC3);
|
||||
auto c1 = w.login(carol, region_a);
|
||||
const auto c_east = w.cross(*c1, 1);
|
||||
auto c2 = w.follow(c_east);
|
||||
CHECK(c2->state() == RegionSessionState::established && w.probe(carol) == "162:2:false");
|
||||
const auto control_before = w.control_generation(carol);
|
||||
w.kill(false);
|
||||
w.invariants();
|
||||
w.restart(false);
|
||||
// Bob's first ticket, replayed after his probe moved on (and before it expires).
|
||||
auto replay = w.follow(b_east);
|
||||
std::printf("handoff process replay at %llu s of the ticket's lifetime\n",
|
||||
static_cast<unsigned long long>(wall_seconds() - ticket_issued));
|
||||
CHECK(replay->state() == RegionSessionState::closed &&
|
||||
replay->close_reason() == SessionCloseReason::unauthenticated &&
|
||||
w.probe(bob) == "161:3:false" && w.control_holder(bob) == "161");
|
||||
auto c3 = w.follow(c_east);
|
||||
CHECK(c3->state() == RegionSessionState::established && w.probe(carol) == "162:2:false" &&
|
||||
w.transfer(c_east.transfer) == "committed" &&
|
||||
w.control_generation(carol) > control_before);
|
||||
// A duplicate in the same process is refused locally (the ticket is used).
|
||||
auto again = w.follow(c_east);
|
||||
CHECK(again->close_reason() == SessionCloseReason::stale_grant &&
|
||||
w.control_holder(carol) == "162");
|
||||
w.invariants();
|
||||
|
||||
// 4. Source SIGKILL after prepare: Dave's ticket still commits at the destination; Erin
|
||||
// never arrives, and her frozen probe is aborted by the restarted source's recovery
|
||||
// sweep (the dead process's in-memory deadline is gone).
|
||||
const auto dave = account(0xD4), erin = account(0xE5);
|
||||
auto d1 = w.login(dave, region_a);
|
||||
auto e1 = w.login(erin, region_a);
|
||||
const auto d_east = w.cross(*d1, 1);
|
||||
const auto e_east = w.cross(*e1, 1);
|
||||
w.kill(true);
|
||||
auto d2 = w.follow(d_east);
|
||||
CHECK(d2->state() == RegionSessionState::established && w.probe(dave) == "162:2:false");
|
||||
w.invariants();
|
||||
|
||||
// 5. A destination that is down: Frank's ticket cannot be used; the live source aborts
|
||||
// after the ticket's lifetime. Erin's wait (above) overlaps.
|
||||
w.kill(false);
|
||||
w.restart(true);
|
||||
auto e2 = w.login(erin, region_a); // Frozen under the open transfer: retry later.
|
||||
CHECK(e2->close_reason() == SessionCloseReason::capacity && w.probe(erin) == "161:1:true");
|
||||
const auto frank = account(0xF6);
|
||||
auto f1 = w.login(frank, region_a);
|
||||
const auto f_east = w.cross(*f1, 1);
|
||||
auto f2 = w.dial(w.pair.port_b, f_east.ticket);
|
||||
w.settle(*f2);
|
||||
CHECK(f2->state() == RegionSessionState::closed && !f2->welcome());
|
||||
w.invariants();
|
||||
CHECK(w.until(
|
||||
{},
|
||||
[&] {
|
||||
return w.transfer(e_east.transfer) == "aborted" &&
|
||||
w.transfer(f_east.transfer) == "aborted";
|
||||
},
|
||||
90s));
|
||||
CHECK(w.probe(erin) == "161:1:false" && w.probe(frank) == "161:1:false");
|
||||
auto e3 = w.login(erin, region_a);
|
||||
auto f3 = w.login(frank, region_a);
|
||||
CHECK(e3->state() == RegionSessionState::established &&
|
||||
f3->state() == RegionSessionState::established);
|
||||
w.restart(false);
|
||||
auto late = w.follow(f_east); // Expired by now, and its transfer aborted.
|
||||
CHECK(late->close_reason() == SessionCloseReason::expired && w.probe(frank) == "161:1:false");
|
||||
w.invariants();
|
||||
for (auto* s : {w.a.get(), w.b.get()}) {
|
||||
s->child->signal(SIGTERM);
|
||||
CHECK(s->child->wait_exit(30s) == 0);
|
||||
}
|
||||
// Output is flushed at exit. This B process only refused (expired).
|
||||
CHECK(w.a->child->printed("m10 opened ") && w.b->child->printed("m10 refused "));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_handoff_process_tests() {
|
||||
handoffs();
|
||||
std::puts("region handoff process integration passed");
|
||||
}
|
||||
@@ -179,6 +179,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
" -v migration_sha_11=$(s 0011_region_actions.sql)"
|
||||
" -v migration_sha_12=$(s 0012_probe_handoff.sql)"
|
||||
" -v migration_sha_13=$(s 0013_probe_incoming.sql)"
|
||||
" -v migration_sha_14=$(s 0014_probe_overdue.sql)"
|
||||
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
|
||||
" > /dev/null 2>&1") != 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
|
||||
@@ -194,7 +195,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
CHECK(shell("PGDATABASE=" + database +
|
||||
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
|
||||
"tools/migrate-postgres.sh\" > /dev/null") == 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11,12,13");
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11,12,13,14");
|
||||
CHECK(renew_region_lease(*runtime, r.key, *lease, 20000ms, 5s));
|
||||
confirm(publish_next(20)); // 41..60, now v8.
|
||||
CHECK(head.head.end_tick == 60);
|
||||
|
||||
@@ -61,7 +61,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
|
||||
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14
|
||||
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
|
||||
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
|
||||
v2_sha="${v2_sha%% *}"
|
||||
@@ -82,7 +82,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 13:1,2,3,4,5,6,7,8,9,10,11,12,13
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 14:1,2,3,4,5,6,7,8,9,10,11,12,13,14
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
|
||||
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
|
||||
@@ -136,7 +136,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
|
||||
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
|
||||
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
|
||||
@@ -181,9 +181,11 @@ expect_audit_failure 'GRANT SELECT ON evolution.account_controls TO evo_runtime_
|
||||
expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) FROM evo_runtime_test' 'GRANT EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) TO evo_runtime_test'
|
||||
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
|
||||
# Unknown future versions and checksum drift must not be repaired silently.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=14 WHERE version=13'
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=15 WHERE version=14'
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=14 WHERE version=15"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=13 WHERE version=14"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=12"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# exported). Real evo-server processes on world 500:
|
||||
# 1. Failed migration: a migration that takes the migration lock, alters a history
|
||||
# table, holds its locks for 3 s and then fails runs while build b1 serves.
|
||||
# Nothing it did remains (schema still v13, no new column); the host keeps its
|
||||
# Nothing it did remains (schema still v14, no new column); the host keeps its
|
||||
# authority, and every acknowledgement stays durable.
|
||||
# 2. Barrier (ADR-0068 operator order): build b2 is retained, and a b2 host is started
|
||||
# while b1 still holds the lease (it waits, never ready). upgrade_barrier(b1) fences
|
||||
@@ -61,7 +61,7 @@ SQL
|
||||
[ "$code" != 0 ] && grep -q 'division by zero' "$work/migration.log" || fail "the rehearsal migration did not fail"
|
||||
[ "$(q -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'region_commits' AND column_name = 'rehearsal_column'")" = 0 ] ||
|
||||
fail "the failed migration left a column"
|
||||
[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 13 ] || fail "schema version moved"
|
||||
[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 14 ] || fail "schema version moved"
|
||||
wait_ack "$work/b1.log" $((before + 40)) || fail "b1 did not resume after the failed migration"
|
||||
kill -0 "$b1_pid" 2> /dev/null || fail "b1 exited during the failed migration"
|
||||
bash "$root/tools/migrate-postgres.sh" > /dev/null || fail "migrate after the failure"
|
||||
|
||||
@@ -38,11 +38,14 @@ digest_12="$(sha256sum "$root/migrations/0012_probe_handoff.sql")"
|
||||
digest_12="${digest_12%% *}"
|
||||
digest_13="$(sha256sum "$root/migrations/0013_probe_incoming.sql")"
|
||||
digest_13="${digest_13%% *}"
|
||||
digest_14="$(sha256sum "$root/migrations/0014_probe_overdue.sql")"
|
||||
digest_14="${digest_14%% *}"
|
||||
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
|
||||
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
|
||||
--set="migration_sha_6=$digest_6" --set="migration_sha_7=$digest_7" \
|
||||
--set="migration_sha_8=$digest_8" --set="migration_sha_9=$digest_9" \
|
||||
--set="migration_sha_10=$digest_10" --set="migration_sha_11=$digest_11" --set="migration_sha_12=$digest_12" --set="migration_sha_13=$digest_13" \
|
||||
--set="migration_sha_14=$digest_14" \
|
||||
--set="runtime_role=$runtime_role" \
|
||||
--file="$root/migrations/apply.sql"
|
||||
|
||||
Reference in New Issue
Block a user