Serialize build retirement and clone classification, bound ops accepts
Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the barrier takes a database-wide upgrade lock exclusively, while acquisitions, lease checks, provisioning and the retired-build trigger take it shared first, so a queued acquisition can no longer regain authority for a just-retired build and concurrent barriers cannot retire the last current build. Provisioning takes a per-world lock before classifying a world as original or a clone of one source. The operations listener makes at most four accept attempts per serve(). Race regressions prove each contender is blocked before commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -954,3 +954,8 @@ lease:
|
||||
run. Host-loss durability is M8.
|
||||
|
||||
Next: M8.1, the host inventory and budgets.
|
||||
|
||||
Review fixes (2026-09-29): schema v9 serializes build retirement with acquisition,
|
||||
leases and provisioning under one lock order. It also serializes clone-world
|
||||
classification per world. The operations listener's accept work per loop pass is
|
||||
now bounded. Regression tests and five mutations cover each fix.
|
||||
|
||||
@@ -99,3 +99,15 @@ zlib, CA certificates):
|
||||
- A later gateway must honor readiness and the lease epoch before routing.
|
||||
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
|
||||
(M8).
|
||||
|
||||
## Amendment: bounded accept work (2026-09-29 review)
|
||||
`serve()` used to accept, and close excess connections, until the backlog was
|
||||
empty. A replenished stream of connections could therefore hold the host loop
|
||||
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
|
||||
attempts, refused ones included. Pending connections wait in the 16-entry listen
|
||||
backlog, and deadlines are judged after the accept work.
|
||||
|
||||
Tests check the per-call budget, including at the connection cap, and that an
|
||||
accepted probe is still answered under replenished pressure. A real server keeps
|
||||
its lease and readiness while a client opens and closes connections continuously
|
||||
through 60 ticks.
|
||||
|
||||
@@ -109,3 +109,18 @@ alone does not prove retention; the audit does.
|
||||
defeats isolation.
|
||||
- **Recording provenance only in a table.** The chain would not commit to its
|
||||
origin, and cross-database clones could not verify it.
|
||||
|
||||
## Amendment: serialized world classification (schema v9, 2026-09-29 review)
|
||||
The rule "original, or a clone of exactly one source" was enforced by unlocked
|
||||
existence checks. Two concurrent clones of different sources into different
|
||||
regions of one world could therefore both commit (reproduced by review). Schema
|
||||
v9 fixes this:
|
||||
- Both provisioning functions take a per-world advisory lock, keyed by a hash of
|
||||
the world ID, after the shared upgrade lock (ADR-0068 amendment).
|
||||
- They then check the classification. Because the checks run after the lock wait,
|
||||
they see every committed provisioning of that world.
|
||||
- Unrelated worlds provision in parallel. A hash collision only adds waiting.
|
||||
|
||||
Regression tests race different-source clones, and an original region against a
|
||||
clone, into one world, and expect exactly one to succeed. Same-source clones into
|
||||
different regions both succeed.
|
||||
|
||||
@@ -94,3 +94,28 @@ The results are in engineering/DURABILITY_QUALIFICATION.md.
|
||||
mid-upgrade before the new build was deployed.
|
||||
- **In-place rules migration of a live region.** It would change physical laws
|
||||
without the clone-and-compare approval that the vision requires.
|
||||
|
||||
## Amendment: serialized retirement (schema v9, 2026-09-29 review)
|
||||
In v8, acquisition checked eligibility before locking the lease row, without
|
||||
serializing against the barrier. A request queued behind a barrier could
|
||||
therefore acquire for the build just retired, then keep renewing (reproduced by
|
||||
review). Schema v9 fixes this with one lock order: the upgrade lock (advisory,
|
||||
database-wide), then the world classification lock, then the lease row, then
|
||||
head and commit rows.
|
||||
- `upgrade_barrier` takes the upgrade lock exclusively before anything else. This
|
||||
also serializes barriers, so two operators cannot each retire what they believe
|
||||
is a non-last build.
|
||||
- Both acquisition forms, `check_region_lease` (and with it publication, renewal
|
||||
and release), provisioning and the retired-build trigger take the upgrade lock
|
||||
shared first. Their eligibility checks therefore run after any concurrent
|
||||
barrier commits.
|
||||
|
||||
The lock order is the same everywhere, so publication cannot deadlock against the
|
||||
barrier. Regression tests hold a barrier open, prove the contender is blocked,
|
||||
commit, and check that a queued acquisition of the retired build is refused (both
|
||||
forms) and that the last current build cannot be retired by racing barriers.
|
||||
|
||||
A runtime login can still take PostgreSQL advisory locks directly, and so could
|
||||
delay barriers or acquisitions. It cannot gain authority that way. An owner can
|
||||
terminate such sessions. Removing that capability needs a pooler or role policy
|
||||
(M8).
|
||||
|
||||
@@ -2534,3 +2534,49 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
The migration test caught this one after the host-test correction.
|
||||
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
|
||||
host or storage loss (M8).
|
||||
|
||||
## M7.6–M7.9 review fixes (schema v9) - 2026-09-29
|
||||
|
||||
Review `.cache/reviews/M7_6-M7_9_REVIEW.md` found three issues. All three are
|
||||
fixed:
|
||||
- **[P1] Build retirement could race acquisition.** Schema v9 applies one
|
||||
upgrade-lock order across the barrier, acquisitions, lease checks, provisioning
|
||||
and the retired-build trigger.
|
||||
- **[P2] Clone-world classification could race.** Provisioning now takes a
|
||||
per-world lock and checks classification after it.
|
||||
- **[P2] The operations listener accepted without bound.** Each `serve()` now
|
||||
makes at most 4 accept attempts.
|
||||
|
||||
Evidence:
|
||||
- WSL Linux `checks` set (Debug, Release, analysis, ASan/UBSan): CTest 59/59
|
||||
each. The format check passed.
|
||||
- PostgreSQL ASan/UBSan: full suite passed. Analysis and Release were rerun green
|
||||
after fixing two clang-tidy findings in the new tests. Upgrade paths now end at
|
||||
1..9.
|
||||
- Race tests (database `evo_race`): each holds a transaction open, confirms the
|
||||
contender is blocked on a lock via `pg_stat_activity`, then commits.
|
||||
- A build-less acquisition queued behind a barrier is refused (EVL06).
|
||||
- A build-aware acquisition of the build being retired is refused, and the
|
||||
region stays unheld.
|
||||
- The next build acquires.
|
||||
- Racing barriers for the last two current builds: one returns 22023 and one
|
||||
build stays current.
|
||||
- Different-source clones into one world, and an original region racing a
|
||||
clone: the second is refused (22023).
|
||||
- Same-source clones into different regions both succeed.
|
||||
- The final origins are exactly `2/1<10, 4/1<10, 4/2<10`.
|
||||
- Accept budget:
|
||||
- With 16 queued connections, successive `serve()` calls give 4 open, then 8
|
||||
open with 0 refused, then 8 open with 4 refused.
|
||||
- Refusals never exceed 4 per call under replenished connections.
|
||||
- The first accepted probe got 200.
|
||||
- A real server acknowledged 60 ticks under continuous connect/close churn
|
||||
without losing readiness.
|
||||
- Mutations, each reverting one fix (restored and rebuilt). Each was caught:
|
||||
- an unlocked build-aware acquire;
|
||||
- an unlocked legacy acquire;
|
||||
- an unlocked barrier;
|
||||
- no per-world lock;
|
||||
- an accept budget of 1000.
|
||||
- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s.
|
||||
- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files.
|
||||
|
||||
@@ -164,6 +164,13 @@ To upgrade the build:
|
||||
returns the number of leases fenced, and refuses to retire the last current
|
||||
build.
|
||||
|
||||
Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The
|
||||
barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions,
|
||||
lease checks, provisioning and the retired-build trigger take it shared, and
|
||||
provisioning also takes a per-world lock (71420003). A barrier therefore briefly
|
||||
waits for in-flight commits and blocks new ones. Keep other advisory locks away
|
||||
from these two keys.
|
||||
|
||||
Old images then quarantine (exit 4) instead of serving. Rules and format changes
|
||||
are refused explicitly; there is no converter.
|
||||
|
||||
|
||||
@@ -27,6 +27,9 @@ enum class OpsError : std::uint8_t { address, socket };
|
||||
class OpsServer {
|
||||
public:
|
||||
static constexpr std::size_t max_connections = 8;
|
||||
// Accept attempts per serve(), refused ones included: sustained connection
|
||||
// pressure cannot hold the host loop; the rest wait in the listen backlog.
|
||||
static constexpr std::size_t max_accepts_per_serve = 4;
|
||||
static constexpr std::size_t max_request_bytes = 4096;
|
||||
static constexpr std::size_t max_response_bytes = 16384;
|
||||
static constexpr std::chrono::milliseconds connection_deadline{1000};
|
||||
@@ -42,13 +45,16 @@ class OpsServer {
|
||||
[[nodiscard]] std::uint16_t port() const noexcept {
|
||||
return port_;
|
||||
}
|
||||
// Accepts, reads, answers and expires connections without waiting. Connections
|
||||
// beyond the bound are closed at once; slow or oversized ones are answered or
|
||||
// dropped without delaying others.
|
||||
// Accepts at most max_accepts_per_serve connections, then reads, answers and
|
||||
// expires connections without waiting. Connections beyond the bound are closed
|
||||
// at once; slow or oversized ones are answered or dropped without delaying others.
|
||||
void serve(const OpsHandler& handler);
|
||||
[[nodiscard]] std::uint64_t refused() const noexcept {
|
||||
return refused_;
|
||||
}
|
||||
[[nodiscard]] std::size_t open_connections() const noexcept {
|
||||
return connections_.size();
|
||||
}
|
||||
|
||||
private:
|
||||
struct Connection {
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
-- M7.9 review fixes (ADR-0067/0068): one locking protocol for authority, build
|
||||
-- retirement and world classification. Replaces function bodies only.
|
||||
--
|
||||
-- Lock order, everywhere: upgrade lock (advisory 71420002) -> world classification
|
||||
-- lock (advisory 71420003, hashtext(world)) -> lease row -> head/commit rows.
|
||||
-- upgrade_barrier takes the upgrade lock exclusively before anything else; every
|
||||
-- acquisition, lease check (so publication, renewal and release), provisioning and
|
||||
-- the retired-build trigger take it shared first. A retirement therefore cannot
|
||||
-- commit between an acquisition's eligibility check and its grant, a commit naming
|
||||
-- a build cannot straddle its retirement, and concurrent barriers serialize. The
|
||||
-- per-world lock serializes original/clone classification of one target world
|
||||
-- while unrelated worlds provision in parallel. Callers use READ COMMITTED
|
||||
-- statements (as the adapter does), so each check after a wait sees committed rows.
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.check_region_lease(p_world evolution.u64,
|
||||
p_region evolution.u64, p_holder uuid, p_epoch evolution.u64)
|
||||
RETURNS timestamptz LANGUAGE plpgsql SECURITY INVOKER
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
l evolution.region_leases%ROWTYPE;
|
||||
t timestamptz;
|
||||
BEGIN
|
||||
IF p_world IS NULL OR p_region IS NULL OR p_holder IS NULL OR p_epoch IS NULL THEN
|
||||
RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
SELECT * INTO l FROM evolution.region_leases r
|
||||
WHERE r.world_id = p_world AND r.region_id = p_region FOR UPDATE;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'region not provisioned' USING ERRCODE = 'EVL04';
|
||||
END IF;
|
||||
-- Read after the lock wait: a queued stale writer sees current database time.
|
||||
t := clock_timestamp();
|
||||
IF l.holder IS DISTINCT FROM p_holder OR l.epoch <> p_epoch OR l.expires_at <= t THEN
|
||||
RAISE EXCEPTION 'region lease fenced' USING ERRCODE = 'EVL02';
|
||||
END IF;
|
||||
RETURN t;
|
||||
END $$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64,
|
||||
p_region evolution.u64, p_duration_ms integer, p_build bytea)
|
||||
RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64,
|
||||
granted_ms integer)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF p_build IS NULL OR octet_length(p_build) <> 32 THEN
|
||||
RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- Eligibility is decided under the upgrade lock and held through the grant.
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b
|
||||
WHERE b.build_identity = p_build AND b.retired_at IS NULL) THEN
|
||||
RAISE EXCEPTION 'build not current' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms);
|
||||
END $$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64,
|
||||
p_region evolution.u64, p_duration_ms integer)
|
||||
RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64,
|
||||
granted_ms integer)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
IF EXISTS (SELECT 1 FROM evolution.retained_builds b WHERE b.retired_at IS NOT NULL) THEN
|
||||
RAISE EXCEPTION 'build-less acquisition ended by an upgrade barrier' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms);
|
||||
END $$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.refuse_retired_build() RETURNS trigger LANGUAGE plpgsql
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
-- Normally already held by the inserting function (same order); re-entrant.
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
IF EXISTS (SELECT 1 FROM evolution.retained_builds b
|
||||
WHERE b.build_identity = NEW.build_identity AND b.retired_at IS NOT NULL) THEN
|
||||
RAISE EXCEPTION 'build retired from publication' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.upgrade_barrier(p_retire bytea)
|
||||
RETURNS integer LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
fenced integer;
|
||||
BEGIN
|
||||
IF p_retire IS NULL OR octet_length(p_retire) <> 32 THEN
|
||||
RAISE EXCEPTION 'invalid build identity' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- First: waits out every in-flight acquisition, lease check and provisioning,
|
||||
-- and serializes barriers so the last-current-build check cannot be raced.
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock(71420002);
|
||||
PERFORM 1 FROM evolution.retained_builds b WHERE b.build_identity = p_retire FOR UPDATE;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'build not retained' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b
|
||||
WHERE b.build_identity <> p_retire AND b.retired_at IS NULL) THEN
|
||||
RAISE EXCEPTION 'no current build would remain' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
UPDATE evolution.retained_builds b SET retired_at = coalesce(b.retired_at, clock_timestamp())
|
||||
WHERE b.build_identity = p_retire;
|
||||
UPDATE evolution.region_leases l SET holder = NULL, expires_at = NULL
|
||||
WHERE l.holder IS NOT NULL;
|
||||
GET DIAGNOSTICS fenced = ROW_COUNT;
|
||||
RETURN fenced;
|
||||
END $$;
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.provision_region_checkpoint(p_manifest bytea)
|
||||
RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
|
||||
d bytea := sha256(p_manifest);
|
||||
BEGIN
|
||||
IF g.commit_sequence <> 0 OR g.predecessor_commit IS NOT NULL THEN
|
||||
RAISE EXCEPTION 'genesis manifest required' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- Upgrade lock (shared), then this world's classification lock; both checks
|
||||
-- below run after the waits, so they see every committed provisioning.
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text));
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_origins o WHERE o.world_id = g.world_id) THEN
|
||||
RAISE EXCEPTION 'world holds clones' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
INSERT INTO evolution.region_leases(world_id, region_id, frame_id)
|
||||
VALUES (g.world_id, g.region_id, g.frame_id);
|
||||
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
|
||||
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
|
||||
versions, build_identity, payload_bytes, payload_sha256)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, 0, 0, 0,
|
||||
g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes,
|
||||
g.payload_sha256);
|
||||
INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest,
|
||||
end_tick, published_epoch)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, 0, 0);
|
||||
END $$;
|
||||
|
||||
-- Owner-only clone genesis. The source may live in another database; when it is
|
||||
-- present here, the origin must be exactly its committed link. Inherited outcomes
|
||||
-- are recorded as decisions of sequence 0 without outbox rows: the source already
|
||||
-- delivered them, and a clone has no production integrations.
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.provision_region_clone(p_manifest bytea, p_source_world evolution.u64,
|
||||
p_source_region evolution.u64, p_source_sequence evolution.u64, p_decisions bytea)
|
||||
RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
|
||||
d bytea := sha256(p_manifest);
|
||||
tick numeric;
|
||||
actor numeric;
|
||||
outcome numeric;
|
||||
organism numeric;
|
||||
last_tick numeric := -1;
|
||||
last_actor numeric := -1;
|
||||
actors integer[] := '{}';
|
||||
BEGIN
|
||||
IF g.commit_sequence <> 0 OR g.predecessor_commit IS NULL THEN
|
||||
RAISE EXCEPTION 'clone genesis manifest required' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF p_source_world IS NULL OR p_source_region IS NULL OR p_source_sequence IS NULL
|
||||
OR p_decisions IS NULL THEN
|
||||
RAISE EXCEPTION 'invalid clone request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF p_source_world = g.world_id THEN
|
||||
RAISE EXCEPTION 'clone requires an isolated world' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- Upgrade lock (shared), then this world's classification lock; both checks
|
||||
-- below run after the waits, so they see every committed provisioning.
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
|
||||
PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text));
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_leases l
|
||||
WHERE l.world_id = g.world_id
|
||||
AND NOT EXISTS (SELECT 1 FROM evolution.region_origins o
|
||||
WHERE o.world_id = l.world_id AND o.region_id = l.region_id))
|
||||
OR EXISTS (SELECT 1 FROM evolution.region_origins o
|
||||
WHERE o.world_id = g.world_id AND o.source_world <> p_source_world) THEN
|
||||
RAISE EXCEPTION 'target world is not an isolated clone of this source' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_leases l
|
||||
WHERE l.world_id = p_source_world AND l.region_id = p_source_region)
|
||||
AND NOT EXISTS (SELECT 1 FROM evolution.region_commits c
|
||||
WHERE c.world_id = p_source_world AND c.region_id = p_source_region
|
||||
AND c.commit_sequence = p_source_sequence
|
||||
AND c.commit_id = g.predecessor_commit
|
||||
AND c.commit_digest = g.predecessor_digest AND c.end_tick = g.end_tick
|
||||
AND c.versions = g.versions
|
||||
AND c.tick_microseconds = g.tick_microseconds
|
||||
AND substring(c.manifest FROM 13 FOR 4) = substring(p_manifest FROM 13 FOR 4)) THEN
|
||||
RAISE EXCEPTION 'origin is not the committed source link' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF octet_length(p_decisions) <> 24 * g.decision_count
|
||||
OR (g.decision_count > 0 AND sha256(p_decisions) <> g.decisions_sha256) THEN
|
||||
RAISE EXCEPTION 'inherited outcomes do not match manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
FOR i IN 0 .. g.decision_count - 1 LOOP
|
||||
tick := evolution.manifest_uint(p_decisions, 24 * i, 8);
|
||||
actor := evolution.manifest_uint(p_decisions, 24 * i + 8, 4);
|
||||
outcome := evolution.manifest_uint(p_decisions, 24 * i + 12, 4);
|
||||
organism := evolution.manifest_uint(p_decisions, 24 * i + 16, 8);
|
||||
IF outcome <> 1 OR actor NOT IN (0, 1) OR tick < 1 OR tick > g.end_tick
|
||||
OR tick < last_tick OR (tick = last_tick AND actor <= last_actor)
|
||||
OR actor::integer = ANY (actors) OR (g.profile = 1) <> (organism = 0) THEN
|
||||
RAISE EXCEPTION 'invalid inherited outcome' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
actors := actors || actor::integer;
|
||||
last_tick := tick;
|
||||
last_actor := actor;
|
||||
END LOOP;
|
||||
INSERT INTO evolution.region_leases(world_id, region_id, frame_id)
|
||||
VALUES (g.world_id, g.region_id, g.frame_id);
|
||||
INSERT INTO evolution.region_origins(world_id, region_id, source_world, source_region,
|
||||
source_sequence, source_commit, source_digest, source_tick)
|
||||
VALUES (g.world_id, g.region_id, p_source_world, p_source_region, p_source_sequence,
|
||||
g.predecessor_commit, g.predecessor_digest, g.end_tick);
|
||||
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
|
||||
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
|
||||
versions, build_identity, payload_bytes, payload_sha256, decisions)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, g.end_tick, g.end_tick, 0,
|
||||
g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes,
|
||||
g.payload_sha256, p_decisions);
|
||||
INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest,
|
||||
end_tick, published_epoch)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, g.end_tick, 0);
|
||||
FOR i IN 0 .. g.decision_count - 1 LOOP
|
||||
INSERT INTO evolution.region_decisions(world_id, region_id, outcome, actor, tick, organism,
|
||||
commit_sequence)
|
||||
VALUES (g.world_id, g.region_id, 1, evolution.manifest_uint(p_decisions, 24 * i + 8, 4),
|
||||
evolution.manifest_uint(p_decisions, 24 * i, 8),
|
||||
evolution.manifest_uint(p_decisions, 24 * i + 16, 8), 0);
|
||||
END LOOP;
|
||||
END $$;
|
||||
|
||||
|
||||
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (9, :'migration_sha_9');
|
||||
+17
-2
@@ -35,23 +35,34 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8 \gset
|
||||
\if :at_v8
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8,
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|
||||
|| ',9:' || :'migration_sha_9' AS at_v9 \gset
|
||||
\if :at_v9
|
||||
\elif :at_v8
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v7
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v6
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v5
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v4
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v3
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
@@ -59,6 +70,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v2
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
@@ -67,6 +79,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\elif :at_v1
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
@@ -76,6 +89,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\else
|
||||
\echo 'Unknown or changed migration history'
|
||||
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
|
||||
@@ -90,6 +104,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
|
||||
|
||||
@@ -123,8 +123,8 @@ void OpsServer::respond(Connection& c, const OpsHandler& handler) {
|
||||
}
|
||||
|
||||
void OpsServer::serve(const OpsHandler& handler) {
|
||||
const auto now = Clock::now();
|
||||
for (;;) {
|
||||
const auto accepted_at = Clock::now();
|
||||
for (std::size_t attempt = 0; attempt < max_accepts_per_serve; ++attempt) {
|
||||
const int fd = ::accept4(fd_, nullptr, nullptr, SOCK_NONBLOCK | SOCK_CLOEXEC);
|
||||
if (fd < 0)
|
||||
break; // EAGAIN: nothing pending (other errors: retried next call).
|
||||
@@ -133,8 +133,9 @@ void OpsServer::serve(const OpsHandler& handler) {
|
||||
++refused_;
|
||||
continue;
|
||||
}
|
||||
connections_.push_back({fd, {}, {}, 0, now + connection_deadline});
|
||||
connections_.push_back({fd, {}, {}, 0, accepted_at + connection_deadline});
|
||||
}
|
||||
const auto now = Clock::now(); // Deadlines are judged after the bounded accept work.
|
||||
for (auto it = connections_.begin(); it != connections_.end();) {
|
||||
auto& c = *it;
|
||||
bool done = false;
|
||||
|
||||
@@ -171,6 +171,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
" -v migration_sha_5=$(s 0005_batch_decisions.sql)"
|
||||
" -v migration_sha_6=$(s 0006_commit_lookup.sql)"
|
||||
" -v migration_sha_7=$(s 0007_region_clones.sql) -v migration_sha_8=invalid"
|
||||
" -v migration_sha_9=$(s 0009_serialized_authority.sql)"
|
||||
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
|
||||
" > /dev/null 2>&1") != 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
|
||||
@@ -186,7 +187,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
CHECK(shell("PGDATABASE=" + database +
|
||||
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
|
||||
"tools/migrate-postgres.sh\" > /dev/null") == 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8");
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9");
|
||||
CHECK(renew_region_lease(*runtime, r.key, *lease, 20000ms, 5s));
|
||||
confirm(publish_next(20)); // 41..60, now v8.
|
||||
CHECK(head.head.end_tick == 60);
|
||||
@@ -395,9 +396,161 @@ void durability_qualification() {
|
||||
static_cast<unsigned long long>(audit_ms),
|
||||
static_cast<unsigned long long>(clone_ms));
|
||||
}
|
||||
// Review fixes (schema v9): builds, authority and world classification follow one
|
||||
// lock order. Each case holds a transaction open, proves the contender is blocked
|
||||
// on a lock in the database, commits, and checks the contender's outcome.
|
||||
struct Race {
|
||||
std::string database;
|
||||
// Commits the holder's open transaction once `contender` is waiting on a lock.
|
||||
template <typename F> auto run(Postgres& holder, F contender) {
|
||||
std::optional<decltype(contender())> result;
|
||||
std::thread thread([&] { result.emplace(contender()); });
|
||||
auto observer = connect_to(database, env("PGUSER"));
|
||||
const std::array<std::string, 1> p{database};
|
||||
bool waiting = false;
|
||||
for (int i = 0; i < 500 && !waiting; ++i) {
|
||||
auto rows = observer->query("SELECT count(*)::text FROM pg_catalog.pg_stat_activity "
|
||||
"WHERE datname = $1 AND wait_event_type = 'Lock'",
|
||||
p, 5s);
|
||||
waiting = rows && rows->size() == 1 && (*rows)[0][0].value_or("0") != "0";
|
||||
if (!waiting)
|
||||
std::this_thread::sleep_for(20ms);
|
||||
}
|
||||
CHECK(holder.query("COMMIT", {}, 5s));
|
||||
thread.join();
|
||||
CHECK(waiting);
|
||||
if (!result.has_value())
|
||||
throw std::runtime_error("race contender produced no result");
|
||||
return std::move(*result);
|
||||
}
|
||||
};
|
||||
std::string bytea_of(std::span<const std::uint8_t> bytes) {
|
||||
constexpr char digits[] = "0123456789abcdef";
|
||||
std::string out = "\\x";
|
||||
for (const auto b : bytes) {
|
||||
out.push_back(digits[b >> 4U]);
|
||||
out.push_back(digits[b & 0x0FU]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::string build_literal(std::uint8_t b) {
|
||||
char out[64];
|
||||
std::snprintf(out, sizeof out, "decode(repeat('%02x', 32), 'hex')", b);
|
||||
return out;
|
||||
}
|
||||
void concurrent_authority_and_classification() {
|
||||
const std::string database = "evo_race";
|
||||
CHECK(shell("createdb " + database + " && PGDATABASE=" + database +
|
||||
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
|
||||
"tools/migrate-postgres.sh\" > /dev/null") == 0);
|
||||
auto owner_db = connect_to(database, env("PGUSER"));
|
||||
CHECK(owner_db->query("INSERT INTO evolution.retained_builds(build_identity) SELECT "
|
||||
"decode(repeat(b, 32), 'hex') FROM unnest(ARRAY['c1', 'c2', 'c3', "
|
||||
"'c4']) AS b",
|
||||
{}, 5s));
|
||||
const Sha256Digest c2 = digest_of(0xC2), c3 = digest_of(0xC3), c4 = digest_of(0xC4);
|
||||
const RegionKey key{7, 1};
|
||||
auto genesis = genesis_manifest(key, 9, filled<16>(0x61), 50000, c4, 100, filled<32>(0xA0));
|
||||
CHECK(genesis);
|
||||
const std::array<std::string, 1> gp{bytea_of(*encode_manifest(*genesis))};
|
||||
CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", gp, 5s));
|
||||
Race race{database};
|
||||
const auto barrier_holding_lease_row = [&](std::uint8_t build) {
|
||||
CHECK(owner_db->query("BEGIN", {}, 5s));
|
||||
CHECK(owner_db->query("SELECT 1 FROM evolution.region_leases WHERE world_id = 7 AND "
|
||||
"region_id = 1 FOR UPDATE",
|
||||
{}, 5s));
|
||||
CHECK(scalar(*owner_db, "SELECT evolution.upgrade_barrier(" + build_literal(build) +
|
||||
")::text") == "0");
|
||||
};
|
||||
// Build-less acquisition queued behind the first barrier is refused after it.
|
||||
barrier_holding_lease_row(0xC1);
|
||||
auto legacy = race.run(*owner_db, [&] {
|
||||
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
|
||||
return legacy_acquire(*runtime, key);
|
||||
});
|
||||
CHECK(!legacy && legacy.error() == "EVL06");
|
||||
// Build-aware acquisition for the build being retired is refused after COMMIT.
|
||||
barrier_holding_lease_row(0xC2);
|
||||
auto retired = race.run(*owner_db, [&] {
|
||||
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
|
||||
return acquire_region_lease(*runtime, key, c2, 20000ms, 5s);
|
||||
});
|
||||
CHECK(!retired && retired.error().kind == LeaseStoreError::unsupported);
|
||||
CHECK(scalar(*owner_db, "SELECT (holder IS NULL)::text FROM evolution.region_leases WHERE "
|
||||
"world_id = 7 AND region_id = 1") == "true");
|
||||
// The new build acquires; the rejected one never held anything to renew.
|
||||
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
|
||||
auto current = acquire_region_lease(*runtime, key, c3, 20000ms, 5s);
|
||||
CHECK(current);
|
||||
auto stale = renew_region_lease(*runtime, key, {filled<16>(0x77), 1, 9, 20000ms}, 20000ms, 5s);
|
||||
CHECK(!stale && stale.error().kind == LeaseStoreError::fenced);
|
||||
CHECK(release_region_lease(*runtime, key, *current, 5s));
|
||||
// Two barriers for the last two current builds: exactly one may succeed.
|
||||
CHECK(owner_db->query("BEGIN", {}, 5s));
|
||||
CHECK(scalar(*owner_db,
|
||||
"SELECT evolution.upgrade_barrier(" + build_literal(0xC3) + ")::text") == "0");
|
||||
auto last = race.run(*owner_db, [&] {
|
||||
auto other = connect_to(database, env("PGUSER"));
|
||||
return other->query("SELECT evolution.upgrade_barrier(" + build_literal(0xC4) + ")", {},
|
||||
5s);
|
||||
});
|
||||
CHECK(!last && sqlstate(last.error()) == "22023");
|
||||
CHECK(scalar(*owner_db, "SELECT count(*)::text FROM evolution.retained_builds WHERE "
|
||||
"retired_at IS NULL") == "1");
|
||||
|
||||
// Classification: a synthetic source cut (absent from this database) and clone
|
||||
// manifests naming it; the database never reads payload objects.
|
||||
const auto source_cut = [&](std::uint64_t world, std::uint8_t seed) {
|
||||
auto m =
|
||||
genesis_manifest({world, 1}, 9, filled<16>(seed), 50000, c4, 100, filled<32>(0xA1));
|
||||
CHECK(m);
|
||||
return *m;
|
||||
};
|
||||
const auto clone_sql = [&](Postgres& db, RegionKey target, std::uint64_t source_world,
|
||||
std::uint8_t seed) {
|
||||
auto m = clone_manifest(source_cut(source_world, static_cast<std::uint8_t>(seed + 1)),
|
||||
filled<32>(0xD5), target, filled<16>(seed), 100, filled<32>(0xA2),
|
||||
{0, {}, 0, {}});
|
||||
CHECK(m);
|
||||
const std::array<std::string, 3> p{bytea_of(*encode_manifest(*m)),
|
||||
std::to_string(source_world), "1"};
|
||||
return db.query("SELECT evolution.provision_region_clone($1::bytea, $2::evolution.u64, "
|
||||
"$3::evolution.u64, 0, '\\x'::bytea)",
|
||||
p, 5s);
|
||||
};
|
||||
const auto contender = [&](RegionKey target, std::uint64_t source_world, std::uint8_t seed) {
|
||||
return [&, target, source_world, seed] {
|
||||
auto other = connect_to(database, env("PGUSER"));
|
||||
return clone_sql(*other, target, source_world, seed);
|
||||
};
|
||||
};
|
||||
// Different sources into one target world: exactly one source is accepted.
|
||||
CHECK(owner_db->query("BEGIN", {}, 5s));
|
||||
CHECK(clone_sql(*owner_db, {2, 1}, 10, 0x62));
|
||||
auto mixed = race.run(*owner_db, contender({2, 2}, 11, 0x64));
|
||||
CHECK(!mixed && sqlstate(mixed.error()) == "22023");
|
||||
// An original region racing a clone into the same world.
|
||||
auto original = genesis_manifest({3, 1}, 9, filled<16>(0x66), 50000, c4, 100, filled<32>(0xA0));
|
||||
CHECK(original);
|
||||
const std::array<std::string, 1> op{bytea_of(*encode_manifest(*original))};
|
||||
CHECK(owner_db->query("BEGIN", {}, 5s));
|
||||
CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", op, 5s));
|
||||
auto against_original = race.run(*owner_db, contender({3, 2}, 10, 0x68));
|
||||
CHECK(!against_original && sqlstate(against_original.error()) == "22023");
|
||||
// Same source, different regions: both succeed.
|
||||
CHECK(owner_db->query("BEGIN", {}, 5s));
|
||||
CHECK(clone_sql(*owner_db, {4, 1}, 10, 0x6A));
|
||||
auto same_source = race.run(*owner_db, contender({4, 2}, 10, 0x6C));
|
||||
CHECK(same_source);
|
||||
CHECK(scalar(*owner_db, "SELECT string_agg(world_id || '/' || region_id || '<' || "
|
||||
"source_world, ',' ORDER BY world_id, region_id) FROM "
|
||||
"evolution.region_origins") == "2/1<10,4/1<10,4/2<10");
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_migration_tests() {
|
||||
concurrent_authority_and_classification();
|
||||
durability_qualification();
|
||||
upgrade_across_migration_and_barrier();
|
||||
}
|
||||
|
||||
@@ -1,9 +1,11 @@
|
||||
// M7.8 operational endpoints on real evo-server processes (ADR-0066): liveness is
|
||||
// never authority, readiness follows the lease, metrics are bounded, and the
|
||||
// responder is bounded against idle, excess and malformed clients.
|
||||
#include "evo/server/ops_http.hpp"
|
||||
#include "fixtures/server_process.hpp"
|
||||
#include <algorithm>
|
||||
#include <arpa/inet.h>
|
||||
#include <atomic>
|
||||
#include <netinet/in.h>
|
||||
#include <source_location>
|
||||
#include <sys/socket.h>
|
||||
@@ -184,9 +186,84 @@ void stale_process_is_live_not_ready() {
|
||||
current.signal(SIGTERM);
|
||||
CHECK(stale.wait_exit(20s) == 0 && current.wait_exit(20s) == 0);
|
||||
}
|
||||
|
||||
// Review fix: accept work per serve() is bounded, refused connections included, so
|
||||
// replenished connection pressure cannot hold the host loop; accepted connections
|
||||
// are still answered while the pressure continues.
|
||||
void accept_work_is_bounded_per_serve() {
|
||||
using evo::server::OpsServer;
|
||||
auto made = OpsServer::listen("127.0.0.1:0");
|
||||
CHECK(made);
|
||||
auto& ops = **made;
|
||||
const auto handler = [](std::string_view) { return evo::server::OpsResponse{200, "ok\n"}; };
|
||||
constexpr auto per = OpsServer::max_accepts_per_serve;
|
||||
std::vector<int> clients;
|
||||
clients.reserve(64);
|
||||
for (int i = 0; i < 16; ++i)
|
||||
clients.push_back(connect_local(ops.port())); // Queued in the listen backlog.
|
||||
std::this_thread::sleep_for(50ms);
|
||||
ops.serve(handler);
|
||||
CHECK(ops.open_connections() == per && ops.refused() == 0);
|
||||
ops.serve(handler);
|
||||
CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == 0);
|
||||
// At the connection cap, each call still refuses at most the budget.
|
||||
ops.serve(handler);
|
||||
CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == per);
|
||||
// The first accepted client is answered while new connections keep arriving.
|
||||
const std::string request = "GET /livez HTTP/1.1\r\n\r\n";
|
||||
CHECK(::send(clients[0], request.data(), request.size(), MSG_NOSIGNAL) ==
|
||||
static_cast<ssize_t>(request.size()));
|
||||
for (int round = 0; round < 6; ++round) {
|
||||
clients.push_back(connect_local(ops.port()));
|
||||
const auto before = ops.refused();
|
||||
ops.serve(handler);
|
||||
CHECK(ops.refused() - before <= per);
|
||||
}
|
||||
char reply[64] = {};
|
||||
CHECK(::recv(clients[0], reply, sizeof reply - 1, 0) > 0);
|
||||
CHECK(std::string_view(reply).starts_with("HTTP/1.1 200"));
|
||||
for (const int fd : clients)
|
||||
::close(fd);
|
||||
}
|
||||
|
||||
// A real server under sustained connection churn keeps its lease and readiness.
|
||||
void connection_pressure_keeps_authority() {
|
||||
const auto r = provision(54, 0x0D);
|
||||
Child server{with_ops(r)};
|
||||
const auto port = ops_port(server);
|
||||
CHECK(server.wait_for([](const std::string& l) { return l == "ready 1"; }, 15s));
|
||||
std::atomic<bool> stop{false};
|
||||
std::thread flood([&] {
|
||||
while (!stop.load()) {
|
||||
const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
|
||||
sockaddr_in address{};
|
||||
address.sin_family = AF_INET;
|
||||
address.sin_port = htons(port);
|
||||
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
|
||||
if (fd >= 0) {
|
||||
// Refusal or backlog overflow is part of the pressure; ignore it.
|
||||
[[maybe_unused]] const int connected =
|
||||
::connect(fd, reinterpret_cast<sockaddr*>(&address), sizeof address);
|
||||
::close(fd);
|
||||
}
|
||||
}
|
||||
});
|
||||
// Longer than the fast lease (4 s): renewals must keep happening under pressure.
|
||||
server.send(input_lines(1, 60, false));
|
||||
const bool acked = server.wait_line("ack " + request_hex(60) + " 60 ", 20s);
|
||||
stop = true;
|
||||
flood.join();
|
||||
CHECK(acked);
|
||||
CHECK(!server.printed("ready 0"));
|
||||
CHECK(get(port, "/readyz").status == 200);
|
||||
server.signal(SIGTERM);
|
||||
CHECK(server.wait_exit(20s) == 0);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_ops_tests() {
|
||||
accept_work_is_bounded_per_serve();
|
||||
connection_pressure_keeps_authority();
|
||||
probes_follow_authority_and_stay_bounded();
|
||||
unavailable_database_is_live_not_ready();
|
||||
stale_process_is_live_not_ready();
|
||||
|
||||
@@ -55,7 +55,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
|
||||
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9
|
||||
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
|
||||
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
|
||||
v2_sha="${v2_sha%% *}"
|
||||
@@ -76,7 +76,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 8:1,2,3,4,5,6,7,8
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 9:1,2,3,4,5,6,7,8,9
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
|
||||
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
|
||||
@@ -130,7 +130,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
|
||||
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
|
||||
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
|
||||
@@ -165,9 +165,11 @@ expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.provision_region_check
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.region_commits TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commits FROM evo_runtime_test'
|
||||
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
|
||||
# Unknown future versions and checksum drift must not be repaired silently.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=9 WHERE version=8'
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=10 WHERE version=9'
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=9 WHERE version=10"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=9"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=8 WHERE version=9"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=8"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=7"
|
||||
|
||||
@@ -28,10 +28,12 @@ digest_7="$(sha256sum "$root/migrations/0007_region_clones.sql")"
|
||||
digest_7="${digest_7%% *}"
|
||||
digest_8="$(sha256sum "$root/migrations/0008_upgrade_barrier.sql")"
|
||||
digest_8="${digest_8%% *}"
|
||||
digest_9="$(sha256sum "$root/migrations/0009_serialized_authority.sql")"
|
||||
digest_9="${digest_9%% *}"
|
||||
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
|
||||
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
|
||||
--set="migration_sha_6=$digest_6" --set="migration_sha_7=$digest_7" \
|
||||
--set="migration_sha_8=$digest_8" \
|
||||
--set="migration_sha_8=$digest_8" --set="migration_sha_9=$digest_9" \
|
||||
--set="runtime_role=$runtime_role" \
|
||||
--file="$root/migrations/apply.sql"
|
||||
|
||||
Reference in New Issue
Block a user