Serialize build retirement and clone classification, bound ops accepts

Review fixes for M7.6-M7.9. Schema v9 applies one lock order: the
barrier takes a database-wide upgrade lock exclusively, while
acquisitions, lease checks, provisioning and the retired-build trigger
take it shared first, so a queued acquisition can no longer regain
authority for a just-retired build and concurrent barriers cannot
retire the last current build. Provisioning takes a per-world lock
before classifying a world as original or a clone of one source. The
operations listener makes at most four accept attempts per serve().
Race regressions prove each contender is blocked before commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 08:38:13 -04:00
co-authored by Claude Opus 5.5
parent f59cca2c3b
commit 168c90d5b2
14 changed files with 616 additions and 15 deletions
+5
View File
@@ -954,3 +954,8 @@ lease:
run. Host-loss durability is M8.
Next: M8.1, the host inventory and budgets.
Review fixes (2026-09-29): schema v9 serializes build retirement with acquisition,
leases and provisioning under one lock order. It also serializes clone-world
classification per world. The operations listener's accept work per loop pass is
now bounded. Regression tests and five mutations cover each fix.
+12
View File
@@ -99,3 +99,15 @@ zlib, CA certificates):
- A later gateway must honor readiness and the lease epoch before routing.
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
(M8).
## Amendment: bounded accept work (2026-09-29 review)
`serve()` used to accept, and close excess connections, until the backlog was
empty. A replenished stream of connections could therefore hold the host loop
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
attempts, refused ones included. Pending connections wait in the 16-entry listen
backlog, and deadlines are judged after the accept work.
Tests check the per-call budget, including at the connection cap, and that an
accepted probe is still answered under replenished pressure. A real server keeps
its lease and readiness while a client opens and closes connections continuously
through 60 ticks.
+15
View File
@@ -109,3 +109,18 @@ alone does not prove retention; the audit does.
defeats isolation.
- **Recording provenance only in a table.** The chain would not commit to its
origin, and cross-database clones could not verify it.
## Amendment: serialized world classification (schema v9, 2026-09-29 review)
The rule "original, or a clone of exactly one source" was enforced by unlocked
existence checks. Two concurrent clones of different sources into different
regions of one world could therefore both commit (reproduced by review). Schema
v9 fixes this:
- Both provisioning functions take a per-world advisory lock, keyed by a hash of
the world ID, after the shared upgrade lock (ADR-0068 amendment).
- They then check the classification. Because the checks run after the lock wait,
they see every committed provisioning of that world.
- Unrelated worlds provision in parallel. A hash collision only adds waiting.
Regression tests race different-source clones, and an original region against a
clone, into one world, and expect exactly one to succeed. Same-source clones into
different regions both succeed.
+25
View File
@@ -94,3 +94,28 @@ The results are in engineering/DURABILITY_QUALIFICATION.md.
mid-upgrade before the new build was deployed.
- **In-place rules migration of a live region.** It would change physical laws
without the clone-and-compare approval that the vision requires.
## Amendment: serialized retirement (schema v9, 2026-09-29 review)
In v8, acquisition checked eligibility before locking the lease row, without
serializing against the barrier. A request queued behind a barrier could
therefore acquire for the build just retired, then keep renewing (reproduced by
review). Schema v9 fixes this with one lock order: the upgrade lock (advisory,
database-wide), then the world classification lock, then the lease row, then
head and commit rows.
- `upgrade_barrier` takes the upgrade lock exclusively before anything else. This
also serializes barriers, so two operators cannot each retire what they believe
is a non-last build.
- Both acquisition forms, `check_region_lease` (and with it publication, renewal
and release), provisioning and the retired-build trigger take the upgrade lock
shared first. Their eligibility checks therefore run after any concurrent
barrier commits.
The lock order is the same everywhere, so publication cannot deadlock against the
barrier. Regression tests hold a barrier open, prove the contender is blocked,
commit, and check that a queued acquisition of the retired build is refused (both
forms) and that the last current build cannot be retired by racing barriers.
A runtime login can still take PostgreSQL advisory locks directly, and so could
delay barriers or acquisitions. It cannot gain authority that way. An owner can
terminate such sessions. Removing that capability needs a pooler or role policy
(M8).
+46
View File
@@ -2534,3 +2534,49 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
The migration test caught this one after the host-test correction.
- Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and
host or storage loss (M8).
## M7.6–M7.9 review fixes (schema v9) - 2026-09-29
Review `.cache/reviews/M7_6-M7_9_REVIEW.md` found three issues. All three are
fixed:
- **[P1] Build retirement could race acquisition.** Schema v9 applies one
upgrade-lock order across the barrier, acquisitions, lease checks, provisioning
and the retired-build trigger.
- **[P2] Clone-world classification could race.** Provisioning now takes a
per-world lock and checks classification after it.
- **[P2] The operations listener accepted without bound.** Each `serve()` now
makes at most 4 accept attempts.
Evidence:
- WSL Linux `checks` set (Debug, Release, analysis, ASan/UBSan): CTest 59/59
each. The format check passed.
- PostgreSQL ASan/UBSan: full suite passed. Analysis and Release were rerun green
after fixing two clang-tidy findings in the new tests. Upgrade paths now end at
1..9.
- Race tests (database `evo_race`): each holds a transaction open, confirms the
contender is blocked on a lock via `pg_stat_activity`, then commits.
- A build-less acquisition queued behind a barrier is refused (EVL06).
- A build-aware acquisition of the build being retired is refused, and the
region stays unheld.
- The next build acquires.
- Racing barriers for the last two current builds: one returns 22023 and one
build stays current.
- Different-source clones into one world, and an original region racing a
clone: the second is refused (22023).
- Same-source clones into different regions both succeed.
- The final origins are exactly `2/1<10, 4/1<10, 4/2<10`.
- Accept budget:
- With 16 queued connections, successive `serve()` calls give 4 open, then 8
open with 0 refused, then 8 open with 4 refused.
- Refusals never exceed 4 per call under replenished connections.
- The first accepted probe got 200.
- A real server acknowledged 60 ticks under continuous connect/close churn
without losing readiness.
- Mutations, each reverting one fix (restored and rebuilt). Each was caught:
- an unlocked build-aware acquire;
- an unlocked legacy acquire;
- an unlocked barrier;
- no per-world lock;
- an accept budget of 1000.
- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s.
- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files.
+7
View File
@@ -164,6 +164,13 @@ To upgrade the build:
returns the number of leases fenced, and refuses to retire the last current
build.
Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The
barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions,
lease checks, provisioning and the retired-build trigger take it shared, and
provisioning also takes a per-world lock (71420003). A barrier therefore briefly
waits for in-flight commits and blocks new ones. Keep other advisory locks away
from these two keys.
Old images then quarantine (exit 4) instead of serving. Rules and format changes
are refused explicitly; there is no converter.
+9 -3
View File
@@ -27,6 +27,9 @@ enum class OpsError : std::uint8_t { address, socket };
class OpsServer {
public:
static constexpr std::size_t max_connections = 8;
// Accept attempts per serve(), refused ones included: sustained connection
// pressure cannot hold the host loop; the rest wait in the listen backlog.
static constexpr std::size_t max_accepts_per_serve = 4;
static constexpr std::size_t max_request_bytes = 4096;
static constexpr std::size_t max_response_bytes = 16384;
static constexpr std::chrono::milliseconds connection_deadline{1000};
@@ -42,13 +45,16 @@ class OpsServer {
[[nodiscard]] std::uint16_t port() const noexcept {
return port_;
}
// Accepts, reads, answers and expires connections without waiting. Connections
// beyond the bound are closed at once; slow or oversized ones are answered or
// dropped without delaying others.
// Accepts at most max_accepts_per_serve connections, then reads, answers and
// expires connections without waiting. Connections beyond the bound are closed
// at once; slow or oversized ones are answered or dropped without delaying others.
void serve(const OpsHandler& handler);
[[nodiscard]] std::uint64_t refused() const noexcept {
return refused_;
}
[[nodiscard]] std::size_t open_connections() const noexcept {
return connections_.size();
}
private:
struct Connection {
+235
View File
@@ -0,0 +1,235 @@
-- M7.9 review fixes (ADR-0067/0068): one locking protocol for authority, build
-- retirement and world classification. Replaces function bodies only.
--
-- Lock order, everywhere: upgrade lock (advisory 71420002) -> world classification
-- lock (advisory 71420003, hashtext(world)) -> lease row -> head/commit rows.
-- upgrade_barrier takes the upgrade lock exclusively before anything else; every
-- acquisition, lease check (so publication, renewal and release), provisioning and
-- the retired-build trigger take it shared first. A retirement therefore cannot
-- commit between an acquisition's eligibility check and its grant, a commit naming
-- a build cannot straddle its retirement, and concurrent barriers serialize. The
-- per-world lock serializes original/clone classification of one target world
-- while unrelated worlds provision in parallel. Callers use READ COMMITTED
-- statements (as the adapter does), so each check after a wait sees committed rows.
CREATE OR REPLACE FUNCTION evolution.check_region_lease(p_world evolution.u64,
p_region evolution.u64, p_holder uuid, p_epoch evolution.u64)
RETURNS timestamptz LANGUAGE plpgsql SECURITY INVOKER
SET search_path = pg_catalog, pg_temp AS $$
DECLARE
l evolution.region_leases%ROWTYPE;
t timestamptz;
BEGIN
IF p_world IS NULL OR p_region IS NULL OR p_holder IS NULL OR p_epoch IS NULL THEN
RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023';
END IF;
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
SELECT * INTO l FROM evolution.region_leases r
WHERE r.world_id = p_world AND r.region_id = p_region FOR UPDATE;
IF NOT FOUND THEN
RAISE EXCEPTION 'region not provisioned' USING ERRCODE = 'EVL04';
END IF;
-- Read after the lock wait: a queued stale writer sees current database time.
t := clock_timestamp();
IF l.holder IS DISTINCT FROM p_holder OR l.epoch <> p_epoch OR l.expires_at <= t THEN
RAISE EXCEPTION 'region lease fenced' USING ERRCODE = 'EVL02';
END IF;
RETURN t;
END $$;
CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64,
p_region evolution.u64, p_duration_ms integer, p_build bytea)
RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64,
granted_ms integer)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
BEGIN
IF p_build IS NULL OR octet_length(p_build) <> 32 THEN
RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023';
END IF;
-- Eligibility is decided under the upgrade lock and held through the grant.
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b
WHERE b.build_identity = p_build AND b.retired_at IS NULL) THEN
RAISE EXCEPTION 'build not current' USING ERRCODE = 'EVL06';
END IF;
RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms);
END $$;
CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64,
p_region evolution.u64, p_duration_ms integer)
RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64,
granted_ms integer)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
BEGIN
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
IF EXISTS (SELECT 1 FROM evolution.retained_builds b WHERE b.retired_at IS NOT NULL) THEN
RAISE EXCEPTION 'build-less acquisition ended by an upgrade barrier' USING ERRCODE = 'EVL06';
END IF;
RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms);
END $$;
CREATE OR REPLACE FUNCTION evolution.refuse_retired_build() RETURNS trigger LANGUAGE plpgsql
SET search_path = pg_catalog, pg_temp AS $$
BEGIN
-- Normally already held by the inserting function (same order); re-entrant.
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
IF EXISTS (SELECT 1 FROM evolution.retained_builds b
WHERE b.build_identity = NEW.build_identity AND b.retired_at IS NOT NULL) THEN
RAISE EXCEPTION 'build retired from publication' USING ERRCODE = 'EVL06';
END IF;
RETURN NEW;
END $$;
CREATE OR REPLACE FUNCTION evolution.upgrade_barrier(p_retire bytea)
RETURNS integer LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
DECLARE
fenced integer;
BEGIN
IF p_retire IS NULL OR octet_length(p_retire) <> 32 THEN
RAISE EXCEPTION 'invalid build identity' USING ERRCODE = '22023';
END IF;
-- First: waits out every in-flight acquisition, lease check and provisioning,
-- and serializes barriers so the last-current-build check cannot be raced.
PERFORM pg_catalog.pg_advisory_xact_lock(71420002);
PERFORM 1 FROM evolution.retained_builds b WHERE b.build_identity = p_retire FOR UPDATE;
IF NOT FOUND THEN
RAISE EXCEPTION 'build not retained' USING ERRCODE = '22023';
END IF;
IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b
WHERE b.build_identity <> p_retire AND b.retired_at IS NULL) THEN
RAISE EXCEPTION 'no current build would remain' USING ERRCODE = '22023';
END IF;
UPDATE evolution.retained_builds b SET retired_at = coalesce(b.retired_at, clock_timestamp())
WHERE b.build_identity = p_retire;
UPDATE evolution.region_leases l SET holder = NULL, expires_at = NULL
WHERE l.holder IS NOT NULL;
GET DIAGNOSTICS fenced = ROW_COUNT;
RETURN fenced;
END $$;
CREATE OR REPLACE FUNCTION evolution.provision_region_checkpoint(p_manifest bytea)
RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
DECLARE
g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
d bytea := sha256(p_manifest);
BEGIN
IF g.commit_sequence <> 0 OR g.predecessor_commit IS NOT NULL THEN
RAISE EXCEPTION 'genesis manifest required' USING ERRCODE = '22023';
END IF;
-- Upgrade lock (shared), then this world's classification lock; both checks
-- below run after the waits, so they see every committed provisioning.
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text));
IF EXISTS (SELECT 1 FROM evolution.region_origins o WHERE o.world_id = g.world_id) THEN
RAISE EXCEPTION 'world holds clones' USING ERRCODE = '22023';
END IF;
INSERT INTO evolution.region_leases(world_id, region_id, frame_id)
VALUES (g.world_id, g.region_id, g.frame_id);
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
versions, build_identity, payload_bytes, payload_sha256)
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, 0, 0, 0,
g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes,
g.payload_sha256);
INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest,
end_tick, published_epoch)
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, 0, 0);
END $$;
-- Owner-only clone genesis. The source may live in another database; when it is
-- present here, the origin must be exactly its committed link. Inherited outcomes
-- are recorded as decisions of sequence 0 without outbox rows: the source already
-- delivered them, and a clone has no production integrations.
CREATE OR REPLACE FUNCTION evolution.provision_region_clone(p_manifest bytea, p_source_world evolution.u64,
p_source_region evolution.u64, p_source_sequence evolution.u64, p_decisions bytea)
RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
DECLARE
g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
d bytea := sha256(p_manifest);
tick numeric;
actor numeric;
outcome numeric;
organism numeric;
last_tick numeric := -1;
last_actor numeric := -1;
actors integer[] := '{}';
BEGIN
IF g.commit_sequence <> 0 OR g.predecessor_commit IS NULL THEN
RAISE EXCEPTION 'clone genesis manifest required' USING ERRCODE = '22023';
END IF;
IF p_source_world IS NULL OR p_source_region IS NULL OR p_source_sequence IS NULL
OR p_decisions IS NULL THEN
RAISE EXCEPTION 'invalid clone request' USING ERRCODE = '22023';
END IF;
IF p_source_world = g.world_id THEN
RAISE EXCEPTION 'clone requires an isolated world' USING ERRCODE = '22023';
END IF;
-- Upgrade lock (shared), then this world's classification lock; both checks
-- below run after the waits, so they see every committed provisioning.
PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002);
PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text));
IF EXISTS (SELECT 1 FROM evolution.region_leases l
WHERE l.world_id = g.world_id
AND NOT EXISTS (SELECT 1 FROM evolution.region_origins o
WHERE o.world_id = l.world_id AND o.region_id = l.region_id))
OR EXISTS (SELECT 1 FROM evolution.region_origins o
WHERE o.world_id = g.world_id AND o.source_world <> p_source_world) THEN
RAISE EXCEPTION 'target world is not an isolated clone of this source' USING ERRCODE = '22023';
END IF;
IF EXISTS (SELECT 1 FROM evolution.region_leases l
WHERE l.world_id = p_source_world AND l.region_id = p_source_region)
AND NOT EXISTS (SELECT 1 FROM evolution.region_commits c
WHERE c.world_id = p_source_world AND c.region_id = p_source_region
AND c.commit_sequence = p_source_sequence
AND c.commit_id = g.predecessor_commit
AND c.commit_digest = g.predecessor_digest AND c.end_tick = g.end_tick
AND c.versions = g.versions
AND c.tick_microseconds = g.tick_microseconds
AND substring(c.manifest FROM 13 FOR 4) = substring(p_manifest FROM 13 FOR 4)) THEN
RAISE EXCEPTION 'origin is not the committed source link' USING ERRCODE = '22023';
END IF;
IF octet_length(p_decisions) <> 24 * g.decision_count
OR (g.decision_count > 0 AND sha256(p_decisions) <> g.decisions_sha256) THEN
RAISE EXCEPTION 'inherited outcomes do not match manifest' USING ERRCODE = '22023';
END IF;
FOR i IN 0 .. g.decision_count - 1 LOOP
tick := evolution.manifest_uint(p_decisions, 24 * i, 8);
actor := evolution.manifest_uint(p_decisions, 24 * i + 8, 4);
outcome := evolution.manifest_uint(p_decisions, 24 * i + 12, 4);
organism := evolution.manifest_uint(p_decisions, 24 * i + 16, 8);
IF outcome <> 1 OR actor NOT IN (0, 1) OR tick < 1 OR tick > g.end_tick
OR tick < last_tick OR (tick = last_tick AND actor <= last_actor)
OR actor::integer = ANY (actors) OR (g.profile = 1) <> (organism = 0) THEN
RAISE EXCEPTION 'invalid inherited outcome' USING ERRCODE = '22023';
END IF;
actors := actors || actor::integer;
last_tick := tick;
last_actor := actor;
END LOOP;
INSERT INTO evolution.region_leases(world_id, region_id, frame_id)
VALUES (g.world_id, g.region_id, g.frame_id);
INSERT INTO evolution.region_origins(world_id, region_id, source_world, source_region,
source_sequence, source_commit, source_digest, source_tick)
VALUES (g.world_id, g.region_id, p_source_world, p_source_region, p_source_sequence,
g.predecessor_commit, g.predecessor_digest, g.end_tick);
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
versions, build_identity, payload_bytes, payload_sha256, decisions)
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, g.end_tick, g.end_tick, 0,
g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes,
g.payload_sha256, p_decisions);
INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest,
end_tick, published_epoch)
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, g.end_tick, 0);
FOR i IN 0 .. g.decision_count - 1 LOOP
INSERT INTO evolution.region_decisions(world_id, region_id, outcome, actor, tick, organism,
commit_sequence)
VALUES (g.world_id, g.region_id, 1, evolution.manifest_uint(p_decisions, 24 * i + 8, 4),
evolution.manifest_uint(p_decisions, 24 * i, 8),
evolution.manifest_uint(p_decisions, 24 * i + 16, 8), 0);
END LOOP;
END $$;
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (9, :'migration_sha_9');
+17 -2
View File
@@ -35,23 +35,34 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8 \gset
\if :at_v8
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8,
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|| ',9:' || :'migration_sha_9' AS at_v9 \gset
\if :at_v9
\elif :at_v8
\ir 0009_serialized_authority.sql
\elif :at_v7
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v6
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v5
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v4
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v3
\ir 0004_lineage_profile.sql
\ir pre_v5_guard.sql
@@ -59,6 +70,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v2
\ir 0003_region_checkpoints.sql
\ir 0004_lineage_profile.sql
@@ -67,6 +79,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\elif :at_v1
\ir 0002_region_fencing.sql
\ir 0003_region_checkpoints.sql
@@ -76,6 +89,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\else
\echo 'Unknown or changed migration history'
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
@@ -90,6 +104,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
\ir 0006_commit_lookup.sql
\ir 0007_region_clones.sql
\ir 0008_upgrade_barrier.sql
\ir 0009_serialized_authority.sql
\endif
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
+4 -3
View File
@@ -123,8 +123,8 @@ void OpsServer::respond(Connection& c, const OpsHandler& handler) {
}
void OpsServer::serve(const OpsHandler& handler) {
const auto now = Clock::now();
for (;;) {
const auto accepted_at = Clock::now();
for (std::size_t attempt = 0; attempt < max_accepts_per_serve; ++attempt) {
const int fd = ::accept4(fd_, nullptr, nullptr, SOCK_NONBLOCK | SOCK_CLOEXEC);
if (fd < 0)
break; // EAGAIN: nothing pending (other errors: retried next call).
@@ -133,8 +133,9 @@ void OpsServer::serve(const OpsHandler& handler) {
++refused_;
continue;
}
connections_.push_back({fd, {}, {}, 0, now + connection_deadline});
connections_.push_back({fd, {}, {}, 0, accepted_at + connection_deadline});
}
const auto now = Clock::now(); // Deadlines are judged after the bounded accept work.
for (auto it = connections_.begin(); it != connections_.end();) {
auto& c = *it;
bool done = false;
+154 -1
View File
@@ -171,6 +171,7 @@ void upgrade_across_migration_and_barrier() {
" -v migration_sha_5=$(s 0005_batch_decisions.sql)"
" -v migration_sha_6=$(s 0006_commit_lookup.sql)"
" -v migration_sha_7=$(s 0007_region_clones.sql) -v migration_sha_8=invalid"
" -v migration_sha_9=$(s 0009_serialized_authority.sql)"
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
" > /dev/null 2>&1") != 0);
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
@@ -186,7 +187,7 @@ void upgrade_across_migration_and_barrier() {
CHECK(shell("PGDATABASE=" + database +
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
"tools/migrate-postgres.sh\" > /dev/null") == 0);
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8");
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9");
CHECK(renew_region_lease(*runtime, r.key, *lease, 20000ms, 5s));
confirm(publish_next(20)); // 41..60, now v8.
CHECK(head.head.end_tick == 60);
@@ -395,9 +396,161 @@ void durability_qualification() {
static_cast<unsigned long long>(audit_ms),
static_cast<unsigned long long>(clone_ms));
}
// Review fixes (schema v9): builds, authority and world classification follow one
// lock order. Each case holds a transaction open, proves the contender is blocked
// on a lock in the database, commits, and checks the contender's outcome.
struct Race {
std::string database;
// Commits the holder's open transaction once `contender` is waiting on a lock.
template <typename F> auto run(Postgres& holder, F contender) {
std::optional<decltype(contender())> result;
std::thread thread([&] { result.emplace(contender()); });
auto observer = connect_to(database, env("PGUSER"));
const std::array<std::string, 1> p{database};
bool waiting = false;
for (int i = 0; i < 500 && !waiting; ++i) {
auto rows = observer->query("SELECT count(*)::text FROM pg_catalog.pg_stat_activity "
"WHERE datname = $1 AND wait_event_type = 'Lock'",
p, 5s);
waiting = rows && rows->size() == 1 && (*rows)[0][0].value_or("0") != "0";
if (!waiting)
std::this_thread::sleep_for(20ms);
}
CHECK(holder.query("COMMIT", {}, 5s));
thread.join();
CHECK(waiting);
if (!result.has_value())
throw std::runtime_error("race contender produced no result");
return std::move(*result);
}
};
std::string bytea_of(std::span<const std::uint8_t> bytes) {
constexpr char digits[] = "0123456789abcdef";
std::string out = "\\x";
for (const auto b : bytes) {
out.push_back(digits[b >> 4U]);
out.push_back(digits[b & 0x0FU]);
}
return out;
}
std::string build_literal(std::uint8_t b) {
char out[64];
std::snprintf(out, sizeof out, "decode(repeat('%02x', 32), 'hex')", b);
return out;
}
void concurrent_authority_and_classification() {
const std::string database = "evo_race";
CHECK(shell("createdb " + database + " && PGDATABASE=" + database +
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
"tools/migrate-postgres.sh\" > /dev/null") == 0);
auto owner_db = connect_to(database, env("PGUSER"));
CHECK(owner_db->query("INSERT INTO evolution.retained_builds(build_identity) SELECT "
"decode(repeat(b, 32), 'hex') FROM unnest(ARRAY['c1', 'c2', 'c3', "
"'c4']) AS b",
{}, 5s));
const Sha256Digest c2 = digest_of(0xC2), c3 = digest_of(0xC3), c4 = digest_of(0xC4);
const RegionKey key{7, 1};
auto genesis = genesis_manifest(key, 9, filled<16>(0x61), 50000, c4, 100, filled<32>(0xA0));
CHECK(genesis);
const std::array<std::string, 1> gp{bytea_of(*encode_manifest(*genesis))};
CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", gp, 5s));
Race race{database};
const auto barrier_holding_lease_row = [&](std::uint8_t build) {
CHECK(owner_db->query("BEGIN", {}, 5s));
CHECK(owner_db->query("SELECT 1 FROM evolution.region_leases WHERE world_id = 7 AND "
"region_id = 1 FOR UPDATE",
{}, 5s));
CHECK(scalar(*owner_db, "SELECT evolution.upgrade_barrier(" + build_literal(build) +
")::text") == "0");
};
// Build-less acquisition queued behind the first barrier is refused after it.
barrier_holding_lease_row(0xC1);
auto legacy = race.run(*owner_db, [&] {
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
return legacy_acquire(*runtime, key);
});
CHECK(!legacy && legacy.error() == "EVL06");
// Build-aware acquisition for the build being retired is refused after COMMIT.
barrier_holding_lease_row(0xC2);
auto retired = race.run(*owner_db, [&] {
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
return acquire_region_lease(*runtime, key, c2, 20000ms, 5s);
});
CHECK(!retired && retired.error().kind == LeaseStoreError::unsupported);
CHECK(scalar(*owner_db, "SELECT (holder IS NULL)::text FROM evolution.region_leases WHERE "
"world_id = 7 AND region_id = 1") == "true");
// The new build acquires; the rejected one never held anything to renew.
auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER"));
auto current = acquire_region_lease(*runtime, key, c3, 20000ms, 5s);
CHECK(current);
auto stale = renew_region_lease(*runtime, key, {filled<16>(0x77), 1, 9, 20000ms}, 20000ms, 5s);
CHECK(!stale && stale.error().kind == LeaseStoreError::fenced);
CHECK(release_region_lease(*runtime, key, *current, 5s));
// Two barriers for the last two current builds: exactly one may succeed.
CHECK(owner_db->query("BEGIN", {}, 5s));
CHECK(scalar(*owner_db,
"SELECT evolution.upgrade_barrier(" + build_literal(0xC3) + ")::text") == "0");
auto last = race.run(*owner_db, [&] {
auto other = connect_to(database, env("PGUSER"));
return other->query("SELECT evolution.upgrade_barrier(" + build_literal(0xC4) + ")", {},
5s);
});
CHECK(!last && sqlstate(last.error()) == "22023");
CHECK(scalar(*owner_db, "SELECT count(*)::text FROM evolution.retained_builds WHERE "
"retired_at IS NULL") == "1");
// Classification: a synthetic source cut (absent from this database) and clone
// manifests naming it; the database never reads payload objects.
const auto source_cut = [&](std::uint64_t world, std::uint8_t seed) {
auto m =
genesis_manifest({world, 1}, 9, filled<16>(seed), 50000, c4, 100, filled<32>(0xA1));
CHECK(m);
return *m;
};
const auto clone_sql = [&](Postgres& db, RegionKey target, std::uint64_t source_world,
std::uint8_t seed) {
auto m = clone_manifest(source_cut(source_world, static_cast<std::uint8_t>(seed + 1)),
filled<32>(0xD5), target, filled<16>(seed), 100, filled<32>(0xA2),
{0, {}, 0, {}});
CHECK(m);
const std::array<std::string, 3> p{bytea_of(*encode_manifest(*m)),
std::to_string(source_world), "1"};
return db.query("SELECT evolution.provision_region_clone($1::bytea, $2::evolution.u64, "
"$3::evolution.u64, 0, '\\x'::bytea)",
p, 5s);
};
const auto contender = [&](RegionKey target, std::uint64_t source_world, std::uint8_t seed) {
return [&, target, source_world, seed] {
auto other = connect_to(database, env("PGUSER"));
return clone_sql(*other, target, source_world, seed);
};
};
// Different sources into one target world: exactly one source is accepted.
CHECK(owner_db->query("BEGIN", {}, 5s));
CHECK(clone_sql(*owner_db, {2, 1}, 10, 0x62));
auto mixed = race.run(*owner_db, contender({2, 2}, 11, 0x64));
CHECK(!mixed && sqlstate(mixed.error()) == "22023");
// An original region racing a clone into the same world.
auto original = genesis_manifest({3, 1}, 9, filled<16>(0x66), 50000, c4, 100, filled<32>(0xA0));
CHECK(original);
const std::array<std::string, 1> op{bytea_of(*encode_manifest(*original))};
CHECK(owner_db->query("BEGIN", {}, 5s));
CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", op, 5s));
auto against_original = race.run(*owner_db, contender({3, 2}, 10, 0x68));
CHECK(!against_original && sqlstate(against_original.error()) == "22023");
// Same source, different regions: both succeed.
CHECK(owner_db->query("BEGIN", {}, 5s));
CHECK(clone_sql(*owner_db, {4, 1}, 10, 0x6A));
auto same_source = race.run(*owner_db, contender({4, 2}, 10, 0x6C));
CHECK(same_source);
CHECK(scalar(*owner_db, "SELECT string_agg(world_id || '/' || region_id || '<' || "
"source_world, ',' ORDER BY world_id, region_id) FROM "
"evolution.region_origins") == "2/1<10,4/1<10,4/2<10");
}
} // namespace
void run_region_migration_tests() {
concurrent_authority_and_classification();
durability_qualification();
upgrade_across_migration_and_barrier();
}
+77
View File
@@ -1,9 +1,11 @@
// M7.8 operational endpoints on real evo-server processes (ADR-0066): liveness is
// never authority, readiness follows the lease, metrics are bounded, and the
// responder is bounded against idle, excess and malformed clients.
#include "evo/server/ops_http.hpp"
#include "fixtures/server_process.hpp"
#include <algorithm>
#include <arpa/inet.h>
#include <atomic>
#include <netinet/in.h>
#include <source_location>
#include <sys/socket.h>
@@ -184,9 +186,84 @@ void stale_process_is_live_not_ready() {
current.signal(SIGTERM);
CHECK(stale.wait_exit(20s) == 0 && current.wait_exit(20s) == 0);
}
// Review fix: accept work per serve() is bounded, refused connections included, so
// replenished connection pressure cannot hold the host loop; accepted connections
// are still answered while the pressure continues.
void accept_work_is_bounded_per_serve() {
using evo::server::OpsServer;
auto made = OpsServer::listen("127.0.0.1:0");
CHECK(made);
auto& ops = **made;
const auto handler = [](std::string_view) { return evo::server::OpsResponse{200, "ok\n"}; };
constexpr auto per = OpsServer::max_accepts_per_serve;
std::vector<int> clients;
clients.reserve(64);
for (int i = 0; i < 16; ++i)
clients.push_back(connect_local(ops.port())); // Queued in the listen backlog.
std::this_thread::sleep_for(50ms);
ops.serve(handler);
CHECK(ops.open_connections() == per && ops.refused() == 0);
ops.serve(handler);
CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == 0);
// At the connection cap, each call still refuses at most the budget.
ops.serve(handler);
CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == per);
// The first accepted client is answered while new connections keep arriving.
const std::string request = "GET /livez HTTP/1.1\r\n\r\n";
CHECK(::send(clients[0], request.data(), request.size(), MSG_NOSIGNAL) ==
static_cast<ssize_t>(request.size()));
for (int round = 0; round < 6; ++round) {
clients.push_back(connect_local(ops.port()));
const auto before = ops.refused();
ops.serve(handler);
CHECK(ops.refused() - before <= per);
}
char reply[64] = {};
CHECK(::recv(clients[0], reply, sizeof reply - 1, 0) > 0);
CHECK(std::string_view(reply).starts_with("HTTP/1.1 200"));
for (const int fd : clients)
::close(fd);
}
// A real server under sustained connection churn keeps its lease and readiness.
void connection_pressure_keeps_authority() {
const auto r = provision(54, 0x0D);
Child server{with_ops(r)};
const auto port = ops_port(server);
CHECK(server.wait_for([](const std::string& l) { return l == "ready 1"; }, 15s));
std::atomic<bool> stop{false};
std::thread flood([&] {
while (!stop.load()) {
const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
sockaddr_in address{};
address.sin_family = AF_INET;
address.sin_port = htons(port);
address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (fd >= 0) {
// Refusal or backlog overflow is part of the pressure; ignore it.
[[maybe_unused]] const int connected =
::connect(fd, reinterpret_cast<sockaddr*>(&address), sizeof address);
::close(fd);
}
}
});
// Longer than the fast lease (4 s): renewals must keep happening under pressure.
server.send(input_lines(1, 60, false));
const bool acked = server.wait_line("ack " + request_hex(60) + " 60 ", 20s);
stop = true;
flood.join();
CHECK(acked);
CHECK(!server.printed("ready 0"));
CHECK(get(port, "/readyz").status == 200);
server.signal(SIGTERM);
CHECK(server.wait_exit(20s) == 0);
}
} // namespace
void run_region_ops_tests() {
accept_work_is_bounded_per_serve();
connection_pressure_keeps_authority();
probes_follow_authority_and_stay_bounded();
unavailable_database_is_live_not_ready();
stale_process_is_live_not_ready();
+7 -5
View File
@@ -55,7 +55,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
v2_sha="${v2_sha%% *}"
@@ -76,7 +76,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 8:1,2,3,4,5,6,7,8
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 9:1,2,3,4,5,6,7,8,9
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
@@ -130,7 +130,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
@@ -165,9 +165,11 @@ expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.provision_region_check
expect_audit_failure 'GRANT SELECT ON evolution.region_commits TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commits FROM evo_runtime_test'
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
# Unknown future versions and checksum drift must not be repaired silently.
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=9 WHERE version=8'
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=10 WHERE version=9'
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=9 WHERE version=10"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=9"
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=8 WHERE version=9"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=8"
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=7"
+3 -1
View File
@@ -28,10 +28,12 @@ digest_7="$(sha256sum "$root/migrations/0007_region_clones.sql")"
digest_7="${digest_7%% *}"
digest_8="$(sha256sum "$root/migrations/0008_upgrade_barrier.sql")"
digest_8="${digest_8%% *}"
digest_9="$(sha256sum "$root/migrations/0009_serialized_authority.sql")"
digest_9="${digest_9%% *}"
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
--set="migration_sha_6=$digest_6" --set="migration_sha_7=$digest_7" \
--set="migration_sha_8=$digest_8" \
--set="migration_sha_8=$digest_8" --set="migration_sha_9=$digest_9" \
--set="runtime_role=$runtime_role" \
--file="$root/migrations/apply.sql"