diff --git a/docs/STATUS.md b/docs/STATUS.md index 8be3b7d..0847c37 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -954,3 +954,8 @@ lease: run. Host-loss durability is M8. Next: M8.1, the host inventory and budgets. + +Review fixes (2026-09-29): schema v9 serializes build retirement with acquisition, +leases and provisioning under one lock order. It also serializes clone-world +classification per world. The operations listener's accept work per loop pass is +now bounded. Regression tests and five mutations cover each fix. diff --git a/docs/adr/0066-operations-endpoints.md b/docs/adr/0066-operations-endpoints.md index 6c00ca4..882a9ed 100644 --- a/docs/adr/0066-operations-endpoints.md +++ b/docs/adr/0066-operations-endpoints.md @@ -99,3 +99,15 @@ zlib, CA certificates): - A later gateway must honor readiness and the lease epoch before routing. - Promoting to World0 needs a scanned, signed image and a reviewed alerting policy (M8). + +## Amendment: bounded accept work (2026-09-29 review) +`serve()` used to accept, and close excess connections, until the backlog was +empty. A replenished stream of connections could therefore hold the host loop +without bound. Each call now makes at most `max_accepts_per_serve` (4) accept +attempts, refused ones included. Pending connections wait in the 16-entry listen +backlog, and deadlines are judged after the accept work. + +Tests check the per-call budget, including at the connection cap, and that an +accepted probe is still answered under replenished pressure. A real server keeps +its lease and readiness while a client opens and closes connections continuously +through 60 ticks. diff --git a/docs/adr/0067-region-clones.md b/docs/adr/0067-region-clones.md index ff643d8..b3b4ede 100644 --- a/docs/adr/0067-region-clones.md +++ b/docs/adr/0067-region-clones.md @@ -109,3 +109,18 @@ alone does not prove retention; the audit does. defeats isolation. - **Recording provenance only in a table.** The chain would not commit to its origin, and cross-database clones could not verify it. + +## Amendment: serialized world classification (schema v9, 2026-09-29 review) +The rule "original, or a clone of exactly one source" was enforced by unlocked +existence checks. Two concurrent clones of different sources into different +regions of one world could therefore both commit (reproduced by review). Schema +v9 fixes this: +- Both provisioning functions take a per-world advisory lock, keyed by a hash of + the world ID, after the shared upgrade lock (ADR-0068 amendment). +- They then check the classification. Because the checks run after the lock wait, + they see every committed provisioning of that world. +- Unrelated worlds provision in parallel. A hash collision only adds waiting. + +Regression tests race different-source clones, and an original region against a +clone, into one world, and expect exactly one to succeed. Same-source clones into +different regions both succeed. diff --git a/docs/adr/0068-upgrade-barrier.md b/docs/adr/0068-upgrade-barrier.md index 73fe639..97486eb 100644 --- a/docs/adr/0068-upgrade-barrier.md +++ b/docs/adr/0068-upgrade-barrier.md @@ -94,3 +94,28 @@ The results are in engineering/DURABILITY_QUALIFICATION.md. mid-upgrade before the new build was deployed. - **In-place rules migration of a live region.** It would change physical laws without the clone-and-compare approval that the vision requires. + +## Amendment: serialized retirement (schema v9, 2026-09-29 review) +In v8, acquisition checked eligibility before locking the lease row, without +serializing against the barrier. A request queued behind a barrier could +therefore acquire for the build just retired, then keep renewing (reproduced by +review). Schema v9 fixes this with one lock order: the upgrade lock (advisory, +database-wide), then the world classification lock, then the lease row, then +head and commit rows. +- `upgrade_barrier` takes the upgrade lock exclusively before anything else. This + also serializes barriers, so two operators cannot each retire what they believe + is a non-last build. +- Both acquisition forms, `check_region_lease` (and with it publication, renewal + and release), provisioning and the retired-build trigger take the upgrade lock + shared first. Their eligibility checks therefore run after any concurrent + barrier commits. + +The lock order is the same everywhere, so publication cannot deadlock against the +barrier. Regression tests hold a barrier open, prove the contender is blocked, +commit, and check that a queued acquisition of the retired build is refused (both +forms) and that the last current build cannot be retired by racing barriers. + +A runtime login can still take PostgreSQL advisory locks directly, and so could +delay barriers or acquisitions. It cannot gain authority that way. An owner can +terminate such sessions. Removing that capability needs a pooler or role policy +(M8). diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 1417a26..8000f77 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2534,3 +2534,49 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate The migration test caught this one after the host-test correction. - Not executed: Docker, the Gitea CI, cluster runs, a separate database host, and host or storage loss (M8). + +## M7.6–M7.9 review fixes (schema v9) - 2026-09-29 + +Review `.cache/reviews/M7_6-M7_9_REVIEW.md` found three issues. All three are +fixed: +- **[P1] Build retirement could race acquisition.** Schema v9 applies one + upgrade-lock order across the barrier, acquisitions, lease checks, provisioning + and the retired-build trigger. +- **[P2] Clone-world classification could race.** Provisioning now takes a + per-world lock and checks classification after it. +- **[P2] The operations listener accepted without bound.** Each `serve()` now + makes at most 4 accept attempts. + +Evidence: +- WSL Linux `checks` set (Debug, Release, analysis, ASan/UBSan): CTest 59/59 + each. The format check passed. +- PostgreSQL ASan/UBSan: full suite passed. Analysis and Release were rerun green + after fixing two clang-tidy findings in the new tests. Upgrade paths now end at + 1..9. +- Race tests (database `evo_race`): each holds a transaction open, confirms the + contender is blocked on a lock via `pg_stat_activity`, then commits. + - A build-less acquisition queued behind a barrier is refused (EVL06). + - A build-aware acquisition of the build being retired is refused, and the + region stays unheld. + - The next build acquires. + - Racing barriers for the last two current builds: one returns 22023 and one + build stays current. + - Different-source clones into one world, and an original region racing a + clone: the second is refused (22023). + - Same-source clones into different regions both succeed. + - The final origins are exactly `2/1<10, 4/1<10, 4/2<10`. +- Accept budget: + - With 16 queued connections, successive `serve()` calls give 4 open, then 8 + open with 0 refused, then 8 open with 4 refused. + - Refusals never exceed 4 per call under replenished connections. + - The first accepted probe got 200. + - A real server acknowledged 60 ticks under continuous connect/close churn + without losing readiness. +- Mutations, each reverting one fix (restored and rebuilt). Each was caught: + - an unlocked build-aware acquire; + - an unlocked legacy acquire; + - an unlocked barrier; + - no per-world lock; + - an accept budget of 1000. +- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s. +- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files. diff --git a/docs/infrastructure/POSTGRESQL.md b/docs/infrastructure/POSTGRESQL.md index 9c634cf..30eee1f 100644 --- a/docs/infrastructure/POSTGRESQL.md +++ b/docs/infrastructure/POSTGRESQL.md @@ -164,6 +164,13 @@ To upgrade the build: returns the number of leases fenced, and refuses to retire the last current build. +Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The +barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions, +lease checks, provisioning and the retired-build trigger take it shared, and +provisioning also takes a per-world lock (71420003). A barrier therefore briefly +waits for in-flight commits and blocks new ones. Keep other advisory locks away +from these two keys. + Old images then quarantine (exit 4) instead of serving. Rules and format changes are refused explicitly; there is no converter. diff --git a/include/evo/server/ops_http.hpp b/include/evo/server/ops_http.hpp index 8a68744..f69a26c 100644 --- a/include/evo/server/ops_http.hpp +++ b/include/evo/server/ops_http.hpp @@ -27,6 +27,9 @@ enum class OpsError : std::uint8_t { address, socket }; class OpsServer { public: static constexpr std::size_t max_connections = 8; + // Accept attempts per serve(), refused ones included: sustained connection + // pressure cannot hold the host loop; the rest wait in the listen backlog. + static constexpr std::size_t max_accepts_per_serve = 4; static constexpr std::size_t max_request_bytes = 4096; static constexpr std::size_t max_response_bytes = 16384; static constexpr std::chrono::milliseconds connection_deadline{1000}; @@ -42,13 +45,16 @@ class OpsServer { [[nodiscard]] std::uint16_t port() const noexcept { return port_; } - // Accepts, reads, answers and expires connections without waiting. Connections - // beyond the bound are closed at once; slow or oversized ones are answered or - // dropped without delaying others. + // Accepts at most max_accepts_per_serve connections, then reads, answers and + // expires connections without waiting. Connections beyond the bound are closed + // at once; slow or oversized ones are answered or dropped without delaying others. void serve(const OpsHandler& handler); [[nodiscard]] std::uint64_t refused() const noexcept { return refused_; } + [[nodiscard]] std::size_t open_connections() const noexcept { + return connections_.size(); + } private: struct Connection { diff --git a/migrations/0009_serialized_authority.sql b/migrations/0009_serialized_authority.sql new file mode 100644 index 0000000..2713192 --- /dev/null +++ b/migrations/0009_serialized_authority.sql @@ -0,0 +1,235 @@ +-- M7.9 review fixes (ADR-0067/0068): one locking protocol for authority, build +-- retirement and world classification. Replaces function bodies only. +-- +-- Lock order, everywhere: upgrade lock (advisory 71420002) -> world classification +-- lock (advisory 71420003, hashtext(world)) -> lease row -> head/commit rows. +-- upgrade_barrier takes the upgrade lock exclusively before anything else; every +-- acquisition, lease check (so publication, renewal and release), provisioning and +-- the retired-build trigger take it shared first. A retirement therefore cannot +-- commit between an acquisition's eligibility check and its grant, a commit naming +-- a build cannot straddle its retirement, and concurrent barriers serialize. The +-- per-world lock serializes original/clone classification of one target world +-- while unrelated worlds provision in parallel. Callers use READ COMMITTED +-- statements (as the adapter does), so each check after a wait sees committed rows. + +CREATE OR REPLACE FUNCTION evolution.check_region_lease(p_world evolution.u64, + p_region evolution.u64, p_holder uuid, p_epoch evolution.u64) +RETURNS timestamptz LANGUAGE plpgsql SECURITY INVOKER +SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + l evolution.region_leases%ROWTYPE; + t timestamptz; +BEGIN + IF p_world IS NULL OR p_region IS NULL OR p_holder IS NULL OR p_epoch IS NULL THEN + RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023'; + END IF; + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + SELECT * INTO l FROM evolution.region_leases r + WHERE r.world_id = p_world AND r.region_id = p_region FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'region not provisioned' USING ERRCODE = 'EVL04'; + END IF; + -- Read after the lock wait: a queued stale writer sees current database time. + t := clock_timestamp(); + IF l.holder IS DISTINCT FROM p_holder OR l.epoch <> p_epoch OR l.expires_at <= t THEN + RAISE EXCEPTION 'region lease fenced' USING ERRCODE = 'EVL02'; + END IF; + RETURN t; +END $$; + +CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64, + p_region evolution.u64, p_duration_ms integer, p_build bytea) +RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64, + granted_ms integer) +LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + IF p_build IS NULL OR octet_length(p_build) <> 32 THEN + RAISE EXCEPTION 'invalid lease request' USING ERRCODE = '22023'; + END IF; + -- Eligibility is decided under the upgrade lock and held through the grant. + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b + WHERE b.build_identity = p_build AND b.retired_at IS NULL) THEN + RAISE EXCEPTION 'build not current' USING ERRCODE = 'EVL06'; + END IF; + RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms); +END $$; + +CREATE OR REPLACE FUNCTION evolution.acquire_region_lease(p_world evolution.u64, + p_region evolution.u64, p_duration_ms integer) +RETURNS TABLE (lease_holder uuid, lease_epoch evolution.u64, region_frame evolution.u64, + granted_ms integer) +LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + IF EXISTS (SELECT 1 FROM evolution.retained_builds b WHERE b.retired_at IS NOT NULL) THEN + RAISE EXCEPTION 'build-less acquisition ended by an upgrade barrier' USING ERRCODE = 'EVL06'; + END IF; + RETURN QUERY SELECT * FROM evolution.grant_region_lease(p_world, p_region, p_duration_ms); +END $$; + +CREATE OR REPLACE FUNCTION evolution.refuse_retired_build() RETURNS trigger LANGUAGE plpgsql +SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + -- Normally already held by the inserting function (same order); re-entrant. + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + IF EXISTS (SELECT 1 FROM evolution.retained_builds b + WHERE b.build_identity = NEW.build_identity AND b.retired_at IS NOT NULL) THEN + RAISE EXCEPTION 'build retired from publication' USING ERRCODE = 'EVL06'; + END IF; + RETURN NEW; +END $$; + +CREATE OR REPLACE FUNCTION evolution.upgrade_barrier(p_retire bytea) +RETURNS integer LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + fenced integer; +BEGIN + IF p_retire IS NULL OR octet_length(p_retire) <> 32 THEN + RAISE EXCEPTION 'invalid build identity' USING ERRCODE = '22023'; + END IF; + -- First: waits out every in-flight acquisition, lease check and provisioning, + -- and serializes barriers so the last-current-build check cannot be raced. + PERFORM pg_catalog.pg_advisory_xact_lock(71420002); + PERFORM 1 FROM evolution.retained_builds b WHERE b.build_identity = p_retire FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'build not retained' USING ERRCODE = '22023'; + END IF; + IF NOT EXISTS (SELECT 1 FROM evolution.retained_builds b + WHERE b.build_identity <> p_retire AND b.retired_at IS NULL) THEN + RAISE EXCEPTION 'no current build would remain' USING ERRCODE = '22023'; + END IF; + UPDATE evolution.retained_builds b SET retired_at = coalesce(b.retired_at, clock_timestamp()) + WHERE b.build_identity = p_retire; + UPDATE evolution.region_leases l SET holder = NULL, expires_at = NULL + WHERE l.holder IS NOT NULL; + GET DIAGNOSTICS fenced = ROW_COUNT; + RETURN fenced; +END $$; + +CREATE OR REPLACE FUNCTION evolution.provision_region_checkpoint(p_manifest bytea) +RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest); + d bytea := sha256(p_manifest); +BEGIN + IF g.commit_sequence <> 0 OR g.predecessor_commit IS NOT NULL THEN + RAISE EXCEPTION 'genesis manifest required' USING ERRCODE = '22023'; + END IF; + -- Upgrade lock (shared), then this world's classification lock; both checks + -- below run after the waits, so they see every committed provisioning. + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text)); + IF EXISTS (SELECT 1 FROM evolution.region_origins o WHERE o.world_id = g.world_id) THEN + RAISE EXCEPTION 'world holds clones' USING ERRCODE = '22023'; + END IF; + INSERT INTO evolution.region_leases(world_id, region_id, frame_id) + VALUES (g.world_id, g.region_id, g.frame_id); + INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id, + commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds, + versions, build_identity, payload_bytes, payload_sha256) + VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, 0, 0, 0, + g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes, + g.payload_sha256); + INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest, + end_tick, published_epoch) + VALUES (g.world_id, g.region_id, 0, g.commit_id, d, 0, 0); +END $$; + +-- Owner-only clone genesis. The source may live in another database; when it is +-- present here, the origin must be exactly its committed link. Inherited outcomes +-- are recorded as decisions of sequence 0 without outbox rows: the source already +-- delivered them, and a clone has no production integrations. + +CREATE OR REPLACE FUNCTION evolution.provision_region_clone(p_manifest bytea, p_source_world evolution.u64, + p_source_region evolution.u64, p_source_sequence evolution.u64, p_decisions bytea) +RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest); + d bytea := sha256(p_manifest); + tick numeric; + actor numeric; + outcome numeric; + organism numeric; + last_tick numeric := -1; + last_actor numeric := -1; + actors integer[] := '{}'; +BEGIN + IF g.commit_sequence <> 0 OR g.predecessor_commit IS NULL THEN + RAISE EXCEPTION 'clone genesis manifest required' USING ERRCODE = '22023'; + END IF; + IF p_source_world IS NULL OR p_source_region IS NULL OR p_source_sequence IS NULL + OR p_decisions IS NULL THEN + RAISE EXCEPTION 'invalid clone request' USING ERRCODE = '22023'; + END IF; + IF p_source_world = g.world_id THEN + RAISE EXCEPTION 'clone requires an isolated world' USING ERRCODE = '22023'; + END IF; + -- Upgrade lock (shared), then this world's classification lock; both checks + -- below run after the waits, so they see every committed provisioning. + PERFORM pg_catalog.pg_advisory_xact_lock_shared(71420002); + PERFORM pg_catalog.pg_advisory_xact_lock(71420003, pg_catalog.hashtext(g.world_id::text)); + IF EXISTS (SELECT 1 FROM evolution.region_leases l + WHERE l.world_id = g.world_id + AND NOT EXISTS (SELECT 1 FROM evolution.region_origins o + WHERE o.world_id = l.world_id AND o.region_id = l.region_id)) + OR EXISTS (SELECT 1 FROM evolution.region_origins o + WHERE o.world_id = g.world_id AND o.source_world <> p_source_world) THEN + RAISE EXCEPTION 'target world is not an isolated clone of this source' USING ERRCODE = '22023'; + END IF; + IF EXISTS (SELECT 1 FROM evolution.region_leases l + WHERE l.world_id = p_source_world AND l.region_id = p_source_region) + AND NOT EXISTS (SELECT 1 FROM evolution.region_commits c + WHERE c.world_id = p_source_world AND c.region_id = p_source_region + AND c.commit_sequence = p_source_sequence + AND c.commit_id = g.predecessor_commit + AND c.commit_digest = g.predecessor_digest AND c.end_tick = g.end_tick + AND c.versions = g.versions + AND c.tick_microseconds = g.tick_microseconds + AND substring(c.manifest FROM 13 FOR 4) = substring(p_manifest FROM 13 FOR 4)) THEN + RAISE EXCEPTION 'origin is not the committed source link' USING ERRCODE = '22023'; + END IF; + IF octet_length(p_decisions) <> 24 * g.decision_count + OR (g.decision_count > 0 AND sha256(p_decisions) <> g.decisions_sha256) THEN + RAISE EXCEPTION 'inherited outcomes do not match manifest' USING ERRCODE = '22023'; + END IF; + FOR i IN 0 .. g.decision_count - 1 LOOP + tick := evolution.manifest_uint(p_decisions, 24 * i, 8); + actor := evolution.manifest_uint(p_decisions, 24 * i + 8, 4); + outcome := evolution.manifest_uint(p_decisions, 24 * i + 12, 4); + organism := evolution.manifest_uint(p_decisions, 24 * i + 16, 8); + IF outcome <> 1 OR actor NOT IN (0, 1) OR tick < 1 OR tick > g.end_tick + OR tick < last_tick OR (tick = last_tick AND actor <= last_actor) + OR actor::integer = ANY (actors) OR (g.profile = 1) <> (organism = 0) THEN + RAISE EXCEPTION 'invalid inherited outcome' USING ERRCODE = '22023'; + END IF; + actors := actors || actor::integer; + last_tick := tick; + last_actor := actor; + END LOOP; + INSERT INTO evolution.region_leases(world_id, region_id, frame_id) + VALUES (g.world_id, g.region_id, g.frame_id); + INSERT INTO evolution.region_origins(world_id, region_id, source_world, source_region, + source_sequence, source_commit, source_digest, source_tick) + VALUES (g.world_id, g.region_id, p_source_world, p_source_region, p_source_sequence, + g.predecessor_commit, g.predecessor_digest, g.end_tick); + INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id, + commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds, + versions, build_identity, payload_bytes, payload_sha256, decisions) + VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, g.end_tick, g.end_tick, 0, + g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes, + g.payload_sha256, p_decisions); + INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest, + end_tick, published_epoch) + VALUES (g.world_id, g.region_id, 0, g.commit_id, d, g.end_tick, 0); + FOR i IN 0 .. g.decision_count - 1 LOOP + INSERT INTO evolution.region_decisions(world_id, region_id, outcome, actor, tick, organism, + commit_sequence) + VALUES (g.world_id, g.region_id, 1, evolution.manifest_uint(p_decisions, 24 * i + 8, 4), + evolution.manifest_uint(p_decisions, 24 * i, 8), + evolution.manifest_uint(p_decisions, 24 * i + 16, 8), 0); + END LOOP; +END $$; + + +INSERT INTO evolution.schema_migrations(version, sha256) VALUES (9, :'migration_sha_9'); diff --git a/migrations/apply.sql b/migrations/apply.sql index a93c39c..3b9c1d7 100644 --- a/migrations/apply.sql +++ b/migrations/apply.sql @@ -35,23 +35,34 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') :'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2' || ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' || ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6' - || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8 \gset - \if :at_v8 + || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' AS at_v8, + :'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2' + || ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' + || ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6' + || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' + || ',9:' || :'migration_sha_9' AS at_v9 \gset + \if :at_v9 + \elif :at_v8 + \ir 0009_serialized_authority.sql \elif :at_v7 \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v6 \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v5 \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v4 \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v3 \ir 0004_lineage_profile.sql \ir pre_v5_guard.sql @@ -59,6 +70,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v2 \ir 0003_region_checkpoints.sql \ir 0004_lineage_profile.sql @@ -67,6 +79,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \elif :at_v1 \ir 0002_region_fencing.sql \ir 0003_region_checkpoints.sql @@ -76,6 +89,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \else \echo 'Unknown or changed migration history' DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$; @@ -90,6 +104,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql + \ir 0009_serialized_authority.sql \endif -- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes -- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC diff --git a/src/server/ops_http.cpp b/src/server/ops_http.cpp index dd393c2..d2b84ca 100644 --- a/src/server/ops_http.cpp +++ b/src/server/ops_http.cpp @@ -123,8 +123,8 @@ void OpsServer::respond(Connection& c, const OpsHandler& handler) { } void OpsServer::serve(const OpsHandler& handler) { - const auto now = Clock::now(); - for (;;) { + const auto accepted_at = Clock::now(); + for (std::size_t attempt = 0; attempt < max_accepts_per_serve; ++attempt) { const int fd = ::accept4(fd_, nullptr, nullptr, SOCK_NONBLOCK | SOCK_CLOEXEC); if (fd < 0) break; // EAGAIN: nothing pending (other errors: retried next call). @@ -133,8 +133,9 @@ void OpsServer::serve(const OpsHandler& handler) { ++refused_; continue; } - connections_.push_back({fd, {}, {}, 0, now + connection_deadline}); + connections_.push_back({fd, {}, {}, 0, accepted_at + connection_deadline}); } + const auto now = Clock::now(); // Deadlines are judged after the bounded accept work. for (auto it = connections_.begin(); it != connections_.end();) { auto& c = *it; bool done = false; diff --git a/tests/region_migration_tests.cpp b/tests/region_migration_tests.cpp index db60b74..7dc4f92 100644 --- a/tests/region_migration_tests.cpp +++ b/tests/region_migration_tests.cpp @@ -171,6 +171,7 @@ void upgrade_across_migration_and_barrier() { " -v migration_sha_5=$(s 0005_batch_decisions.sql)" " -v migration_sha_6=$(s 0006_commit_lookup.sql)" " -v migration_sha_7=$(s 0007_region_clones.sql) -v migration_sha_8=invalid" + " -v migration_sha_9=$(s 0009_serialized_authority.sql)" " -v runtime_role=evo_runtime_test -f migrations/apply.sql" " > /dev/null 2>&1") != 0); CHECK(history(*owner_db) == "1,2,3,4,5,6,7"); @@ -186,7 +187,7 @@ void upgrade_across_migration_and_barrier() { CHECK(shell("PGDATABASE=" + database + " EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/" "tools/migrate-postgres.sh\" > /dev/null") == 0); - CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8"); + CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9"); CHECK(renew_region_lease(*runtime, r.key, *lease, 20000ms, 5s)); confirm(publish_next(20)); // 41..60, now v8. CHECK(head.head.end_tick == 60); @@ -395,9 +396,161 @@ void durability_qualification() { static_cast(audit_ms), static_cast(clone_ms)); } +// Review fixes (schema v9): builds, authority and world classification follow one +// lock order. Each case holds a transaction open, proves the contender is blocked +// on a lock in the database, commits, and checks the contender's outcome. +struct Race { + std::string database; + // Commits the holder's open transaction once `contender` is waiting on a lock. + template auto run(Postgres& holder, F contender) { + std::optional result; + std::thread thread([&] { result.emplace(contender()); }); + auto observer = connect_to(database, env("PGUSER")); + const std::array p{database}; + bool waiting = false; + for (int i = 0; i < 500 && !waiting; ++i) { + auto rows = observer->query("SELECT count(*)::text FROM pg_catalog.pg_stat_activity " + "WHERE datname = $1 AND wait_event_type = 'Lock'", + p, 5s); + waiting = rows && rows->size() == 1 && (*rows)[0][0].value_or("0") != "0"; + if (!waiting) + std::this_thread::sleep_for(20ms); + } + CHECK(holder.query("COMMIT", {}, 5s)); + thread.join(); + CHECK(waiting); + if (!result.has_value()) + throw std::runtime_error("race contender produced no result"); + return std::move(*result); + } +}; +std::string bytea_of(std::span bytes) { + constexpr char digits[] = "0123456789abcdef"; + std::string out = "\\x"; + for (const auto b : bytes) { + out.push_back(digits[b >> 4U]); + out.push_back(digits[b & 0x0FU]); + } + return out; +} +std::string build_literal(std::uint8_t b) { + char out[64]; + std::snprintf(out, sizeof out, "decode(repeat('%02x', 32), 'hex')", b); + return out; +} +void concurrent_authority_and_classification() { + const std::string database = "evo_race"; + CHECK(shell("createdb " + database + " && PGDATABASE=" + database + + " EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/" + "tools/migrate-postgres.sh\" > /dev/null") == 0); + auto owner_db = connect_to(database, env("PGUSER")); + CHECK(owner_db->query("INSERT INTO evolution.retained_builds(build_identity) SELECT " + "decode(repeat(b, 32), 'hex') FROM unnest(ARRAY['c1', 'c2', 'c3', " + "'c4']) AS b", + {}, 5s)); + const Sha256Digest c2 = digest_of(0xC2), c3 = digest_of(0xC3), c4 = digest_of(0xC4); + const RegionKey key{7, 1}; + auto genesis = genesis_manifest(key, 9, filled<16>(0x61), 50000, c4, 100, filled<32>(0xA0)); + CHECK(genesis); + const std::array gp{bytea_of(*encode_manifest(*genesis))}; + CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", gp, 5s)); + Race race{database}; + const auto barrier_holding_lease_row = [&](std::uint8_t build) { + CHECK(owner_db->query("BEGIN", {}, 5s)); + CHECK(owner_db->query("SELECT 1 FROM evolution.region_leases WHERE world_id = 7 AND " + "region_id = 1 FOR UPDATE", + {}, 5s)); + CHECK(scalar(*owner_db, "SELECT evolution.upgrade_barrier(" + build_literal(build) + + ")::text") == "0"); + }; + // Build-less acquisition queued behind the first barrier is refused after it. + barrier_holding_lease_row(0xC1); + auto legacy = race.run(*owner_db, [&] { + auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER")); + return legacy_acquire(*runtime, key); + }); + CHECK(!legacy && legacy.error() == "EVL06"); + // Build-aware acquisition for the build being retired is refused after COMMIT. + barrier_holding_lease_row(0xC2); + auto retired = race.run(*owner_db, [&] { + auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER")); + return acquire_region_lease(*runtime, key, c2, 20000ms, 5s); + }); + CHECK(!retired && retired.error().kind == LeaseStoreError::unsupported); + CHECK(scalar(*owner_db, "SELECT (holder IS NULL)::text FROM evolution.region_leases WHERE " + "world_id = 7 AND region_id = 1") == "true"); + // The new build acquires; the rejected one never held anything to renew. + auto runtime = connect_to(database, env("EVO_PG_RUNTIME_USER")); + auto current = acquire_region_lease(*runtime, key, c3, 20000ms, 5s); + CHECK(current); + auto stale = renew_region_lease(*runtime, key, {filled<16>(0x77), 1, 9, 20000ms}, 20000ms, 5s); + CHECK(!stale && stale.error().kind == LeaseStoreError::fenced); + CHECK(release_region_lease(*runtime, key, *current, 5s)); + // Two barriers for the last two current builds: exactly one may succeed. + CHECK(owner_db->query("BEGIN", {}, 5s)); + CHECK(scalar(*owner_db, + "SELECT evolution.upgrade_barrier(" + build_literal(0xC3) + ")::text") == "0"); + auto last = race.run(*owner_db, [&] { + auto other = connect_to(database, env("PGUSER")); + return other->query("SELECT evolution.upgrade_barrier(" + build_literal(0xC4) + ")", {}, + 5s); + }); + CHECK(!last && sqlstate(last.error()) == "22023"); + CHECK(scalar(*owner_db, "SELECT count(*)::text FROM evolution.retained_builds WHERE " + "retired_at IS NULL") == "1"); + + // Classification: a synthetic source cut (absent from this database) and clone + // manifests naming it; the database never reads payload objects. + const auto source_cut = [&](std::uint64_t world, std::uint8_t seed) { + auto m = + genesis_manifest({world, 1}, 9, filled<16>(seed), 50000, c4, 100, filled<32>(0xA1)); + CHECK(m); + return *m; + }; + const auto clone_sql = [&](Postgres& db, RegionKey target, std::uint64_t source_world, + std::uint8_t seed) { + auto m = clone_manifest(source_cut(source_world, static_cast(seed + 1)), + filled<32>(0xD5), target, filled<16>(seed), 100, filled<32>(0xA2), + {0, {}, 0, {}}); + CHECK(m); + const std::array p{bytea_of(*encode_manifest(*m)), + std::to_string(source_world), "1"}; + return db.query("SELECT evolution.provision_region_clone($1::bytea, $2::evolution.u64, " + "$3::evolution.u64, 0, '\\x'::bytea)", + p, 5s); + }; + const auto contender = [&](RegionKey target, std::uint64_t source_world, std::uint8_t seed) { + return [&, target, source_world, seed] { + auto other = connect_to(database, env("PGUSER")); + return clone_sql(*other, target, source_world, seed); + }; + }; + // Different sources into one target world: exactly one source is accepted. + CHECK(owner_db->query("BEGIN", {}, 5s)); + CHECK(clone_sql(*owner_db, {2, 1}, 10, 0x62)); + auto mixed = race.run(*owner_db, contender({2, 2}, 11, 0x64)); + CHECK(!mixed && sqlstate(mixed.error()) == "22023"); + // An original region racing a clone into the same world. + auto original = genesis_manifest({3, 1}, 9, filled<16>(0x66), 50000, c4, 100, filled<32>(0xA0)); + CHECK(original); + const std::array op{bytea_of(*encode_manifest(*original))}; + CHECK(owner_db->query("BEGIN", {}, 5s)); + CHECK(owner_db->query("SELECT evolution.provision_region_checkpoint($1::bytea)", op, 5s)); + auto against_original = race.run(*owner_db, contender({3, 2}, 10, 0x68)); + CHECK(!against_original && sqlstate(against_original.error()) == "22023"); + // Same source, different regions: both succeed. + CHECK(owner_db->query("BEGIN", {}, 5s)); + CHECK(clone_sql(*owner_db, {4, 1}, 10, 0x6A)); + auto same_source = race.run(*owner_db, contender({4, 2}, 10, 0x6C)); + CHECK(same_source); + CHECK(scalar(*owner_db, "SELECT string_agg(world_id || '/' || region_id || '<' || " + "source_world, ',' ORDER BY world_id, region_id) FROM " + "evolution.region_origins") == "2/1<10,4/1<10,4/2<10"); +} } // namespace void run_region_migration_tests() { + concurrent_authority_and_classification(); durability_qualification(); upgrade_across_migration_and_barrier(); } diff --git a/tests/region_ops_tests.cpp b/tests/region_ops_tests.cpp index caee0b1..7d8b5a2 100644 --- a/tests/region_ops_tests.cpp +++ b/tests/region_ops_tests.cpp @@ -1,9 +1,11 @@ // M7.8 operational endpoints on real evo-server processes (ADR-0066): liveness is // never authority, readiness follows the lease, metrics are bounded, and the // responder is bounded against idle, excess and malformed clients. +#include "evo/server/ops_http.hpp" #include "fixtures/server_process.hpp" #include #include +#include #include #include #include @@ -184,9 +186,84 @@ void stale_process_is_live_not_ready() { current.signal(SIGTERM); CHECK(stale.wait_exit(20s) == 0 && current.wait_exit(20s) == 0); } + +// Review fix: accept work per serve() is bounded, refused connections included, so +// replenished connection pressure cannot hold the host loop; accepted connections +// are still answered while the pressure continues. +void accept_work_is_bounded_per_serve() { + using evo::server::OpsServer; + auto made = OpsServer::listen("127.0.0.1:0"); + CHECK(made); + auto& ops = **made; + const auto handler = [](std::string_view) { return evo::server::OpsResponse{200, "ok\n"}; }; + constexpr auto per = OpsServer::max_accepts_per_serve; + std::vector clients; + clients.reserve(64); + for (int i = 0; i < 16; ++i) + clients.push_back(connect_local(ops.port())); // Queued in the listen backlog. + std::this_thread::sleep_for(50ms); + ops.serve(handler); + CHECK(ops.open_connections() == per && ops.refused() == 0); + ops.serve(handler); + CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == 0); + // At the connection cap, each call still refuses at most the budget. + ops.serve(handler); + CHECK(ops.open_connections() == OpsServer::max_connections && ops.refused() == per); + // The first accepted client is answered while new connections keep arriving. + const std::string request = "GET /livez HTTP/1.1\r\n\r\n"; + CHECK(::send(clients[0], request.data(), request.size(), MSG_NOSIGNAL) == + static_cast(request.size())); + for (int round = 0; round < 6; ++round) { + clients.push_back(connect_local(ops.port())); + const auto before = ops.refused(); + ops.serve(handler); + CHECK(ops.refused() - before <= per); + } + char reply[64] = {}; + CHECK(::recv(clients[0], reply, sizeof reply - 1, 0) > 0); + CHECK(std::string_view(reply).starts_with("HTTP/1.1 200")); + for (const int fd : clients) + ::close(fd); +} + +// A real server under sustained connection churn keeps its lease and readiness. +void connection_pressure_keeps_authority() { + const auto r = provision(54, 0x0D); + Child server{with_ops(r)}; + const auto port = ops_port(server); + CHECK(server.wait_for([](const std::string& l) { return l == "ready 1"; }, 15s)); + std::atomic stop{false}; + std::thread flood([&] { + while (!stop.load()) { + const int fd = ::socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); + sockaddr_in address{}; + address.sin_family = AF_INET; + address.sin_port = htons(port); + address.sin_addr.s_addr = htonl(INADDR_LOOPBACK); + if (fd >= 0) { + // Refusal or backlog overflow is part of the pressure; ignore it. + [[maybe_unused]] const int connected = + ::connect(fd, reinterpret_cast(&address), sizeof address); + ::close(fd); + } + } + }); + // Longer than the fast lease (4 s): renewals must keep happening under pressure. + server.send(input_lines(1, 60, false)); + const bool acked = server.wait_line("ack " + request_hex(60) + " 60 ", 20s); + stop = true; + flood.join(); + CHECK(acked); + CHECK(!server.printed("ready 0")); + CHECK(get(port, "/readyz").status == 200); + server.signal(SIGTERM); + CHECK(server.wait_exit(20s) == 0); +} } // namespace void run_region_ops_tests() { + accept_work_is_bounded_per_serve(); + connection_pressure_keeps_authority(); probes_follow_authority_and_stay_bounded(); unavailable_database_is_live_not_ready(); stale_process_is_live_not_ready(); diff --git a/tools/ci/postgres-check.sh b/tools/ci/postgres-check.sh index 10eec1f..0136346 100644 --- a/tools/ci/postgres-check.sh +++ b/tools/ci/postgres-check.sh @@ -55,7 +55,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" -f "$root/migrations/0001_region_bootstrap.sql" > /dev/null test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1 PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8 +test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9 # v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2. v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")" v2_sha="${v2_sha%% *}" @@ -76,7 +76,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 8:1,2,3,4,5,6,7,8 +test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 9:1,2,3,4,5,6,7,8,9 test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1 # Review P1: a v4 database holding a published (possibly terminal) checkpoint has # no durable decisions for it. The v5 upgrade must refuse before changing anything. @@ -130,7 +130,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8 +test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick # Inherited privileges bypass fencing, so every migrate run audits effective rights. @@ -165,9 +165,11 @@ expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.provision_region_check expect_audit_failure 'GRANT SELECT ON evolution.region_commits TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commits FROM evo_runtime_test' bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again. # Unknown future versions and checksum drift must not be repaired silently. -psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=9 WHERE version=8' +psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=10 WHERE version=9' +if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi +psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=9 WHERE version=10" +psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=9" if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi -psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=8 WHERE version=9" psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=8" if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=7" diff --git a/tools/migrate-postgres.sh b/tools/migrate-postgres.sh index 8bf94b9..6ef9e2a 100644 --- a/tools/migrate-postgres.sh +++ b/tools/migrate-postgres.sh @@ -28,10 +28,12 @@ digest_7="$(sha256sum "$root/migrations/0007_region_clones.sql")" digest_7="${digest_7%% *}" digest_8="$(sha256sum "$root/migrations/0008_upgrade_barrier.sql")" digest_8="${digest_8%% *}" +digest_9="$(sha256sum "$root/migrations/0009_serialized_authority.sql")" +digest_9="${digest_9%% *}" psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \ --set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \ --set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \ --set="migration_sha_6=$digest_6" --set="migration_sha_7=$digest_7" \ - --set="migration_sha_8=$digest_8" \ + --set="migration_sha_8=$digest_8" --set="migration_sha_9=$digest_9" \ --set="runtime_role=$runtime_role" \ --file="$root/migrations/apply.sql"