diff --git a/docs/STATUS.md b/docs/STATUS.md index f3770e5..868b8a4 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -1,5 +1,22 @@ # Project status and next steps +## M12.4i durable read-only destination staging - 2026-10-18 + +ADR-0148/schema 48 stages the destination under current authority at its exact +captured head. A verified source capsule and complete destination ghost are +retained for replay, while the source owns all outgoing stock and neither head +moves. Both migration roles pause publication/admission and host readiness +across replacement. Explicit source abort releases both pointers atomically; +stage/abort races and exact retries cannot activate or reopen an aborted ghost. + +Native Release passes 78/78; the private PostgreSQL staging suite, ASan/UBSan, +Clang analysis, format and shell syntax checks pass. The complete private +persistence/process/backup/DR/migration/permission suite passes in 501 s. +[Evidence and limits](engineering/VALIDATION.md) +retain atomic paired publication, its common immutable receipt, commit/abort +races and recovery/activation as the next M12.4 segment. All of M12 remains in +progress. No deployment, automatic movement or live region changed. + ## M12.4h durable source freeze and explicit abort - 2026-10-18 ADR-0147/schema 47 retains exact biological migration preparation, replay capsules diff --git a/docs/adr/0148-durable-cellular-migration-staging.md b/docs/adr/0148-durable-cellular-migration-staging.md new file mode 100644 index 0000000..675a28c --- /dev/null +++ b/docs/adr/0148-durable-cellular-migration-staging.md @@ -0,0 +1,56 @@ +# ADR-0148: Durable destination staging without activation + +## Status + +Accepted and locally qualified for scoped M12.4i implementation, 2026-10-18. +Executed evidence and limits are in [VALIDATION](../engineering/VALIDATION.md). +Paired result publication and activation remain subsequent +work. This selects no route, automatic migration or deployment. + +## Decision + +Extend ADR-0147 preparation with destination-authorized staging at the exact +captured destination head. Lock the two regional lease rows in region order, +check current destination authority and the retained source freeze, and refuse +pending destination work. Retain one immutable staging record and one bounded +destination pending pointer. A region cannot participate in two workflows or +simultaneously be a source and destination. Both biological heads stay unchanged. + +Restore both verified predecessor payloads and replay the exact retained trace. +Copy the source predecessor into the destination object namespace, and upload +and read back the derived destination candidate there. Record its SHA-256 and +length. It is read-only evidence: it is never a committed checkpoint, owned +stock, active population, census entry or permission to run biology. Recovery +replays both cuts from the destination namespace and compares the complete +candidate bytes. Equal aggregate material alone is insufficient. + +New checkpoints, profile changes, intervals and player reservations are refused +at a staged destination. Its host pauses readiness and advancement, retains exact +requests after unknown replies, renews authority and recovers the pause across +replacement. No lease expiry, drain or timeout releases it. Explicit source +abort atomically appends the existing immutable abort and clears both pending +pointers. It requires no destination authority because no stock has moved. +Stage and abort serialize on the source lease row: either staging wins then +abort removes both pointers, or abort wins and new staging is refused. An exact +retry of a previously staged, aborted request returns that retained outcome and +never reopens either pointer. Destination movement before first staging requires +explicit source abort and a new preparation, without silent rebasing. + +Keep all staged objects and predecessor evidence after abort. Backup already +copies immutable namespace objects and shared metadata; this is not cellular +clone/restore qualification. The future paired publisher must recheck both +current fences, heads, staged object bindings and unsettled request, then commit +both results and the common receipt atomically before activation. + +## Alternatives, risks and qualification + +An in-memory ghost disappears on process loss. Activating the uploaded payload +duplicates source stock. Advancing a destination after staging invalidates the +paired candidate. Freeze the whole destination instead; the coarse pause and +extra immutable storage are explicit costs, not a final movement latency policy. + +Require both representations, real player-origin wildlife, full ghost replay, +unchanged heads and stock, conflicting/duplicate requests, pending-work refusal, +stale destination authority, replacement/restart, missing/corrupt/semantically +false ghost evidence, ambiguous replies, process kills, stage/abort races and +host readiness. No coupled ecology, world-cut or multigeneration claim follows. diff --git a/docs/adr/README.md b/docs/adr/README.md index e2fbdfd..7774ce8 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -1,6 +1,8 @@ # Architecture decision records -- [ADR-0147: Durable biological preparation and source freeze](0147-durable-cellular-migration-freeze.md) - schema 47 retains exact preparation/capsules and explicit abort; the source stays authoritative and paused across worker replacement. Destination staging and paired activation remain open. +- [ADR-0148: Durable destination staging without activation](0148-durable-cellular-migration-staging.md) - schema 48 freezes the destination at its exact cut, retains a verified read-only ghost and replays it on recovery; source abort releases both pauses. Paired publication/activation remain open. + +- [ADR-0147: Durable biological preparation and source freeze](0147-durable-cellular-migration-freeze.md) - schema 47 retains exact preparation/capsules and explicit abort; the source stays authoritative and paused across worker replacement. Paired activation remains open. - [ADR-0146: Durable boundary-accounting upgrade admission](0146-durable-boundary-accounting-upgrade.md) - schema 46 extends immutable version-upgrade evidence to same-profile bundle 9->10, full upgrade replay, population batch 2 and preserved representation/interval guards; paired migration escrow remains open. diff --git a/docs/architecture/PERSISTENCE_ARCHITECTURE.md b/docs/architecture/PERSISTENCE_ARCHITECTURE.md index af92684..0c22664 100644 --- a/docs/architecture/PERSISTENCE_ARCHITECTURE.md +++ b/docs/architecture/PERSISTENCE_ARCHITECTURE.md @@ -16,8 +16,15 @@ uses both retained manifests and a verified destination capsule in the source's object namespace. Both biological heads stay unchanged; destination inspection is optimistic and grants no destination authority. Source publication and new interval/player reservations are blocked until explicit settlement. See -[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md). Destination staging, -atomic paired result publication and activation remain required. +[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md). + +M12.4i/schema 48 adds destination-authorized staging under ordered endpoint locks +and the exact retained preparation/heads. A verified source capsule and derived +destination ghost stay read-only in the destination namespace. Both roles pause +publication, interval/player admission and host readiness across replacement. +Explicit source abort releases both pointers atomically; stage/abort races cannot +activate or strand stock. See [ADR-0148](../adr/0148-durable-cellular-migration-staging.md). +Atomic paired result publication, its common receipt and activation remain required. M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md) and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not diff --git a/docs/architecture/REGION_DURABILITY.md b/docs/architecture/REGION_DURABILITY.md index 65d4d80..350a10a 100644 --- a/docs/architecture/REGION_DURABILITY.md +++ b/docs/architecture/REGION_DURABILITY.md @@ -70,7 +70,13 @@ destination predecessor capsule in the source namespace allows complete pair replay without consulting destination live state. The owning host pauses readiness, retains unknown requests and never treats timeout/drain as abort. See [the freeze contract](../../schemas/ecology/cellular-migration-freeze-v1.md). -Destination staging, paired commit/abort races and activation remain open. +ADR-0148/schema 48 now stages a verified destination ghost and source replay +capsule under current destination authority and ordered endpoint locks. The +destination head remains unchanged and is durably paused alongside the source. +Recovery verifies the complete ghost without activating it; explicit source abort +releases both pointers atomically. Stage/abort races serialize on source authority. +See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md). +Atomic paired publication, commit/abort races and activation remain open. World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion diff --git a/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md b/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md index 0219549..d803025 100644 --- a/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md +++ b/docs/engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md @@ -140,3 +140,26 @@ and canonically encoded false member/material declarations fail complete replay. See [the contract](../../schemas/ecology/cellular-migration-freeze-v1.md). Destination staging, atomic paired head publication, commit/abort races and activation remain the next segment; no route or automatic migration is enabled. + +## M12.4i durable read-only destination staging + +ADR-0148/schema 48 freezes the destination at its exact captured head under +current destination authority and ordered endpoint locks. The adapter restores +both cuts, replays the retained trace, copies the source capsule into the +destination namespace and uploads/reads back the derived destination ghost. +Its exact digest/length is immutable retry evidence, not an ownership receipt. +Recovery uses those local objects and compares the complete rederived ghost. +Neither biological head moves, and the source still owns all outgoing stock. + +Both local migration roles are exclusive and block new publication, profile +changes, intervals and player reservations. Unknown staging retains exact bytes; +the owning destination pauses readiness/advancement through worker replacement. +Source abort serializes with staging and releases both pointers atomically. +Aborted-stage retries return their old outcome and never reopen a pause. +Missing/corrupt evidence and real but semantically false objects fail closed. +See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md). + +Atomic paired publication/common receipt, commit/abort races, recovery/activation +and linked ecology remain subsequent work. The ghost cannot run biology or +contribute owned census/extinction stock. No biological format, rule, route, +automatic movement, live region or deployment changes. diff --git a/docs/engineering/M12_REMAINING_GATES.md b/docs/engineering/M12_REMAINING_GATES.md index f3515c4..ee1d09b 100644 --- a/docs/engineering/M12_REMAINING_GATES.md +++ b/docs/engineering/M12_REMAINING_GATES.md @@ -64,9 +64,12 @@ or close source-freeze/destination-activation and escrow qualification. ADR-0147/schema 47 now adds durable source preparation/freeze, retained replay capsules, explicit abort and host pause/recovery. Source stock remains owned at its exact head through worker replacement; no timeout or drain releases it. -Destination staging, atomic paired commit, commit/abort races and activation -remain open. The destination head is only checked optimistically at preparation -and must be rechecked before any future stage/commit. +ADR-0148/schema 48 now supplies destination-authorized staging at its exact +captured head, verified read-only ghost/capsule retention and full destination +recovery. Both endpoints stay paused; explicit source abort atomically releases +both pointers and races safely with staging. Atomic paired commit, its common +receipt, commit/abort races and activation remain open. Preparation remains an +optimistic destination observation; first staging rechecks both captured heads. The transfer tests also expose the current rules-8 maturation gap: buds retain their subadult realization while reproduction requires a fully realized parent. Finite extra food alone does not close repeated wildlife generations. A versioned diff --git a/docs/engineering/TEST_STRATEGY.md b/docs/engineering/TEST_STRATEGY.md index eb12f0d..e2ae94a 100644 --- a/docs/engineering/TEST_STRATEGY.md +++ b/docs/engineering/TEST_STRATEGY.md @@ -32,6 +32,19 @@ the C++ decoder. The metric has one fixed unlabelled series and remains bounded. These are preparation/abort checks, not proof of destination activation or atomic paired publication; full historical persistence regression is also required. +M12.4i's private staging matrix covers both bundle-10 representations and actual +player-origin wildlife. It requires unchanged biological heads, complete ghost +replay, exact/conflicting retries, refusal of moved destination heads and pending +intervals, publication/profile/life-reservation guards and exclusive endpoint +roles. Stale destination epochs fail; replacement retains the read-only ghost. +Missing source capsules, missing/same-length corrupt ghosts and valid but false +ghost objects fail recovery. Real lock-timeout ambiguity, SIGKILL before/after +stage commit and independent-connection stage/abort races preserve one outcome. +The owning destination host retains exact uncertain work across restart and +resumes only its original biology after confirmed source abort. Full schema, +permission, backup/DR and historical regression remain required. These tests do +not qualify paired commit, destination activation or linked ecological movement. + Tests defend contracts, not directory coverage. All checks remain active in Release builds. Current proof has no actual network, biology or distributed recovery, so those categories below are milestone gates rather than passing diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 254482a..f247795 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -1,5 +1,53 @@ # Validation report +## 2026-10-18 - M12.4i durable read-only destination staging + +ADR-0148/schema 48 retains a destination-authorized staging record and bounded +pending pointer at the exact captured head. The adapter verifies both cuts, +replays the trace and retains a source capsule plus verified destination ghost +in the destination namespace. The source owns all outgoing stock; neither head +moves. Source abort releases both pending pointers atomically, and exact stage +retries retain their original outcome. Both migration roles are exclusive. + +Executed locally: + +- Native MSVC Release **78/78 passed**, 12.62 s. +- Actual GCC 14 optional server/persistence targets rebuilt; final private + PostgreSQL population/staging suite passed, **11 s** (8 s test phase). +- Linux Clang 19 ASan/UBSan private population/staging suite passed, **33 s** + (30 s test phase), with no sanitizer reports. +- Explicit clang-tidy 19 on the adapter, owning host and PostgreSQL population + tests passed with no owned warnings. Repository clang-format 19 and both + changed shell scripts' syntax checks passed. +- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**, + **501 s**: C++ integration/process/load/history (426 s), life reservations, + backup/restore, host-loss DR, upgrade, migration history/rollback, runtime + permission audit and checksum/future-version rejection. The detached runner + recorded its completion code. + +Both representations use real funded player-origin wildlife and unchanged cuts. +Cases cover exact/conflicting retries, a moved destination, pending destination +intervals, source/destination role exclusivity, blocked publication/profile/life +reservations and stale destination authority. Replacement epochs retain staging. +Missing source capsules, missing and same-length corrupt ghosts fail strong +binding. An existing valid object with false destination biology fails complete +replay. Real lock-timeout ambiguity retains exact requests; SIGKILL before/after +stage commit leaves absence/a retained pause respectively. Independent database +connections race stage against source abort without stranded destination work. +The owning destination host pauses on unknown staging, survives drain/recreation +without adopting the ghost and resumes its identical original state after source +abort. The existing fixed freeze metric covers either role without new labels. + +Limits: this is staging evidence, not a migration result or activation receipt. +Atomic paired publication, its common immutable outcome, commit/abort races and +recovery/activation remain open. Backup retains all immutable namespace objects; +existing backup/DR fixtures do not qualify cellular cloning or a new cellular +restore procedure. No elapsed-day, linked-geometry or multigeneration claim +follows. No live database, World0, deployment, dependency, biological rule, format +or golden changed. The earlier captured `Build_Errors.txt` and unrelated local +work remain untracked; remote CI confirmation of the previous runner repair +remains separate. + ## 2026-10-18 - M12.4h durable migration source freeze and explicit abort ADR-0147/schema 47 retains exact source-authorized preparation, both predecessor diff --git a/include/evo/persistence/cellular_migration_store.hpp b/include/evo/persistence/cellular_migration_store.hpp index 51bfca3..81426d8 100644 --- a/include/evo/persistence/cellular_migration_store.hpp +++ b/include/evo/persistence/cellular_migration_store.hpp @@ -42,4 +42,36 @@ abort_cellular_migration_freeze(Postgres&, RegionKey, const LeaseGrant&, const U [[nodiscard]] std::expected, RecoveryFailure> load_cellular_migration_freeze(Postgres&, const FilesystemObjectStore&, RegionKey, const LeaseGrant&, std::chrono::milliseconds timeout); + +struct CellularMigrationStagePrepared { + RegionKey key{}; // Destination region. + std::vector trace; + VerifiedObject ghost; +}; +struct CellularMigrationStage { + CellularMigrationFreeze freeze; + VerifiedObject ghost; // Read-only evidence, never an ownership receipt. +}; +struct CellularMigrationStageFailure { + PublicationError kind{}; + LeaseStoreFailure store{}; + ObjectError object{}; + CellularMigrationStagePrepared prepared; +}; +// Replay the retained pair, retain a source capsule and verified destination +// ghost in the destination namespace, then freeze its unchanged exact head. +[[nodiscard]] std::expected +stage_cellular_migration(Postgres&, const FilesystemObjectStore& source_objects, + const FilesystemObjectStore& destination_objects, + const LeaseGrant& destination_lease, const CommittedCheckpoint& source, + const CommittedCheckpoint& destination, + const server::CellularMigrationTrace&, std::chrono::milliseconds timeout, + const PublishHooks& hooks = {}); +[[nodiscard]] std::expected +retry_cellular_migration_stage(Postgres&, const LeaseGrant&, const CellularMigrationStagePrepared&, + std::chrono::milliseconds timeout); +// Fresh destination authority; complete local capsule and ghost replay. +[[nodiscard]] std::expected, RecoveryFailure> +load_cellular_migration_stage(Postgres&, const FilesystemObjectStore&, RegionKey, const LeaseGrant&, + std::chrono::milliseconds timeout); } // namespace evo::persistence diff --git a/include/evo/server/cellular_region_host.hpp b/include/evo/server/cellular_region_host.hpp index 2c33de3..8de3230 100644 --- a/include/evo/server/cellular_region_host.hpp +++ b/include/evo/server/cellular_region_host.hpp @@ -100,6 +100,12 @@ class CellularRegionHost { const CellularMigrationRequest&); [[nodiscard]] std::expected abort_migration_freeze(const persistence::Uuid& request); + // Explicit destination staging. The ghost stays read-only until a future + // paired publication receipt; source abort releases this pause. + [[nodiscard]] std::expected + stage_migration(const persistence::CommittedCheckpoint& source, + const persistence::FilesystemObjectStore& source_objects, + const CellularMigrationTrace&); // Explicit no-tick fenced transition. Uncertain publication pauses the host. [[nodiscard]] std::expected set_population_mode(bool enabled); [[nodiscard]] bool population_mode() const noexcept; @@ -228,6 +234,7 @@ class CellularRegionHost { bool migration_reload_{}, migration_frozen_{}; persistence::LeaseClock::time_point migration_poll_at_{}; std::optional migration_request_; + std::optional migration_stage_request_; std::optional migration_abort_; bool interval_reload_{}; std::optional admitted_interval_; diff --git a/migrations/0048_cellular_migration_staging.sql b/migrations/0048_cellular_migration_staging.sql new file mode 100644 index 0000000..f733547 --- /dev/null +++ b/migrations/0048_cellular_migration_staging.sql @@ -0,0 +1,140 @@ +-- ADR-0148: a staged destination is frozen evidence, never active ownership. +CREATE TABLE evolution.cellular_migration_stages ( + request_id uuid PRIMARY KEY REFERENCES evolution.cellular_migration_preparations, + ghost_sha256 bytea NOT NULL CHECK (octet_length(ghost_sha256)=32 AND ghost_sha256<>decode(repeat('00',32),'hex')), + ghost_bytes evolution.u64 NOT NULL CHECK (ghost_bytes BETWEEN 1 AND 16781400), + holder uuid NOT NULL, epoch evolution.u64 NOT NULL CHECK (epoch>0) +); +CREATE TABLE evolution.cellular_migration_destination_pending ( + world_id evolution.u64 NOT NULL, region_id evolution.u64 NOT NULL, + request_id uuid UNIQUE NOT NULL REFERENCES evolution.cellular_migration_stages, + PRIMARY KEY(world_id,region_id) +); +REVOKE ALL ON evolution.cellular_migration_stages,evolution.cellular_migration_destination_pending FROM PUBLIC; +CREATE TRIGGER cellular_migration_stages_immutable BEFORE UPDATE OR DELETE ON evolution.cellular_migration_stages + FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite(); +CREATE TRIGGER cellular_migration_stages_no_truncate BEFORE TRUNCATE ON evolution.cellular_migration_stages + FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite(); + +CREATE FUNCTION evolution.cellular_migration_stage_record(w evolution.u64,r evolution.u64,id uuid) +RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64) +LANGUAGE sql STABLE SET search_path = pg_catalog, pg_temp AS $$ + SELECT p.trace,s.manifest,d.manifest,EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=p.request_id), + g.ghost_sha256,g.ghost_bytes + FROM evolution.cellular_migration_preparations p JOIN evolution.cellular_migration_stages g USING(request_id) + JOIN evolution.region_commits s ON s.commit_id=p.source_commit + JOIN evolution.region_commits d ON d.commit_id=p.destination_commit + WHERE p.world_id=w AND p.destination_region=r AND p.request_id=id +$$; +REVOKE ALL ON FUNCTION evolution.cellular_migration_stage_record(evolution.u64,evolution.u64,uuid) FROM PUBLIC; + +CREATE FUNCTION evolution.stage_cellular_migration(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,t bytea,digest bytea,size evolution.u64) +RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64) +LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$ +DECLARE id uuid; p evolution.cellular_migration_preparations%ROWTYPE; + old evolution.cellular_migration_stages%ROWTYPE; +BEGIN + PERFORM evolution.check_cellular_migration_trace(t); + IF w IS NULL OR r IS NULL OR evolution.manifest_uint(t,16,8)<>w OR evolution.manifest_uint(t,136,8)<>r + OR digest IS NULL OR octet_length(digest)<>32 OR digest=decode(repeat('00',32),'hex') + OR size IS NULL OR size NOT BETWEEN 1 AND (CASE evolution.manifest_uint(t,12,4) WHEN 1 THEN 16777216 ELSE 16781400 END) THEN + RAISE EXCEPTION 'invalid destination stage' USING ERRCODE='22023'; + END IF; + id := encode(substring(t FROM 49 FOR 16),'hex')::uuid; + SELECT * INTO p FROM evolution.cellular_migration_preparations x WHERE x.request_id=id; + IF NOT FOUND OR p.world_id<>w OR p.destination_region<>r OR p.trace<>t THEN + RAISE EXCEPTION 'stage differs from retained preparation' USING ERRCODE='EVL07'; + END IF; + -- Shared order for future two-owner work. Source abort serializes on its row. + PERFORM 1 FROM evolution.region_leases x WHERE x.world_id=w AND x.region_id IN (p.source_region,r) + ORDER BY x.region_id FOR UPDATE; + PERFORM evolution.check_region_lease(w,r,holder,epoch); + SELECT * INTO old FROM evolution.cellular_migration_stages x WHERE x.request_id=id; + IF FOUND THEN + IF old.ghost_sha256<>digest OR old.ghost_bytes<>size THEN + RAISE EXCEPTION 'stage request reused' USING ERRCODE='EVL07'; + END IF; + RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id); + RETURN; + END IF; + IF EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=id) + OR NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=p.source_region AND x.request_id=id) + OR EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=r) + OR EXISTS(SELECT 1 FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.region_id=r) + OR EXISTS(SELECT 1 FROM evolution.population_interval_pending x WHERE x.world_id=w AND x.region_id=r) + OR EXISTS(SELECT 1 FROM evolution.cellular_mutation_reservations x WHERE x.world_id=w AND x.region_id=r + AND NOT EXISTS(SELECT 1 FROM evolution.cellular_mutation_action_publications a + WHERE a.world_id=x.world_id AND a.account_id=x.account_id AND a.action_id=x.action_id)) + OR EXISTS(SELECT 1 FROM evolution.life_reservations x WHERE x.world_id=w AND x.region_id=r AND x.rejection=0 AND x.ended_commit IS NULL + AND NOT EXISTS(SELECT 1 FROM evolution.primitive_life_bindings b + WHERE b.world_id=x.world_id AND b.account_id=x.account_id AND b.request_id=x.request_id)) THEN + RAISE EXCEPTION 'migration endpoint has pending work or was aborted' USING ERRCODE='EVL18'; + END IF; + IF NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=p.source_region AND h.commit_id=p.source_commit + AND h.digest=substring(t FROM 89 FOR 32) AND h.end_tick=evolution.manifest_uint(t,32,8)) + OR NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=r AND h.commit_id=p.destination_commit + AND h.digest=substring(t FROM 161 FOR 32) AND h.end_tick=evolution.manifest_uint(t,32,8)) THEN + RAISE EXCEPTION 'staged migration head moved' USING ERRCODE='EVL03'; + END IF; + INSERT INTO evolution.cellular_migration_stages VALUES(id,digest,size,holder,epoch); + INSERT INTO evolution.cellular_migration_destination_pending VALUES(w,r,id); + RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id); +END $$; +REVOKE ALL ON FUNCTION evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64) TO :"runtime_role"; + +CREATE FUNCTION evolution.load_cellular_migration_stage(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64) +RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64) +LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$ +DECLARE id uuid; initial_id uuid; src evolution.u64; +BEGIN + SELECT x.request_id,p.source_region INTO id,src FROM evolution.cellular_migration_destination_pending x + JOIN evolution.cellular_migration_preparations p USING(request_id) WHERE x.world_id=w AND x.region_id=r; + IF FOUND THEN + initial_id := id; + PERFORM 1 FROM evolution.region_leases x WHERE x.world_id=w AND x.region_id IN (src,r) ORDER BY x.region_id FOR UPDATE; + END IF; + PERFORM evolution.check_region_lease(w,r,holder,epoch); + -- Abort may have won while we waited. Never turn its completed release into corruption. + SELECT x.request_id INTO id FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.region_id=r; + IF NOT FOUND THEN RETURN; END IF; + IF initial_id IS DISTINCT FROM id THEN + RAISE EXCEPTION 'destination workflow changed while waiting' USING ERRCODE='40001'; + END IF; + IF NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_preparations p JOIN evolution.region_heads h + ON h.world_id=p.world_id AND h.region_id=p.destination_region + WHERE p.request_id=id AND p.world_id=w AND p.destination_region=r AND h.commit_id=p.destination_commit + AND h.digest=substring(p.trace FROM 161 FOR 32) AND h.end_tick=evolution.manifest_uint(p.trace,32,8) + AND EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=p.source_region AND x.request_id=id) + AND NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=id)) THEN + RAISE EXCEPTION 'migration destination freeze inconsistent' USING ERRCODE='XX000'; + END IF; + RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id); +END $$; +REVOKE ALL ON FUNCTION evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64) TO :"runtime_role"; + +ALTER FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) RENAME TO abort_cellular_migration_freeze_v47; +REVOKE ALL ON FUNCTION evolution.abort_cellular_migration_freeze_v47(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) FROM :"runtime_role",PUBLIC; +CREATE FUNCTION evolution.abort_cellular_migration_freeze(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,id uuid) +RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean) +LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + RETURN QUERY SELECT * FROM evolution.abort_cellular_migration_freeze_v47(w,r,holder,epoch,id); + DELETE FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.request_id=id; +END $$; +REVOKE ALL ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) TO :"runtime_role"; + +CREATE OR REPLACE FUNCTION evolution.refuse_frozen_cellular_work() RETURNS trigger +LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$ +BEGIN + IF EXISTS(SELECT 1 FROM evolution.cellular_migration_pending p WHERE p.world_id=NEW.world_id AND p.region_id=NEW.region_id) + OR EXISTS(SELECT 1 FROM evolution.cellular_migration_destination_pending p WHERE p.world_id=NEW.world_id AND p.region_id=NEW.region_id) THEN + RAISE EXCEPTION 'cellular endpoint frozen for migration' USING ERRCODE='EVL18'; + END IF; + RETURN NEW; +END $$; +CREATE TRIGGER source_migration_endpoint_exclusive BEFORE INSERT ON evolution.cellular_migration_pending + FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work(); +INSERT INTO evolution.schema_migrations(version,sha256) VALUES(48,:'migration_sha_48'); diff --git a/migrations/apply.sql b/migrations/apply.sql index e930b33..d4d866f 100644 --- a/migrations/apply.sql +++ b/migrations/apply.sql @@ -123,6 +123,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') SELECT :'history_v44' || ',45:' || :'migration_sha_45' AS history_v45 \gset SELECT :'history_v45' || ',46:' || :'migration_sha_46' AS history_v46 \gset SELECT :'history_v46' || ',47:' || :'migration_sha_47' AS history_v47 \gset + SELECT :'history_v47' || ',48:' || :'migration_sha_48' AS history_v48 \gset SELECT :'history' = :'history_v22' AS at_v22, :'history' = :'history_v23' AS at_v23, :'history' = :'history_v24' AS at_v24, @@ -148,8 +149,10 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') :'history' = :'history_v44' AS at_v44, :'history' = :'history_v45' AS at_v45, :'history' = :'history_v46' AS at_v46, - :'history' = :'history_v47' AS at_v47 \gset - \if :at_v47 + :'history' = :'history_v47' AS at_v47, + :'history' = :'history_v48' AS at_v48 \gset + \if :at_v48 + \elif :at_v47 \elif :at_v46 \elif :at_v45 \elif :at_v44 @@ -537,7 +540,8 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24, coalesce(max(version), 0) = 44 AS at_v44, coalesce(max(version), 0) = 45 AS at_v45, coalesce(max(version), 0) = 46 AS at_v46, - coalesce(max(version), 0) = 47 AS at_v47 + coalesce(max(version), 0) = 47 AS at_v47, + coalesce(max(version), 0) = 48 AS at_v48 FROM evolution.schema_migrations \gset \if :at_v24 \ir 0025_cellular_v2_bundle.sql @@ -753,6 +757,7 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24, \elif :at_v45 \elif :at_v46 \elif :at_v47 +\elif :at_v48 \else \echo 'Unknown or changed migration history after v43' DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$; @@ -773,6 +778,10 @@ SELECT coalesce(max(version), 0) = 46 AS needs_v47 FROM evolution.schema_migrati \if :needs_v47 \ir 0047_cellular_migration_freeze.sql \endif +SELECT coalesce(max(version), 0) = 47 AS needs_v48 FROM evolution.schema_migrations \gset +\if :needs_v48 + \ir 0048_cellular_migration_staging.sql +\endif -- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes -- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC -- grants or SET-able memberships would bypass the fenced functions. Checks the @@ -792,6 +801,8 @@ BEGIN UNION SELECT m.member FROM pg_catalog.pg_auth_members m JOIN closure c ON m.roleid = c.oid ), fenced(oid) AS (VALUES + ('evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64)'::regprocedure::oid), + ('evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid), ('evolution.prepare_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,bytea)'::regprocedure::oid), ('evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid)'::regprocedure::oid), ('evolution.load_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid), diff --git a/schemas/ecology/cellular-migration-freeze-v1.md b/schemas/ecology/cellular-migration-freeze-v1.md index b851992..fc1b319 100644 --- a/schemas/ecology/cellular-migration-freeze-v1.md +++ b/schemas/ecology/cellular-migration-freeze-v1.md @@ -59,6 +59,8 @@ abort. The current backup copies all immutable namespace objects and the shared database snapshot; no GC or pruning is authorized. This does not qualify cellular history cloning/restore beyond the existing supported fixtures. -Destination ghost staging, atomic two-head publication, commit/abort races and -activation remain subsequent work. This segment adds no automatic migration, +Schema 48/ADR-0148 now adds [destination staging](cellular-migration-stage-v1.md) +and atomic release of both pending pointers on source abort. Atomic two-head +publication, commit/abort races and activation remain subsequent work. +This segment adds no automatic migration, atlas, cross-edge movement, environmental coupling or census/extinction rule. diff --git a/schemas/ecology/cellular-migration-stage-v1.md b/schemas/ecology/cellular-migration-stage-v1.md new file mode 100644 index 0000000..034fa43 --- /dev/null +++ b/schemas/ecology/cellular-migration-stage-v1.md @@ -0,0 +1,54 @@ +# Biological destination staging + +Authority: ADR-0148, SQL schema 48. This extends +[source preparation](cellular-migration-freeze-v1.md) with destination-authorized +read-only evidence. Biological/checkpoint/trace versions retain their meanings. +There is no new active checkpoint profile or migration result receipt. + +The caller supplies the exact retained migration trace, both predecessor +manifests and current destination authority. The adapter verifies manifests and +payload SHA-256/length, replays the complete pair, copies the source predecessor +into the destination namespace and uploads/reads back the derived destination +candidate. Its content address and byte length form the exact retry input with +the canonical trace. Unpublished uploads remain retained orphans on failure. + +SQL independently checks the retained preparation's exact trace, current +destination lease, both exact heads and source pending pointer. Both regional +lease rows are locked in increasing region order. Destination pending intervals, +unpublished mutation choices, successful outstanding life reservations and any +other local migration role block first staging. Moving the destination before +staging refuses the request; no implicit rebase is performed. + +Schema 48 retains immutable `cellular_migration_stages`, keyed by the preparation +request UUID, with ghost SHA-256, byte length and staging holder/epoch. One +`cellular_migration_destination_pending` pointer per region bounds active lookup. +The source/destination pending roles are exclusive. Existing indexed publication +and interval/player-reservation guards cover either role. + +The source owns all outgoing organisms and material until a future paired +commit. The staged candidate owns nothing. Neither region head changes. No tick, +feeding, passive healing, movement, birth or death is performed by staging. In +particular, a ghost cannot supply an owned census or terminal extinction count. + +Destination recovery uses its own namespace's two predecessor capsules and ghost. +It rederives the whole pair and compares the complete destination bytes, including +membership, policy, biology and history. Missing or same-length corrupt objects +fail strong binding; a valid object with false biology fails semantic replay. +There is no fallback to the live source or an older head. + +Both first staging and replacement pause host readiness/advancement. Unknown +replies retain the exact trace/content address; reconciliation keeps the same +request and polls paused work at most once per second. Normal stepping has no +per-tick migration query. The existing fixed unlabelled migration-freeze metric +covers either role; no request/organism labels are introduced. + +Explicit current-source abort appends the existing immutable outcome and clears +both pointers in the same transaction. It serializes with staging on the source +lease row. If abort wins first, new staging is refused. If stage wins first, +abort releases both pauses. Exact retries of a retained stage after abort return +`aborted` and do not reopen either role. Neither expiry, drain nor timeout implies +abort. Keep staged objects/capsules and journals after abort; no GC is authorized. + +Atomic paired result publication, its immutable common receipt, commit/abort +races, recovery/activation and linked ecological geometry remain subsequent +work. This segment enables no automatic movement, route, atlas or deployment. diff --git a/src/persistence/cellular_migration_store.cpp b/src/persistence/cellular_migration_store.cpp index f04b4ec..07eb1be 100644 --- a/src/persistence/cellular_migration_store.cpp +++ b/src/persistence/cellular_migration_store.cpp @@ -252,4 +252,179 @@ load_cellular_migration_freeze(Postgres& db, const FilesystemObjectStore& object return fail(RecoveryError::history); return result; } + +namespace { +constexpr auto stage_columns = "encode(trace,'hex'),encode(source_manifest,'hex')," + "encode(destination_manifest,'hex'),aborted::text," + "encode(ghost_sha256,'hex'),ghost_bytes::text"; +struct StageRecord { + Record cut; + VerifiedObject ghost; +}; +std::optional parse_stage(const PgRows::value_type& row) { + if (row.size() != 6) + return std::nullopt; + auto cut = parse(PgRows::value_type(row.begin(), row.begin() + 4)); + const auto digest = unhex(row[4].value_or(""), 32); + const auto size = row[5].value_or(""); + std::uint64_t bytes{}; + const auto [end, error] = std::from_chars(size.data(), size.data() + size.size(), bytes); + if (!cut || !digest || digest->size() != 32 || error != std::errc{} || + end != size.data() + size.size() || bytes == 0) + return std::nullopt; + VerifiedObject ghost{{}, bytes}; + std::ranges::copy(*digest, ghost.sha256.begin()); + return StageRecord{std::move(*cut), ghost}; +} +std::optional checked_stage(const StageRecord& record, RegionKey key, + const LeaseGrant& lease) { + const auto trace = server::decode_migration_trace(record.cut.trace); + if (!trace || key != RegionKey{trace->world, trace->request.destination.region}) + return std::nullopt; + const auto freeze = + checked_record(record.cut, {trace->world, trace->request.source.region}, lease); + const auto destination = decode_manifest(record.cut.destination); + if (!freeze || !destination || record.ghost.sha256 == Sha256Digest{} || + record.ghost.bytes > max_payload_bytes(destination->profile)) + return std::nullopt; + return CellularMigrationStage{*freeze, record.ghost}; +} +std::optional> +ghost_candidate(const CheckpointManifest& s, const CheckpointManifest& d, + std::span source, std::span destination, + const server::CellularMigrationTrace& trace) { + if (s.profile == CheckpointProfile::cellular_lives_v1) { + auto src = restore_cellular_payload(source, s), + dst = restore_cellular_payload(destination, d); + if (!src || !dst) + return std::nullopt; + auto pair = server::replay_migration_trace(**src, **dst, trace.request, trace); + if (!pair) + return std::nullopt; + return pair->destination->snapshot(); + } + auto src = restore_population_payload(source, s), + dst = restore_population_payload(destination, d); + if (!src || !dst) + return std::nullopt; + auto pair = server::replay_migration_trace(**src, **dst, trace.request, trace); + if (!pair) + return std::nullopt; + return pair->destination->snapshot(); +} +CellularMigrationStageFailure stage_failed(PublicationError kind, LeaseStoreFailure store = {}, + ObjectError object = {}, + CellularMigrationStagePrepared prepared = {}) { + return {kind, store, object, std::move(prepared)}; +} +} // namespace + +std::expected +retry_cellular_migration_stage(Postgres& db, const LeaseGrant& lease, + const CellularMigrationStagePrepared& prepared, + milliseconds timeout) { + const auto trace = server::decode_migration_trace(prepared.trace); + if (!trace || prepared.key != RegionKey{trace->world, trace->request.destination.region} || + lease.frame != trace->frame || prepared.ghost.sha256 == Sha256Digest{} || + prepared.ghost.bytes == 0) + return std::unexpected(stage_failed(PublicationError::candidate)); + const std::array args{std::to_string(prepared.key.world), + std::to_string(prepared.key.region), + uuid(lease.holder), + std::to_string(lease.epoch), + "\\x" + hex(prepared.trace), + "\\x" + hex(prepared.ghost.sha256), + std::to_string(prepared.ghost.bytes)}; + auto rows = db.query(std::string("SELECT ") + stage_columns + + " FROM evolution.stage_cellular_migration($1::evolution.u64," + "$2::evolution.u64,$3::uuid,$4::evolution.u64,$5::bytea,$6::bytea," + "$7::evolution.u64)", + args, timeout); + if (!rows) { + const auto store = failure(rows.error()); + return std::unexpected(stage_failed(store.kind == LeaseStoreError::unknown + ? PublicationError::unknown + : PublicationError::rejected, + store, {}, prepared)); + } + const auto record = rows->size() == 1 ? parse_stage(rows->front()) : std::nullopt; + const auto result = record ? checked_stage(*record, prepared.key, lease) : std::nullopt; + if (!result || record->cut.trace != prepared.trace || result->ghost != prepared.ghost) + return std::unexpected(stage_failed(PublicationError::unknown, {}, {}, prepared)); + return *result; +} + +std::expected +stage_cellular_migration(Postgres& db, const FilesystemObjectStore& source_objects, + const FilesystemObjectStore& destination_objects, const LeaseGrant& lease, + const CommittedCheckpoint& source, const CommittedCheckpoint& destination, + const server::CellularMigrationTrace& trace, milliseconds timeout, + const PublishHooks& hooks) { + const auto s = decode_manifest(source.manifest), d = decode_manifest(destination.manifest); + if (!s || !d || !bound(source, *s) || !bound(destination, *d) || + !cuts(*s, *d, source.head.digest, destination.head.digest, trace) || + lease.frame != d->frame) + return std::unexpected(stage_failed(PublicationError::predecessor)); + auto src = source_objects.get({s->payload_sha256, s->payload_bytes}); + auto dst = destination_objects.get({d->payload_sha256, d->payload_bytes}); + if (!src || !dst) + return std::unexpected( + stage_failed(PublicationError::predecessor, {}, !src ? src.error() : dst.error())); + const auto candidate = ghost_candidate(*s, *d, *src, *dst, trace); + const auto encoded = server::encode_migration_trace(trace); + if (!candidate || !encoded) + return std::unexpected(stage_failed(PublicationError::candidate)); + auto capsule = destination_objects.put(*src); + if (!capsule) + return std::unexpected(stage_failed(PublicationError::upload, {}, capsule.error())); + auto capsule_bytes = destination_objects.get(*capsule); + if (!capsule_bytes || *capsule_bytes != *src || + *capsule != VerifiedObject{s->payload_sha256, s->payload_bytes}) + return std::unexpected( + stage_failed(PublicationError::verify, {}, + capsule_bytes ? ObjectError::corrupt : capsule_bytes.error())); + auto ghost = destination_objects.put(*candidate); + if (!ghost) + return std::unexpected(stage_failed(PublicationError::upload, {}, ghost.error())); + auto readback = destination_objects.get(*ghost); + if (!readback || *readback != *candidate) + return std::unexpected(stage_failed(PublicationError::verify, {}, + readback ? ObjectError::corrupt : readback.error())); + if (hooks.before_transaction) + hooks.before_transaction(); + return retry_cellular_migration_stage(db, lease, {d->key, *encoded, *ghost}, timeout); +} + +std::expected, RecoveryFailure> +load_cellular_migration_stage(Postgres& db, const FilesystemObjectStore& objects, RegionKey key, + const LeaseGrant& lease, milliseconds timeout) { + const auto fail = [](RecoveryError kind) { return std::unexpected(RecoveryFailure{kind, {}}); }; + const std::array args{std::to_string(key.world), std::to_string(key.region), + uuid(lease.holder), std::to_string(lease.epoch)}; + auto rows = db.query(std::string("SELECT ") + stage_columns + + " FROM evolution.load_cellular_migration_stage($1::evolution.u64," + "$2::evolution.u64,$3::uuid,$4::evolution.u64)", + args, timeout); + if (!rows) + return std::unexpected(RecoveryFailure{RecoveryError::store, failure(rows.error())}); + if (rows->empty()) + return std::optional{}; + const auto record = rows->size() == 1 ? parse_stage(rows->front()) : std::nullopt; + auto result = record ? checked_stage(*record, key, lease) : std::nullopt; + if (!result || result->freeze.aborted) + return fail(RecoveryError::history); + const auto s = decode_manifest(record->cut.source), + d = decode_manifest(record->cut.destination); + if (!s || !d) + return fail(RecoveryError::manifest); + auto src = objects.get({s->payload_sha256, s->payload_bytes}); + auto dst = objects.get({d->payload_sha256, d->payload_bytes}); + auto ghost = objects.get(result->ghost); + if (!src || !dst || !ghost) + return fail(RecoveryError::tail); + const auto candidate = ghost_candidate(*s, *d, *src, *dst, result->freeze.trace); + if (!candidate || *candidate != *ghost) + return fail(RecoveryError::history); + return result; +} } // namespace evo::persistence diff --git a/src/server/cellular_region_host.cpp b/src/server/cellular_region_host.cpp index d93dda0..397c4f7 100644 --- a/src/server/cellular_region_host.cpp +++ b/src/server/cellular_region_host.cpp @@ -416,7 +416,8 @@ void CellularRegionHost::refresh() { status_.ready = status_.phase == HostPhase::running && state_ && committed_ && grant && authority_.may_originate(LeaseClock::now()) && !interval_reload_ && !interval_request_.has_value() && !migration_reload_ && !migration_frozen_ && - !migration_request_.has_value() && !migration_abort_.has_value(); + !migration_request_.has_value() && !migration_stage_request_.has_value() && + !migration_abort_.has_value(); status_.migration_frozen = migration_frozen_; const auto& interval = admitted_interval_; status_.ecological_pending_ticks = @@ -430,7 +431,8 @@ void CellularRegionHost::refresh() { } void CellularRegionHost::reconcile_migration_freeze() { - if ((!migration_reload_ && !migration_frozen_ && !migration_request_ && !migration_abort_) || + if ((!migration_reload_ && !migration_frozen_ && !migration_request_ && + !migration_stage_request_ && !migration_abort_) || LeaseClock::now() < migration_poll_at_) return; const auto grant = authority_.grant(); @@ -460,6 +462,30 @@ void CellularRegionHost::reconcile_migration_freeze() { } migration_request_.reset(); } + if (migration_stage_request_) { + const auto staged = persistence::retry_cellular_migration_stage( + *db_, *grant, *migration_stage_request_, config_.lease.commit_timeout); + if (!staged) { + if (staged.error().kind == persistence::PublicationError::unknown) { + db_.reset(); + ++status_.unknown_outcomes; + return; + } + // A concurrent source abort can win before the uncertain stage. + // Reload authoritative pointers before making the region ready. + if (staged.error().store.kind != LeaseStoreError::commit_conflict) { + authority_.fence(); + status_.phase = staged.error().store.kind == LeaseStoreError::fenced || + staged.error().store.kind == LeaseStoreError::head_mismatch + ? HostPhase::acquiring + : HostPhase::quarantined; + state_.reset(); + committed_.reset(); + return; + } + } + migration_stage_request_.reset(); + } if (migration_abort_) { auto result = persistence::abort_cellular_migration_freeze( *db_, config_.key, *grant, *migration_abort_, config_.lease.commit_timeout); @@ -492,9 +518,65 @@ void CellularRegionHost::reconcile_migration_freeze() { return; } migration_frozen_ = frozen->has_value(); + if (!migration_frozen_) { + const auto staged = persistence::load_cellular_migration_stage( + *db_, *objects_, config_.key, *grant, config_.lease.commit_timeout); + if (!staged) { + migration_reload_ = true; + if (staged.error().kind == persistence::RecoveryError::store) { + db_.reset(); + ++status_.store_failures; + } else { + authority_.fence(); + status_.phase = HostPhase::quarantined; + } + return; + } + migration_frozen_ = staged->has_value(); + } migration_reload_ = false; } +std::expected +CellularRegionHost::stage_migration(const persistence::CommittedCheckpoint& source, + const persistence::FilesystemObjectStore& source_objects, + const CellularMigrationTrace& trace) { + const auto grant = authority_.grant(); + if (!grant.has_value() || !status_.ready || pending_ || !state_ || !committed_ || + !queued_births_.empty() || !queued_mutations_.empty() || mutations_uncertain_ || + !pending_endings_.empty() || interval_request_ || admitted_interval_ || interval_reload_ || + !population_interval_safe(*state_) || !authority_.may_begin_commit(LeaseClock::now()) || + !database()) + return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}}); + migration_reload_ = true; + refresh(); + const auto staged = + persistence::stage_cellular_migration(*db_, source_objects, *objects_, *grant, source, + *committed_, trace, config_.lease.commit_timeout); + if (!staged) { + if (staged.error().kind == persistence::PublicationError::unknown) { + migration_stage_request_ = staged.error().prepared; + db_.reset(); + ++status_.unknown_outcomes; + } else if (staged.error().store.kind == LeaseStoreError::fenced || + staged.error().store.kind == LeaseStoreError::head_mismatch) { + authority_.fence(); + state_.reset(); + committed_.reset(); + status_.phase = HostPhase::acquiring; + } + migration_poll_at_ = {}; + refresh(); + return std::unexpected(CellularPublishFailure{CellularHostError::publication, + staged.error().kind, staged.error().store, + staged.error().object}); + } + migration_poll_at_ = {}; + reconcile_migration_freeze(); + refresh(); + return {}; +} + std::expected CellularRegionHost::prepare_migration_freeze( const persistence::CommittedCheckpoint& destination, const persistence::FilesystemObjectStore& destination_objects, @@ -573,7 +655,8 @@ const HostStatus& CellularRegionHost::step() { acquire(); if (!pending_ && state_ && status_.phase == HostPhase::running) reconcile_migration_freeze(); - if (migration_reload_ || migration_frozen_ || migration_request_ || migration_abort_) { + if (migration_reload_ || migration_frozen_ || migration_request_ || migration_stage_request_ || + migration_abort_) { refresh(); return status_; } diff --git a/tests/population_publication_tests.cpp b/tests/population_publication_tests.cpp index 582284e..5b90b5d 100644 --- a/tests/population_publication_tests.cpp +++ b/tests/population_publication_tests.cpp @@ -5,6 +5,7 @@ #include "evo/server/cellular_region_host.hpp" #include "fixtures/cellular_ecology.hpp" +#include #include #include #include @@ -868,6 +869,155 @@ void migration_freezes(Fixture& f) { !take(retry_cellular_migration_freeze(*f.db, f.lease, prepared, 5s), "duplicate freeze") .aborted, "duplicate freeze lost its state"); + const auto stage = [&](const CellularMigrationTrace& trace, + std::chrono::milliseconds timeout = 5s) { + return stage_cellular_migration(*f.db, *f.objects, *destination_objects, + destination_lease, f.head, destination_head, trace, + timeout); + }; + // The optimistic destination cut must still be exact at first staging. + take(f.db->query("BEGIN", {}, 5s), "move destination before stage transaction"); + if (representation == 0) + take(migrate_cellular_to_population(*f.db, *destination_objects, destination_lease, + destination_head, f.id(), f.build, {}, 5s), + "temporarily move destination profile"); + else + take(materialize_population_profile(*f.db, *destination_objects, destination_lease, + destination_head, f.id(), f.build, 5s), + "temporarily move population destination"); + const auto moved = stage(frozen.trace); + take(f.db->query("ROLLBACK", {}, 5s), "rollback moved destination fixture"); + require(!moved && moved.error().store.kind == LeaseStoreError::head_mismatch, + "stage silently rebased a moved destination"); + if (representation == 1) { + take(f.db->query("BEGIN", {}, 5s), "stage pending interval transaction"); + take(admit_population_interval(*f.db, destination_key, destination_lease, + destination_head.head, + destination_head.head.end_tick + 20, 5s), + "destination admitted work before staging"); + const auto pending = stage(frozen.trace); + take(f.db->query("ROLLBACK", {}, 5s), "rollback destination interval fixture"); + require(!pending && pending.error().store.kind == LeaseStoreError::commit_conflict, + "stage stranded destination interval work"); + } + const auto staged = take(stage(frozen.trace), "durable destination staging"); + const CellularMigrationStagePrepared stage_request{destination_key, prepared.trace, + staged.ghost}; + require( + !staged.freeze.aborted && + take(take(load_cellular_migration_stage(*f.db, *destination_objects, + destination_key, destination_lease, 5s), + "recover destination ghost"), + "missing staged destination") + .ghost == staged.ghost && + take(load_region_checkpoint(*f.db, destination_key, 5s), + "staged destination still owns original head") + .head == destination_before.head && + take(retry_cellular_migration_stage(*f.db, destination_lease, stage_request, 5s), + "duplicate stage") + .ghost == staged.ghost, + "staging activated a ghost or lost exact replay"); + auto changed_stage = stage_request; + ++changed_stage.ghost.bytes; + const auto changed = + retry_cellular_migration_stage(*f.db, destination_lease, changed_stage, 5s); + require(!changed && changed.error().store.kind == LeaseStoreError::commit_conflict, + "stage accepted conflicting immutable ghost bytes"); + auto reverse = frozen.trace; + reverse.request.request = f.id(); + std::swap(reverse.request.source, reverse.request.destination); + const auto overlapping = retry_cellular_migration_freeze( + *f.db, destination_lease, + {destination_key, take(encode_migration_trace(reverse), "overlapping source trace")}, + 5s); + require(!overlapping && overlapping.error().store.kind == LeaseStoreError::commit_conflict, + "staged destination simultaneously prepared an outgoing workflow"); + take(f.db->query("BEGIN", {}, 5s), "staged destination life reservation transaction"); + Uuid new_account{}; + new_account.back() = 5; + const auto staged_control = + take(claim_account_control(*f.db, destination_key, destination_lease, new_account, 1, + 9950, 60000ms, 5s), + "staged destination reservation current control"); + const auto staged_life = + reserve_primitive_life(*f.db, destination_key, destination_lease, new_account, + staged_control.generation, 1, 1, 5s); + take(f.db->query("ROLLBACK", {}, 5s), "rollback staged life reservation fixture"); + require(!staged_life && + std::string(staged_life.error().cause.sqlstate.data(), 5) == "EVL18", + "staged destination admitted a new player life reservation"); + if (representation == 0) { + CellularBatch inputs{{f.key.world, 2, 1, destination_head.head.commit, + destination_head.head.digest, destination->tick(), + destination->tick() + 1}, + {{}}}; + const auto advanced = take(replay_cellular_batch(*destination, inputs), + "detached staged destination future"); + const auto denied = + publish_cellular_batch(*f.db, *destination_objects, destination_lease, + destination_head, inputs, *advanced, f.id(), f.build, 5s); + require(!denied && + std::string(denied.error().store.cause.sqlstate.data(), 5) == "EVL18", + "staged destination published an ordinary future"); + } else { + require(!admit_population_interval(*f.db, destination_key, destination_lease, + destination_head.head, + destination_head.head.end_tick + 20, 5s) && + !materialize_population_profile(*f.db, *destination_objects, + destination_lease, destination_head, f.id(), + f.build, 5s), + "staged destination progressed or changed profile"); + } + const auto ghost_path = + destination_objects->directory() / hexadecimal(staged.ghost.sha256, false); + const auto ghost_backup = ghost_path.string() + ".stage-test"; + std::filesystem::rename(ghost_path, ghost_backup); + const auto missing_ghost = load_cellular_migration_stage( + *f.db, *destination_objects, destination_key, destination_lease, 5s); + std::filesystem::rename(ghost_backup, ghost_path); + require(!missing_ghost && missing_ghost.error().kind == RecoveryError::tail, + "missing ghost fell back to the active destination"); + char ghost_first{}; + { + std::fstream file(ghost_path, std::ios::binary | std::ios::in | std::ios::out); + file.get(ghost_first); + file.seekp(0); + file.put(static_cast(static_cast(ghost_first) ^ 1U)); + file.flush(); + require(file.good(), "same-length ghost corruption fixture"); + } + const auto corrupt_ghost = load_cellular_migration_stage( + *f.db, *destination_objects, destination_key, destination_lease, 5s); + { + std::fstream file(ghost_path, std::ios::binary | std::ios::in | std::ios::out); + file.put(ghost_first); + file.flush(); + require(file.good(), "restore corrupt ghost fixture"); + } + require(!corrupt_ghost && corrupt_ghost.error().kind == RecoveryError::tail, + "same-length corrupt ghost bypassed digest verification"); + const auto source_manifest = take(decode_manifest(f.head.manifest), "stage source capsule"); + const auto source_capsule_path = + destination_objects->directory() / hexadecimal(source_manifest.payload_sha256, false); + const auto source_capsule_backup = source_capsule_path.string() + ".stage-test"; + std::filesystem::rename(source_capsule_path, source_capsule_backup); + const auto missing_source = load_cellular_migration_stage( + *f.db, *destination_objects, destination_key, destination_lease, 5s); + std::filesystem::rename(source_capsule_backup, source_capsule_path); + require(!missing_source && missing_source.error().kind == RecoveryError::tail, + "destination recovery used the live source instead of its retained capsule"); + const auto old_destination_lease = destination_lease; + take(release_region_lease(*f.db, destination_key, destination_lease, 5s), + "release staged destination owner"); + destination_lease = take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s), + "replacement staged destination owner"); + require( + take(load_cellular_migration_stage(*f.db, *destination_objects, destination_key, + destination_lease, 5s), + "new epoch ghost replay") + .has_value() && + !retry_cellular_migration_stage(*f.db, old_destination_lease, stage_request, 5s), + "replacement activated the ghost or stale destination staged it"); auto conflict_trace = frozen.trace; ++conflict_trace.transferred.energy.value; auto conflict = retry_cellular_migration_freeze( @@ -960,6 +1110,15 @@ void migration_freezes(Fixture& f) { "aborted pointer load") .has_value(), "abort was not durable or preparation retry reopened it"); + require( + !take(load_cellular_migration_stage(*f.db, *destination_objects, destination_key, + destination_lease, 5s), + "aborted destination pointer") + .has_value() && + take(retry_cellular_migration_stage(*f.db, destination_lease, stage_request, 5s), + "retry aborted stage") + .freeze.aborted, + "source abort stranded the destination or stage retry reopened its pointer"); require(take(load_region_checkpoint(*f.db, f.key, 5s), "aborted source head").head == source_before.head && f.raw->snapshot() == source_bytes, @@ -1004,6 +1163,76 @@ void migration_freezes(Fixture& f) { 5s), "abort reconciled freeze"); + const auto false_stage_freeze = take(prepare(request_for()), "false ghost preparation"); + const auto original_destination = + take(decode_manifest(destination_head.manifest), "false ghost existing object"); + const CellularMigrationStagePrepared false_ghost{ + destination_key, + take(encode_migration_trace(false_stage_freeze.trace), "false ghost trace"), + {original_destination.payload_sha256, original_destination.payload_bytes}}; + take(retry_cellular_migration_stage(*f.db, destination_lease, false_ghost, 5s), + "metadata-only false ghost stage"); + const auto false_ghost_replay = load_cellular_migration_stage( + *f.db, *destination_objects, destination_key, destination_lease, 5s); + require(!false_ghost_replay && false_ghost_replay.error().kind == RecoveryError::history, + "stage recovery accepted a real object with the wrong complete biology"); + take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, + false_stage_freeze.trace.request.request, 5s), + "abort false ghost evidence"); + + const auto uncertain_stage_freeze = + take(prepare(request_for()), "unknown stage preparation"); + take(locker->query("BEGIN", {}, 5s), "stage ambiguity lock"); + take(locker->query("SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND " + "region_id=2 FOR UPDATE", + {}, 5s), + "hold stage reply"); + auto stage_timed = connection(); + const auto stage_unknown = stage_cellular_migration( + *stage_timed, *f.objects, *destination_objects, destination_lease, f.head, + destination_head, uncertain_stage_freeze.trace, 300ms); + require(!stage_unknown && stage_unknown.error().kind == PublicationError::unknown && + !stage_unknown.error().prepared.trace.empty(), + "stage ambiguity lost exact bytes"); + take(locker->query("ROLLBACK", {}, 5s), "release stage ambiguity"); + take(retry_cellular_migration_stage(*f.db, destination_lease, + stage_unknown.error().prepared, 5s), + "resolve unknown stage"); + take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, + uncertain_stage_freeze.trace.request.request, 5s), + "abort reconciled stage"); + + for (unsigned race = 0; race < 4; ++race) { + const auto race_freeze = take(prepare(request_for()), "stage race preparation"); + auto stage_db = connection(), abort_db = connection(); + std::barrier start(2); + std::optional> + staged_race; + std::optional> aborted_race; + std::thread staging([&] { + start.arrive_and_wait(); + staged_race = stage_cellular_migration(*stage_db, *f.objects, *destination_objects, + destination_lease, f.head, destination_head, + race_freeze.trace, 5s); + }); + std::thread aborting([&] { + start.arrive_and_wait(); + aborted_race = abort_cellular_migration_freeze( + *abort_db, f.key, f.lease, race_freeze.trace.request.request, 5s); + }); + staging.join(); + aborting.join(); + require(aborted_race.has_value() && aborted_race->has_value() && + staged_race.has_value() && + (staged_race->has_value() || + staged_race->error().store.kind == LeaseStoreError::commit_conflict) && + !take(load_cellular_migration_stage(*f.db, *destination_objects, + destination_key, destination_lease, 5s), + "stage race final pointer") + .has_value(), + "stage/abort race left an active ghost or stranded pause"); + } + // Both sides of preparation's transaction boundary, with real process death. for (unsigned point = 0; point < 2; ++point) { const auto killed_request = request_for(); @@ -1037,6 +1266,38 @@ void migration_freezes(Fixture& f) { 5s), "abort killed preparation"); } + for (unsigned point = 0; point < 2; ++point) { + const auto killed = take(prepare(request_for()), "stage process-loss preparation"); + const auto child = ::fork(); + require(child >= 0, "stage publisher fork"); + if (child == 0) { + try { + auto db = connection(); + PublishHooks hooks; + if (point == 0) + hooks.before_transaction = [] { ::kill(::getpid(), SIGKILL); }; + if (!stage_cellular_migration(*db, *f.objects, *destination_objects, + destination_lease, f.head, destination_head, + killed.trace, 5s, hooks)) + ::_exit(44); + ::kill(::getpid(), SIGKILL); + } catch (...) { + ::_exit(45); + } + ::_exit(46); + } + int status{}; + require(::waitpid(child, &status, 0) == child && WIFSIGNALED(status) && + WTERMSIG(status) == SIGKILL && + take(load_cellular_migration_stage(*f.db, *destination_objects, + destination_key, destination_lease, 5s), + "recover killed stage") + .has_value() == (point == 1), + "stage process loss changed ownership or lost the durable pause"); + take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, + killed.trace.request.request, 5s), + "abort killed stage"); + } } // The owning host must retain the freeze through drain/recreation with entry off. take(release_region_lease(*f.db, f.key, f.lease, 5s), "release fixture to frozen host"); @@ -1107,6 +1368,64 @@ void migration_freezes(Fixture& f) { host->drain(); f.lease = take(acquire_region_lease(*f.db, f.key, f.build, 60000ms, 5s), "return freeze fixture authority"); + const auto destination_host_freeze = + take(prepare_cellular_migration_freeze(*f.db, *f.objects, *destination_objects, f.lease, + f.head, destination_head, request_for(), 5s), + "destination host source preparation"); + take(release_region_lease(*f.db, destination_key, destination_lease, 5s), + "release destination to owning host"); + const auto make_destination_host = [&] { + auto objects = take(FilesystemObjectStore::open(environment("PGHOST"), destination_key, + max_cellular_population_bytes), + "destination host objects"); + CellularHostConfig config{destination_key, 1, {}, f.build, {f.build}}; + config.lease.commit_timeout = 300ms; + return take( + CellularRegionHost::create( + config, + [] { return std::expected, PgFailure>{connection()}; }, + std::move(objects)), + "destination staging host"); + }; + auto destination_host = make_destination_host(); + require(destination_host->step().ready, "destination host before staging"); + const auto destination_bytes = destination_host->state()->snapshot(); + take(locker->query("BEGIN", {}, 5s), "host stage ambiguity transaction"); + take(locker->query("SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND " + "region_id=1 FOR UPDATE", + {}, 5s), + "hold owning destination stage"); + const auto host_stage_unknown = + destination_host->stage_migration(f.head, *f.objects, destination_host_freeze.trace); + require(!host_stage_unknown && !destination_host->status().ready && + destination_host->state()->snapshot() == destination_bytes, + "owning destination advanced during ambiguous staging"); + take(locker->query("ROLLBACK", {}, 5s), "release owning stage reply"); + for (unsigned attempt = 0; attempt < 200 && !destination_host->status().migration_frozen; + ++attempt) { + destination_host->step(); + std::this_thread::sleep_for(25ms); + } + require(destination_host->status().migration_frozen && !destination_host->status().ready && + !destination_host->advance_unattended(64), + "owning destination activated its ghost"); + destination_host->drain(); + destination_host = make_destination_host(); + require(!destination_host->step().ready && destination_host->status().migration_frozen && + destination_host->state()->snapshot() == destination_bytes && + !destination_host->authority().has_value(), + "replacement destination lost staging or adopted the uncommitted ghost"); + take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, + destination_host_freeze.trace.request.request, 5s), + "source-authorized abort releases destination host"); + for (unsigned attempt = 0; attempt < 200 && !destination_host->step().ready; ++attempt) + std::this_thread::sleep_for(25ms); + require(destination_host->status().ready && !destination_host->status().migration_frozen && + destination_host->state()->snapshot() == destination_bytes, + "destination failed to resume its unchanged original state after source abort"); + destination_host->drain(); + destination_lease = take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s), + "return staged destination fixture authority"); f.head = take( materialize_population_profile(*f.db, *f.objects, f.lease, f.head, f.id(), f.build, 5s), "resume frozen population as raw owner"); diff --git a/tests/region_migration_tests.cpp b/tests/region_migration_tests.cpp index 75001f8..c877457 100644 --- a/tests/region_migration_tests.cpp +++ b/tests/region_migration_tests.cpp @@ -213,6 +213,7 @@ void upgrade_across_migration_and_barrier() { " -v migration_sha_45=$(s 0045_owned_population_intervals.sql)" " -v migration_sha_46=$(s 0046_boundary_accounting_upgrade.sql)" " -v migration_sha_47=$(s 0047_cellular_migration_freeze.sql)" + " -v migration_sha_48=$(s 0048_cellular_migration_staging.sql)" " -v runtime_role=evo_runtime_test -f migrations/apply.sql" " > /dev/null 2>&1") != 0); CHECK(history(*owner_db) == "1,2,3,4,5,6,7"); @@ -230,7 +231,7 @@ void upgrade_across_migration_and_barrier() { "tools/migrate-postgres.sh\" > /dev/null") == 0); CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24," - "25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47"); + "25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48"); CHECK(scalar(*owner_db, "SELECT (c.condeferrable AND c.condeferred AND " "c.confrelid = 'evolution.region_commits'::regclass)::text " "FROM pg_catalog.pg_constraint c WHERE c.conrelid = " diff --git a/tools/ci/postgres-check.sh b/tools/ci/postgres-check.sh index b306edb..2f0f087 100644 --- a/tools/ci/postgres-check.sh +++ b/tools/ci/postgres-check.sh @@ -129,7 +129,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" -f "$root/migrations/0001_region_bootstrap.sql" > /dev/null test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1 PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 +test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 # v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2. v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")" v2_sha="${v2_sha%% *}" @@ -150,7 +150,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 47:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 +test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 48:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1 # Review P1: a v4 database holding a published (possibly terminal) checkpoint has # no durable decisions for it. The v5 upgrade must refuse before changing anything. @@ -204,7 +204,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47 +test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick phase permission-audit @@ -257,9 +257,9 @@ expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again. phase checksum-drift # Unknown future versions and checksum drift must not be repaired silently. -psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=48 WHERE version=47' +psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=49 WHERE version=48' if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi -psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=47 WHERE version=48" +psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=48 WHERE version=49" psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14" if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13" diff --git a/tools/migrate-postgres.sh b/tools/migrate-postgres.sh index a900957..202cd78 100644 --- a/tools/migrate-postgres.sh +++ b/tools/migrate-postgres.sh @@ -106,6 +106,8 @@ digest_46="$(sha256sum "$root/migrations/0046_boundary_accounting_upgrade.sql")" digest_46="${digest_46%% *}" digest_47="$(sha256sum "$root/migrations/0047_cellular_migration_freeze.sql")" digest_47="${digest_47%% *}" +digest_48="$(sha256sum "$root/migrations/0048_cellular_migration_staging.sql")" +digest_48="${digest_48%% *}" psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \ --set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \ --set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \ @@ -146,5 +148,6 @@ psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \ --set="migration_sha_45=$digest_45" \ --set="migration_sha_46=$digest_46" \ --set="migration_sha_47=$digest_47" \ + --set="migration_sha_48=$digest_48" \ --set="runtime_role=$runtime_role" \ --file="$root/migrations/apply.sql"