M12.4: retain read-only destination migration stages
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m25s
Foundation / checks (network) (push) Successful in 20m44s
Foundation / checks (client) (push) Successful in 21m57s
Foundation / checks (tsan) (push) Successful in 22m46s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 30m2s
Foundation / checks (postgres) (push) Canceled after 29m56s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m25s
Foundation / checks (network) (push) Successful in 20m44s
Foundation / checks (client) (push) Successful in 21m57s
Foundation / checks (tsan) (push) Successful in 22m46s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 30m2s
Foundation / checks (postgres) (push) Canceled after 29m56s
This commit is contained in:
@@ -1,5 +1,22 @@
|
||||
# Project status and next steps
|
||||
|
||||
## M12.4i durable read-only destination staging - 2026-10-18
|
||||
|
||||
ADR-0148/schema 48 stages the destination under current authority at its exact
|
||||
captured head. A verified source capsule and complete destination ghost are
|
||||
retained for replay, while the source owns all outgoing stock and neither head
|
||||
moves. Both migration roles pause publication/admission and host readiness
|
||||
across replacement. Explicit source abort releases both pointers atomically;
|
||||
stage/abort races and exact retries cannot activate or reopen an aborted ghost.
|
||||
|
||||
Native Release passes 78/78; the private PostgreSQL staging suite, ASan/UBSan,
|
||||
Clang analysis, format and shell syntax checks pass. The complete private
|
||||
persistence/process/backup/DR/migration/permission suite passes in 501 s.
|
||||
[Evidence and limits](engineering/VALIDATION.md)
|
||||
retain atomic paired publication, its common immutable receipt, commit/abort
|
||||
races and recovery/activation as the next M12.4 segment. All of M12 remains in
|
||||
progress. No deployment, automatic movement or live region changed.
|
||||
|
||||
## M12.4h durable source freeze and explicit abort - 2026-10-18
|
||||
|
||||
ADR-0147/schema 47 retains exact biological migration preparation, replay capsules
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# ADR-0148: Durable destination staging without activation
|
||||
|
||||
## Status
|
||||
|
||||
Accepted and locally qualified for scoped M12.4i implementation, 2026-10-18.
|
||||
Executed evidence and limits are in [VALIDATION](../engineering/VALIDATION.md).
|
||||
Paired result publication and activation remain subsequent
|
||||
work. This selects no route, automatic migration or deployment.
|
||||
|
||||
## Decision
|
||||
|
||||
Extend ADR-0147 preparation with destination-authorized staging at the exact
|
||||
captured destination head. Lock the two regional lease rows in region order,
|
||||
check current destination authority and the retained source freeze, and refuse
|
||||
pending destination work. Retain one immutable staging record and one bounded
|
||||
destination pending pointer. A region cannot participate in two workflows or
|
||||
simultaneously be a source and destination. Both biological heads stay unchanged.
|
||||
|
||||
Restore both verified predecessor payloads and replay the exact retained trace.
|
||||
Copy the source predecessor into the destination object namespace, and upload
|
||||
and read back the derived destination candidate there. Record its SHA-256 and
|
||||
length. It is read-only evidence: it is never a committed checkpoint, owned
|
||||
stock, active population, census entry or permission to run biology. Recovery
|
||||
replays both cuts from the destination namespace and compares the complete
|
||||
candidate bytes. Equal aggregate material alone is insufficient.
|
||||
|
||||
New checkpoints, profile changes, intervals and player reservations are refused
|
||||
at a staged destination. Its host pauses readiness and advancement, retains exact
|
||||
requests after unknown replies, renews authority and recovers the pause across
|
||||
replacement. No lease expiry, drain or timeout releases it. Explicit source
|
||||
abort atomically appends the existing immutable abort and clears both pending
|
||||
pointers. It requires no destination authority because no stock has moved.
|
||||
Stage and abort serialize on the source lease row: either staging wins then
|
||||
abort removes both pointers, or abort wins and new staging is refused. An exact
|
||||
retry of a previously staged, aborted request returns that retained outcome and
|
||||
never reopens either pointer. Destination movement before first staging requires
|
||||
explicit source abort and a new preparation, without silent rebasing.
|
||||
|
||||
Keep all staged objects and predecessor evidence after abort. Backup already
|
||||
copies immutable namespace objects and shared metadata; this is not cellular
|
||||
clone/restore qualification. The future paired publisher must recheck both
|
||||
current fences, heads, staged object bindings and unsettled request, then commit
|
||||
both results and the common receipt atomically before activation.
|
||||
|
||||
## Alternatives, risks and qualification
|
||||
|
||||
An in-memory ghost disappears on process loss. Activating the uploaded payload
|
||||
duplicates source stock. Advancing a destination after staging invalidates the
|
||||
paired candidate. Freeze the whole destination instead; the coarse pause and
|
||||
extra immutable storage are explicit costs, not a final movement latency policy.
|
||||
|
||||
Require both representations, real player-origin wildlife, full ghost replay,
|
||||
unchanged heads and stock, conflicting/duplicate requests, pending-work refusal,
|
||||
stale destination authority, replacement/restart, missing/corrupt/semantically
|
||||
false ghost evidence, ambiguous replies, process kills, stage/abort races and
|
||||
host readiness. No coupled ecology, world-cut or multigeneration claim follows.
|
||||
+3
-1
@@ -1,6 +1,8 @@
|
||||
# Architecture decision records
|
||||
|
||||
- [ADR-0147: Durable biological preparation and source freeze](0147-durable-cellular-migration-freeze.md) - schema 47 retains exact preparation/capsules and explicit abort; the source stays authoritative and paused across worker replacement. Destination staging and paired activation remain open.
|
||||
- [ADR-0148: Durable destination staging without activation](0148-durable-cellular-migration-staging.md) - schema 48 freezes the destination at its exact cut, retains a verified read-only ghost and replays it on recovery; source abort releases both pauses. Paired publication/activation remain open.
|
||||
|
||||
- [ADR-0147: Durable biological preparation and source freeze](0147-durable-cellular-migration-freeze.md) - schema 47 retains exact preparation/capsules and explicit abort; the source stays authoritative and paused across worker replacement. Paired activation remains open.
|
||||
|
||||
- [ADR-0146: Durable boundary-accounting upgrade admission](0146-durable-boundary-accounting-upgrade.md) - schema 46 extends immutable version-upgrade evidence to same-profile bundle 9->10, full upgrade replay, population batch 2 and preserved representation/interval guards; paired migration escrow remains open.
|
||||
|
||||
|
||||
@@ -16,8 +16,15 @@ uses both retained manifests and a verified destination capsule in the source's
|
||||
object namespace. Both biological heads stay unchanged; destination inspection
|
||||
is optimistic and grants no destination authority. Source publication and new
|
||||
interval/player reservations are blocked until explicit settlement. See
|
||||
[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md). Destination staging,
|
||||
atomic paired result publication and activation remain required.
|
||||
[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md).
|
||||
|
||||
M12.4i/schema 48 adds destination-authorized staging under ordered endpoint locks
|
||||
and the exact retained preparation/heads. A verified source capsule and derived
|
||||
destination ghost stay read-only in the destination namespace. Both roles pause
|
||||
publication, interval/player admission and host readiness across replacement.
|
||||
Explicit source abort releases both pointers atomically; stage/abort races cannot
|
||||
activate or strand stock. See [ADR-0148](../adr/0148-durable-cellular-migration-staging.md).
|
||||
Atomic paired result publication, its common receipt and activation remain required.
|
||||
|
||||
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
|
||||
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
|
||||
|
||||
@@ -70,7 +70,13 @@ destination predecessor capsule in the source namespace allows complete pair
|
||||
replay without consulting destination live state. The owning host pauses
|
||||
readiness, retains unknown requests and never treats timeout/drain as abort.
|
||||
See [the freeze contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
|
||||
Destination staging, paired commit/abort races and activation remain open.
|
||||
ADR-0148/schema 48 now stages a verified destination ghost and source replay
|
||||
capsule under current destination authority and ordered endpoint locks. The
|
||||
destination head remains unchanged and is durably paused alongside the source.
|
||||
Recovery verifies the complete ghost without activating it; explicit source abort
|
||||
releases both pointers atomically. Stage/abort races serialize on source authority.
|
||||
See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md).
|
||||
Atomic paired publication, commit/abort races and activation remain open.
|
||||
|
||||
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
|
||||
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
|
||||
|
||||
@@ -140,3 +140,26 @@ and canonically encoded false member/material declarations fail complete replay.
|
||||
See [the contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
|
||||
Destination staging, atomic paired head publication, commit/abort races and
|
||||
activation remain the next segment; no route or automatic migration is enabled.
|
||||
|
||||
## M12.4i durable read-only destination staging
|
||||
|
||||
ADR-0148/schema 48 freezes the destination at its exact captured head under
|
||||
current destination authority and ordered endpoint locks. The adapter restores
|
||||
both cuts, replays the retained trace, copies the source capsule into the
|
||||
destination namespace and uploads/reads back the derived destination ghost.
|
||||
Its exact digest/length is immutable retry evidence, not an ownership receipt.
|
||||
Recovery uses those local objects and compares the complete rederived ghost.
|
||||
Neither biological head moves, and the source still owns all outgoing stock.
|
||||
|
||||
Both local migration roles are exclusive and block new publication, profile
|
||||
changes, intervals and player reservations. Unknown staging retains exact bytes;
|
||||
the owning destination pauses readiness/advancement through worker replacement.
|
||||
Source abort serializes with staging and releases both pointers atomically.
|
||||
Aborted-stage retries return their old outcome and never reopen a pause.
|
||||
Missing/corrupt evidence and real but semantically false objects fail closed.
|
||||
See [the staging contract](../../schemas/ecology/cellular-migration-stage-v1.md).
|
||||
|
||||
Atomic paired publication/common receipt, commit/abort races, recovery/activation
|
||||
and linked ecology remain subsequent work. The ghost cannot run biology or
|
||||
contribute owned census/extinction stock. No biological format, rule, route,
|
||||
automatic movement, live region or deployment changes.
|
||||
|
||||
@@ -64,9 +64,12 @@ or close source-freeze/destination-activation and escrow qualification.
|
||||
ADR-0147/schema 47 now adds durable source preparation/freeze, retained replay
|
||||
capsules, explicit abort and host pause/recovery. Source stock remains owned at
|
||||
its exact head through worker replacement; no timeout or drain releases it.
|
||||
Destination staging, atomic paired commit, commit/abort races and activation
|
||||
remain open. The destination head is only checked optimistically at preparation
|
||||
and must be rechecked before any future stage/commit.
|
||||
ADR-0148/schema 48 now supplies destination-authorized staging at its exact
|
||||
captured head, verified read-only ghost/capsule retention and full destination
|
||||
recovery. Both endpoints stay paused; explicit source abort atomically releases
|
||||
both pointers and races safely with staging. Atomic paired commit, its common
|
||||
receipt, commit/abort races and activation remain open. Preparation remains an
|
||||
optimistic destination observation; first staging rechecks both captured heads.
|
||||
The transfer tests also expose the current rules-8 maturation gap: buds retain
|
||||
their subadult realization while reproduction requires a fully realized parent.
|
||||
Finite extra food alone does not close repeated wildlife generations. A versioned
|
||||
|
||||
@@ -32,6 +32,19 @@ the C++ decoder. The metric has one fixed unlabelled series and remains bounded.
|
||||
These are preparation/abort checks, not proof of destination activation or atomic
|
||||
paired publication; full historical persistence regression is also required.
|
||||
|
||||
M12.4i's private staging matrix covers both bundle-10 representations and actual
|
||||
player-origin wildlife. It requires unchanged biological heads, complete ghost
|
||||
replay, exact/conflicting retries, refusal of moved destination heads and pending
|
||||
intervals, publication/profile/life-reservation guards and exclusive endpoint
|
||||
roles. Stale destination epochs fail; replacement retains the read-only ghost.
|
||||
Missing source capsules, missing/same-length corrupt ghosts and valid but false
|
||||
ghost objects fail recovery. Real lock-timeout ambiguity, SIGKILL before/after
|
||||
stage commit and independent-connection stage/abort races preserve one outcome.
|
||||
The owning destination host retains exact uncertain work across restart and
|
||||
resumes only its original biology after confirmed source abort. Full schema,
|
||||
permission, backup/DR and historical regression remain required. These tests do
|
||||
not qualify paired commit, destination activation or linked ecological movement.
|
||||
|
||||
Tests defend contracts, not directory coverage. All checks remain active in
|
||||
Release builds. Current proof has no actual network, biology or distributed
|
||||
recovery, so those categories below are milestone gates rather than passing
|
||||
|
||||
@@ -1,5 +1,53 @@
|
||||
# Validation report
|
||||
|
||||
## 2026-10-18 - M12.4i durable read-only destination staging
|
||||
|
||||
ADR-0148/schema 48 retains a destination-authorized staging record and bounded
|
||||
pending pointer at the exact captured head. The adapter verifies both cuts,
|
||||
replays the trace and retains a source capsule plus verified destination ghost
|
||||
in the destination namespace. The source owns all outgoing stock; neither head
|
||||
moves. Source abort releases both pending pointers atomically, and exact stage
|
||||
retries retain their original outcome. Both migration roles are exclusive.
|
||||
|
||||
Executed locally:
|
||||
|
||||
- Native MSVC Release **78/78 passed**, 12.62 s.
|
||||
- Actual GCC 14 optional server/persistence targets rebuilt; final private
|
||||
PostgreSQL population/staging suite passed, **11 s** (8 s test phase).
|
||||
- Linux Clang 19 ASan/UBSan private population/staging suite passed, **33 s**
|
||||
(30 s test phase), with no sanitizer reports.
|
||||
- Explicit clang-tidy 19 on the adapter, owning host and PostgreSQL population
|
||||
tests passed with no owned warnings. Repository clang-format 19 and both
|
||||
changed shell scripts' syntax checks passed.
|
||||
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
|
||||
**501 s**: C++ integration/process/load/history (426 s), life reservations,
|
||||
backup/restore, host-loss DR, upgrade, migration history/rollback, runtime
|
||||
permission audit and checksum/future-version rejection. The detached runner
|
||||
recorded its completion code.
|
||||
|
||||
Both representations use real funded player-origin wildlife and unchanged cuts.
|
||||
Cases cover exact/conflicting retries, a moved destination, pending destination
|
||||
intervals, source/destination role exclusivity, blocked publication/profile/life
|
||||
reservations and stale destination authority. Replacement epochs retain staging.
|
||||
Missing source capsules, missing and same-length corrupt ghosts fail strong
|
||||
binding. An existing valid object with false destination biology fails complete
|
||||
replay. Real lock-timeout ambiguity retains exact requests; SIGKILL before/after
|
||||
stage commit leaves absence/a retained pause respectively. Independent database
|
||||
connections race stage against source abort without stranded destination work.
|
||||
The owning destination host pauses on unknown staging, survives drain/recreation
|
||||
without adopting the ghost and resumes its identical original state after source
|
||||
abort. The existing fixed freeze metric covers either role without new labels.
|
||||
|
||||
Limits: this is staging evidence, not a migration result or activation receipt.
|
||||
Atomic paired publication, its common immutable outcome, commit/abort races and
|
||||
recovery/activation remain open. Backup retains all immutable namespace objects;
|
||||
existing backup/DR fixtures do not qualify cellular cloning or a new cellular
|
||||
restore procedure. No elapsed-day, linked-geometry or multigeneration claim
|
||||
follows. No live database, World0, deployment, dependency, biological rule, format
|
||||
or golden changed. The earlier captured `Build_Errors.txt` and unrelated local
|
||||
work remain untracked; remote CI confirmation of the previous runner repair
|
||||
remains separate.
|
||||
|
||||
## 2026-10-18 - M12.4h durable migration source freeze and explicit abort
|
||||
|
||||
ADR-0147/schema 47 retains exact source-authorized preparation, both predecessor
|
||||
|
||||
@@ -42,4 +42,36 @@ abort_cellular_migration_freeze(Postgres&, RegionKey, const LeaseGrant&, const U
|
||||
[[nodiscard]] std::expected<std::optional<CellularMigrationFreeze>, RecoveryFailure>
|
||||
load_cellular_migration_freeze(Postgres&, const FilesystemObjectStore&, RegionKey,
|
||||
const LeaseGrant&, std::chrono::milliseconds timeout);
|
||||
|
||||
struct CellularMigrationStagePrepared {
|
||||
RegionKey key{}; // Destination region.
|
||||
std::vector<std::uint8_t> trace;
|
||||
VerifiedObject ghost;
|
||||
};
|
||||
struct CellularMigrationStage {
|
||||
CellularMigrationFreeze freeze;
|
||||
VerifiedObject ghost; // Read-only evidence, never an ownership receipt.
|
||||
};
|
||||
struct CellularMigrationStageFailure {
|
||||
PublicationError kind{};
|
||||
LeaseStoreFailure store{};
|
||||
ObjectError object{};
|
||||
CellularMigrationStagePrepared prepared;
|
||||
};
|
||||
// Replay the retained pair, retain a source capsule and verified destination
|
||||
// ghost in the destination namespace, then freeze its unchanged exact head.
|
||||
[[nodiscard]] std::expected<CellularMigrationStage, CellularMigrationStageFailure>
|
||||
stage_cellular_migration(Postgres&, const FilesystemObjectStore& source_objects,
|
||||
const FilesystemObjectStore& destination_objects,
|
||||
const LeaseGrant& destination_lease, const CommittedCheckpoint& source,
|
||||
const CommittedCheckpoint& destination,
|
||||
const server::CellularMigrationTrace&, std::chrono::milliseconds timeout,
|
||||
const PublishHooks& hooks = {});
|
||||
[[nodiscard]] std::expected<CellularMigrationStage, CellularMigrationStageFailure>
|
||||
retry_cellular_migration_stage(Postgres&, const LeaseGrant&, const CellularMigrationStagePrepared&,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Fresh destination authority; complete local capsule and ghost replay.
|
||||
[[nodiscard]] std::expected<std::optional<CellularMigrationStage>, RecoveryFailure>
|
||||
load_cellular_migration_stage(Postgres&, const FilesystemObjectStore&, RegionKey, const LeaseGrant&,
|
||||
std::chrono::milliseconds timeout);
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -100,6 +100,12 @@ class CellularRegionHost {
|
||||
const CellularMigrationRequest&);
|
||||
[[nodiscard]] std::expected<void, CellularPublishFailure>
|
||||
abort_migration_freeze(const persistence::Uuid& request);
|
||||
// Explicit destination staging. The ghost stays read-only until a future
|
||||
// paired publication receipt; source abort releases this pause.
|
||||
[[nodiscard]] std::expected<void, CellularPublishFailure>
|
||||
stage_migration(const persistence::CommittedCheckpoint& source,
|
||||
const persistence::FilesystemObjectStore& source_objects,
|
||||
const CellularMigrationTrace&);
|
||||
// Explicit no-tick fenced transition. Uncertain publication pauses the host.
|
||||
[[nodiscard]] std::expected<void, CellularPublishFailure> set_population_mode(bool enabled);
|
||||
[[nodiscard]] bool population_mode() const noexcept;
|
||||
@@ -228,6 +234,7 @@ class CellularRegionHost {
|
||||
bool migration_reload_{}, migration_frozen_{};
|
||||
persistence::LeaseClock::time_point migration_poll_at_{};
|
||||
std::optional<persistence::CellularMigrationPrepared> migration_request_;
|
||||
std::optional<persistence::CellularMigrationStagePrepared> migration_stage_request_;
|
||||
std::optional<persistence::Uuid> migration_abort_;
|
||||
bool interval_reload_{};
|
||||
std::optional<persistence::PopulationInterval> admitted_interval_;
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
-- ADR-0148: a staged destination is frozen evidence, never active ownership.
|
||||
CREATE TABLE evolution.cellular_migration_stages (
|
||||
request_id uuid PRIMARY KEY REFERENCES evolution.cellular_migration_preparations,
|
||||
ghost_sha256 bytea NOT NULL CHECK (octet_length(ghost_sha256)=32 AND ghost_sha256<>decode(repeat('00',32),'hex')),
|
||||
ghost_bytes evolution.u64 NOT NULL CHECK (ghost_bytes BETWEEN 1 AND 16781400),
|
||||
holder uuid NOT NULL, epoch evolution.u64 NOT NULL CHECK (epoch>0)
|
||||
);
|
||||
CREATE TABLE evolution.cellular_migration_destination_pending (
|
||||
world_id evolution.u64 NOT NULL, region_id evolution.u64 NOT NULL,
|
||||
request_id uuid UNIQUE NOT NULL REFERENCES evolution.cellular_migration_stages,
|
||||
PRIMARY KEY(world_id,region_id)
|
||||
);
|
||||
REVOKE ALL ON evolution.cellular_migration_stages,evolution.cellular_migration_destination_pending FROM PUBLIC;
|
||||
CREATE TRIGGER cellular_migration_stages_immutable BEFORE UPDATE OR DELETE ON evolution.cellular_migration_stages
|
||||
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
CREATE TRIGGER cellular_migration_stages_no_truncate BEFORE TRUNCATE ON evolution.cellular_migration_stages
|
||||
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
|
||||
CREATE FUNCTION evolution.cellular_migration_stage_record(w evolution.u64,r evolution.u64,id uuid)
|
||||
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64)
|
||||
LANGUAGE sql STABLE SET search_path = pg_catalog, pg_temp AS $$
|
||||
SELECT p.trace,s.manifest,d.manifest,EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=p.request_id),
|
||||
g.ghost_sha256,g.ghost_bytes
|
||||
FROM evolution.cellular_migration_preparations p JOIN evolution.cellular_migration_stages g USING(request_id)
|
||||
JOIN evolution.region_commits s ON s.commit_id=p.source_commit
|
||||
JOIN evolution.region_commits d ON d.commit_id=p.destination_commit
|
||||
WHERE p.world_id=w AND p.destination_region=r AND p.request_id=id
|
||||
$$;
|
||||
REVOKE ALL ON FUNCTION evolution.cellular_migration_stage_record(evolution.u64,evolution.u64,uuid) FROM PUBLIC;
|
||||
|
||||
CREATE FUNCTION evolution.stage_cellular_migration(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,t bytea,digest bytea,size evolution.u64)
|
||||
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE id uuid; p evolution.cellular_migration_preparations%ROWTYPE;
|
||||
old evolution.cellular_migration_stages%ROWTYPE;
|
||||
BEGIN
|
||||
PERFORM evolution.check_cellular_migration_trace(t);
|
||||
IF w IS NULL OR r IS NULL OR evolution.manifest_uint(t,16,8)<>w OR evolution.manifest_uint(t,136,8)<>r
|
||||
OR digest IS NULL OR octet_length(digest)<>32 OR digest=decode(repeat('00',32),'hex')
|
||||
OR size IS NULL OR size NOT BETWEEN 1 AND (CASE evolution.manifest_uint(t,12,4) WHEN 1 THEN 16777216 ELSE 16781400 END) THEN
|
||||
RAISE EXCEPTION 'invalid destination stage' USING ERRCODE='22023';
|
||||
END IF;
|
||||
id := encode(substring(t FROM 49 FOR 16),'hex')::uuid;
|
||||
SELECT * INTO p FROM evolution.cellular_migration_preparations x WHERE x.request_id=id;
|
||||
IF NOT FOUND OR p.world_id<>w OR p.destination_region<>r OR p.trace<>t THEN
|
||||
RAISE EXCEPTION 'stage differs from retained preparation' USING ERRCODE='EVL07';
|
||||
END IF;
|
||||
-- Shared order for future two-owner work. Source abort serializes on its row.
|
||||
PERFORM 1 FROM evolution.region_leases x WHERE x.world_id=w AND x.region_id IN (p.source_region,r)
|
||||
ORDER BY x.region_id FOR UPDATE;
|
||||
PERFORM evolution.check_region_lease(w,r,holder,epoch);
|
||||
SELECT * INTO old FROM evolution.cellular_migration_stages x WHERE x.request_id=id;
|
||||
IF FOUND THEN
|
||||
IF old.ghost_sha256<>digest OR old.ghost_bytes<>size THEN
|
||||
RAISE EXCEPTION 'stage request reused' USING ERRCODE='EVL07';
|
||||
END IF;
|
||||
RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id);
|
||||
RETURN;
|
||||
END IF;
|
||||
IF EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=id)
|
||||
OR NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=p.source_region AND x.request_id=id)
|
||||
OR EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=r)
|
||||
OR EXISTS(SELECT 1 FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.region_id=r)
|
||||
OR EXISTS(SELECT 1 FROM evolution.population_interval_pending x WHERE x.world_id=w AND x.region_id=r)
|
||||
OR EXISTS(SELECT 1 FROM evolution.cellular_mutation_reservations x WHERE x.world_id=w AND x.region_id=r
|
||||
AND NOT EXISTS(SELECT 1 FROM evolution.cellular_mutation_action_publications a
|
||||
WHERE a.world_id=x.world_id AND a.account_id=x.account_id AND a.action_id=x.action_id))
|
||||
OR EXISTS(SELECT 1 FROM evolution.life_reservations x WHERE x.world_id=w AND x.region_id=r AND x.rejection=0 AND x.ended_commit IS NULL
|
||||
AND NOT EXISTS(SELECT 1 FROM evolution.primitive_life_bindings b
|
||||
WHERE b.world_id=x.world_id AND b.account_id=x.account_id AND b.request_id=x.request_id)) THEN
|
||||
RAISE EXCEPTION 'migration endpoint has pending work or was aborted' USING ERRCODE='EVL18';
|
||||
END IF;
|
||||
IF NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=p.source_region AND h.commit_id=p.source_commit
|
||||
AND h.digest=substring(t FROM 89 FOR 32) AND h.end_tick=evolution.manifest_uint(t,32,8))
|
||||
OR NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=r AND h.commit_id=p.destination_commit
|
||||
AND h.digest=substring(t FROM 161 FOR 32) AND h.end_tick=evolution.manifest_uint(t,32,8)) THEN
|
||||
RAISE EXCEPTION 'staged migration head moved' USING ERRCODE='EVL03';
|
||||
END IF;
|
||||
INSERT INTO evolution.cellular_migration_stages VALUES(id,digest,size,holder,epoch);
|
||||
INSERT INTO evolution.cellular_migration_destination_pending VALUES(w,r,id);
|
||||
RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id);
|
||||
END $$;
|
||||
REVOKE ALL ON FUNCTION evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64) TO :"runtime_role";
|
||||
|
||||
CREATE FUNCTION evolution.load_cellular_migration_stage(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64)
|
||||
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean,ghost_sha256 bytea,ghost_bytes evolution.u64)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE id uuid; initial_id uuid; src evolution.u64;
|
||||
BEGIN
|
||||
SELECT x.request_id,p.source_region INTO id,src FROM evolution.cellular_migration_destination_pending x
|
||||
JOIN evolution.cellular_migration_preparations p USING(request_id) WHERE x.world_id=w AND x.region_id=r;
|
||||
IF FOUND THEN
|
||||
initial_id := id;
|
||||
PERFORM 1 FROM evolution.region_leases x WHERE x.world_id=w AND x.region_id IN (src,r) ORDER BY x.region_id FOR UPDATE;
|
||||
END IF;
|
||||
PERFORM evolution.check_region_lease(w,r,holder,epoch);
|
||||
-- Abort may have won while we waited. Never turn its completed release into corruption.
|
||||
SELECT x.request_id INTO id FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.region_id=r;
|
||||
IF NOT FOUND THEN RETURN; END IF;
|
||||
IF initial_id IS DISTINCT FROM id THEN
|
||||
RAISE EXCEPTION 'destination workflow changed while waiting' USING ERRCODE='40001';
|
||||
END IF;
|
||||
IF NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_preparations p JOIN evolution.region_heads h
|
||||
ON h.world_id=p.world_id AND h.region_id=p.destination_region
|
||||
WHERE p.request_id=id AND p.world_id=w AND p.destination_region=r AND h.commit_id=p.destination_commit
|
||||
AND h.digest=substring(p.trace FROM 161 FOR 32) AND h.end_tick=evolution.manifest_uint(p.trace,32,8)
|
||||
AND EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=p.source_region AND x.request_id=id)
|
||||
AND NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=id)) THEN
|
||||
RAISE EXCEPTION 'migration destination freeze inconsistent' USING ERRCODE='XX000';
|
||||
END IF;
|
||||
RETURN QUERY SELECT * FROM evolution.cellular_migration_stage_record(w,r,id);
|
||||
END $$;
|
||||
REVOKE ALL ON FUNCTION evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64) TO :"runtime_role";
|
||||
|
||||
ALTER FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) RENAME TO abort_cellular_migration_freeze_v47;
|
||||
REVOKE ALL ON FUNCTION evolution.abort_cellular_migration_freeze_v47(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) FROM :"runtime_role",PUBLIC;
|
||||
CREATE FUNCTION evolution.abort_cellular_migration_freeze(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,id uuid)
|
||||
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
RETURN QUERY SELECT * FROM evolution.abort_cellular_migration_freeze_v47(w,r,holder,epoch,id);
|
||||
DELETE FROM evolution.cellular_migration_destination_pending x WHERE x.world_id=w AND x.request_id=id;
|
||||
END $$;
|
||||
REVOKE ALL ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) TO :"runtime_role";
|
||||
|
||||
CREATE OR REPLACE FUNCTION evolution.refuse_frozen_cellular_work() RETURNS trigger
|
||||
LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF EXISTS(SELECT 1 FROM evolution.cellular_migration_pending p WHERE p.world_id=NEW.world_id AND p.region_id=NEW.region_id)
|
||||
OR EXISTS(SELECT 1 FROM evolution.cellular_migration_destination_pending p WHERE p.world_id=NEW.world_id AND p.region_id=NEW.region_id) THEN
|
||||
RAISE EXCEPTION 'cellular endpoint frozen for migration' USING ERRCODE='EVL18';
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END $$;
|
||||
CREATE TRIGGER source_migration_endpoint_exclusive BEFORE INSERT ON evolution.cellular_migration_pending
|
||||
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work();
|
||||
INSERT INTO evolution.schema_migrations(version,sha256) VALUES(48,:'migration_sha_48');
|
||||
+14
-3
@@ -123,6 +123,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
SELECT :'history_v44' || ',45:' || :'migration_sha_45' AS history_v45 \gset
|
||||
SELECT :'history_v45' || ',46:' || :'migration_sha_46' AS history_v46 \gset
|
||||
SELECT :'history_v46' || ',47:' || :'migration_sha_47' AS history_v47 \gset
|
||||
SELECT :'history_v47' || ',48:' || :'migration_sha_48' AS history_v48 \gset
|
||||
SELECT :'history' = :'history_v22' AS at_v22,
|
||||
:'history' = :'history_v23' AS at_v23,
|
||||
:'history' = :'history_v24' AS at_v24,
|
||||
@@ -148,8 +149,10 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
:'history' = :'history_v44' AS at_v44,
|
||||
:'history' = :'history_v45' AS at_v45,
|
||||
:'history' = :'history_v46' AS at_v46,
|
||||
:'history' = :'history_v47' AS at_v47 \gset
|
||||
\if :at_v47
|
||||
:'history' = :'history_v47' AS at_v47,
|
||||
:'history' = :'history_v48' AS at_v48 \gset
|
||||
\if :at_v48
|
||||
\elif :at_v47
|
||||
\elif :at_v46
|
||||
\elif :at_v45
|
||||
\elif :at_v44
|
||||
@@ -537,7 +540,8 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
|
||||
coalesce(max(version), 0) = 44 AS at_v44,
|
||||
coalesce(max(version), 0) = 45 AS at_v45,
|
||||
coalesce(max(version), 0) = 46 AS at_v46,
|
||||
coalesce(max(version), 0) = 47 AS at_v47
|
||||
coalesce(max(version), 0) = 47 AS at_v47,
|
||||
coalesce(max(version), 0) = 48 AS at_v48
|
||||
FROM evolution.schema_migrations \gset
|
||||
\if :at_v24
|
||||
\ir 0025_cellular_v2_bundle.sql
|
||||
@@ -753,6 +757,7 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
|
||||
\elif :at_v45
|
||||
\elif :at_v46
|
||||
\elif :at_v47
|
||||
\elif :at_v48
|
||||
\else
|
||||
\echo 'Unknown or changed migration history after v43'
|
||||
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
|
||||
@@ -773,6 +778,10 @@ SELECT coalesce(max(version), 0) = 46 AS needs_v47 FROM evolution.schema_migrati
|
||||
\if :needs_v47
|
||||
\ir 0047_cellular_migration_freeze.sql
|
||||
\endif
|
||||
SELECT coalesce(max(version), 0) = 47 AS needs_v48 FROM evolution.schema_migrations \gset
|
||||
\if :needs_v48
|
||||
\ir 0048_cellular_migration_staging.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
|
||||
-- grants or SET-able memberships would bypass the fenced functions. Checks the
|
||||
@@ -792,6 +801,8 @@ BEGIN
|
||||
UNION
|
||||
SELECT m.member FROM pg_catalog.pg_auth_members m JOIN closure c ON m.roleid = c.oid
|
||||
), fenced(oid) AS (VALUES
|
||||
('evolution.stage_cellular_migration(evolution.u64,evolution.u64,uuid,evolution.u64,bytea,bytea,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.load_cellular_migration_stage(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.prepare_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,bytea)'::regprocedure::oid),
|
||||
('evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid)'::regprocedure::oid),
|
||||
('evolution.load_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
|
||||
|
||||
@@ -59,6 +59,8 @@ abort. The current backup copies all immutable namespace objects and the shared
|
||||
database snapshot; no GC or pruning is authorized. This does not qualify cellular
|
||||
history cloning/restore beyond the existing supported fixtures.
|
||||
|
||||
Destination ghost staging, atomic two-head publication, commit/abort races and
|
||||
activation remain subsequent work. This segment adds no automatic migration,
|
||||
Schema 48/ADR-0148 now adds [destination staging](cellular-migration-stage-v1.md)
|
||||
and atomic release of both pending pointers on source abort. Atomic two-head
|
||||
publication, commit/abort races and activation remain subsequent work.
|
||||
This segment adds no automatic migration,
|
||||
atlas, cross-edge movement, environmental coupling or census/extinction rule.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Biological destination staging
|
||||
|
||||
Authority: ADR-0148, SQL schema 48. This extends
|
||||
[source preparation](cellular-migration-freeze-v1.md) with destination-authorized
|
||||
read-only evidence. Biological/checkpoint/trace versions retain their meanings.
|
||||
There is no new active checkpoint profile or migration result receipt.
|
||||
|
||||
The caller supplies the exact retained migration trace, both predecessor
|
||||
manifests and current destination authority. The adapter verifies manifests and
|
||||
payload SHA-256/length, replays the complete pair, copies the source predecessor
|
||||
into the destination namespace and uploads/reads back the derived destination
|
||||
candidate. Its content address and byte length form the exact retry input with
|
||||
the canonical trace. Unpublished uploads remain retained orphans on failure.
|
||||
|
||||
SQL independently checks the retained preparation's exact trace, current
|
||||
destination lease, both exact heads and source pending pointer. Both regional
|
||||
lease rows are locked in increasing region order. Destination pending intervals,
|
||||
unpublished mutation choices, successful outstanding life reservations and any
|
||||
other local migration role block first staging. Moving the destination before
|
||||
staging refuses the request; no implicit rebase is performed.
|
||||
|
||||
Schema 48 retains immutable `cellular_migration_stages`, keyed by the preparation
|
||||
request UUID, with ghost SHA-256, byte length and staging holder/epoch. One
|
||||
`cellular_migration_destination_pending` pointer per region bounds active lookup.
|
||||
The source/destination pending roles are exclusive. Existing indexed publication
|
||||
and interval/player-reservation guards cover either role.
|
||||
|
||||
The source owns all outgoing organisms and material until a future paired
|
||||
commit. The staged candidate owns nothing. Neither region head changes. No tick,
|
||||
feeding, passive healing, movement, birth or death is performed by staging. In
|
||||
particular, a ghost cannot supply an owned census or terminal extinction count.
|
||||
|
||||
Destination recovery uses its own namespace's two predecessor capsules and ghost.
|
||||
It rederives the whole pair and compares the complete destination bytes, including
|
||||
membership, policy, biology and history. Missing or same-length corrupt objects
|
||||
fail strong binding; a valid object with false biology fails semantic replay.
|
||||
There is no fallback to the live source or an older head.
|
||||
|
||||
Both first staging and replacement pause host readiness/advancement. Unknown
|
||||
replies retain the exact trace/content address; reconciliation keeps the same
|
||||
request and polls paused work at most once per second. Normal stepping has no
|
||||
per-tick migration query. The existing fixed unlabelled migration-freeze metric
|
||||
covers either role; no request/organism labels are introduced.
|
||||
|
||||
Explicit current-source abort appends the existing immutable outcome and clears
|
||||
both pointers in the same transaction. It serializes with staging on the source
|
||||
lease row. If abort wins first, new staging is refused. If stage wins first,
|
||||
abort releases both pauses. Exact retries of a retained stage after abort return
|
||||
`aborted` and do not reopen either role. Neither expiry, drain nor timeout implies
|
||||
abort. Keep staged objects/capsules and journals after abort; no GC is authorized.
|
||||
|
||||
Atomic paired result publication, its immutable common receipt, commit/abort
|
||||
races, recovery/activation and linked ecological geometry remain subsequent
|
||||
work. This segment enables no automatic movement, route, atlas or deployment.
|
||||
@@ -252,4 +252,179 @@ load_cellular_migration_freeze(Postgres& db, const FilesystemObjectStore& object
|
||||
return fail(RecoveryError::history);
|
||||
return result;
|
||||
}
|
||||
|
||||
namespace {
|
||||
constexpr auto stage_columns = "encode(trace,'hex'),encode(source_manifest,'hex'),"
|
||||
"encode(destination_manifest,'hex'),aborted::text,"
|
||||
"encode(ghost_sha256,'hex'),ghost_bytes::text";
|
||||
struct StageRecord {
|
||||
Record cut;
|
||||
VerifiedObject ghost;
|
||||
};
|
||||
std::optional<StageRecord> parse_stage(const PgRows::value_type& row) {
|
||||
if (row.size() != 6)
|
||||
return std::nullopt;
|
||||
auto cut = parse(PgRows::value_type(row.begin(), row.begin() + 4));
|
||||
const auto digest = unhex(row[4].value_or(""), 32);
|
||||
const auto size = row[5].value_or("");
|
||||
std::uint64_t bytes{};
|
||||
const auto [end, error] = std::from_chars(size.data(), size.data() + size.size(), bytes);
|
||||
if (!cut || !digest || digest->size() != 32 || error != std::errc{} ||
|
||||
end != size.data() + size.size() || bytes == 0)
|
||||
return std::nullopt;
|
||||
VerifiedObject ghost{{}, bytes};
|
||||
std::ranges::copy(*digest, ghost.sha256.begin());
|
||||
return StageRecord{std::move(*cut), ghost};
|
||||
}
|
||||
std::optional<CellularMigrationStage> checked_stage(const StageRecord& record, RegionKey key,
|
||||
const LeaseGrant& lease) {
|
||||
const auto trace = server::decode_migration_trace(record.cut.trace);
|
||||
if (!trace || key != RegionKey{trace->world, trace->request.destination.region})
|
||||
return std::nullopt;
|
||||
const auto freeze =
|
||||
checked_record(record.cut, {trace->world, trace->request.source.region}, lease);
|
||||
const auto destination = decode_manifest(record.cut.destination);
|
||||
if (!freeze || !destination || record.ghost.sha256 == Sha256Digest{} ||
|
||||
record.ghost.bytes > max_payload_bytes(destination->profile))
|
||||
return std::nullopt;
|
||||
return CellularMigrationStage{*freeze, record.ghost};
|
||||
}
|
||||
std::optional<std::vector<std::uint8_t>>
|
||||
ghost_candidate(const CheckpointManifest& s, const CheckpointManifest& d,
|
||||
std::span<const std::uint8_t> source, std::span<const std::uint8_t> destination,
|
||||
const server::CellularMigrationTrace& trace) {
|
||||
if (s.profile == CheckpointProfile::cellular_lives_v1) {
|
||||
auto src = restore_cellular_payload(source, s),
|
||||
dst = restore_cellular_payload(destination, d);
|
||||
if (!src || !dst)
|
||||
return std::nullopt;
|
||||
auto pair = server::replay_migration_trace(**src, **dst, trace.request, trace);
|
||||
if (!pair)
|
||||
return std::nullopt;
|
||||
return pair->destination->snapshot();
|
||||
}
|
||||
auto src = restore_population_payload(source, s),
|
||||
dst = restore_population_payload(destination, d);
|
||||
if (!src || !dst)
|
||||
return std::nullopt;
|
||||
auto pair = server::replay_migration_trace(**src, **dst, trace.request, trace);
|
||||
if (!pair)
|
||||
return std::nullopt;
|
||||
return pair->destination->snapshot();
|
||||
}
|
||||
CellularMigrationStageFailure stage_failed(PublicationError kind, LeaseStoreFailure store = {},
|
||||
ObjectError object = {},
|
||||
CellularMigrationStagePrepared prepared = {}) {
|
||||
return {kind, store, object, std::move(prepared)};
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<CellularMigrationStage, CellularMigrationStageFailure>
|
||||
retry_cellular_migration_stage(Postgres& db, const LeaseGrant& lease,
|
||||
const CellularMigrationStagePrepared& prepared,
|
||||
milliseconds timeout) {
|
||||
const auto trace = server::decode_migration_trace(prepared.trace);
|
||||
if (!trace || prepared.key != RegionKey{trace->world, trace->request.destination.region} ||
|
||||
lease.frame != trace->frame || prepared.ghost.sha256 == Sha256Digest{} ||
|
||||
prepared.ghost.bytes == 0)
|
||||
return std::unexpected(stage_failed(PublicationError::candidate));
|
||||
const std::array<std::string, 7> args{std::to_string(prepared.key.world),
|
||||
std::to_string(prepared.key.region),
|
||||
uuid(lease.holder),
|
||||
std::to_string(lease.epoch),
|
||||
"\\x" + hex(prepared.trace),
|
||||
"\\x" + hex(prepared.ghost.sha256),
|
||||
std::to_string(prepared.ghost.bytes)};
|
||||
auto rows = db.query(std::string("SELECT ") + stage_columns +
|
||||
" FROM evolution.stage_cellular_migration($1::evolution.u64,"
|
||||
"$2::evolution.u64,$3::uuid,$4::evolution.u64,$5::bytea,$6::bytea,"
|
||||
"$7::evolution.u64)",
|
||||
args, timeout);
|
||||
if (!rows) {
|
||||
const auto store = failure(rows.error());
|
||||
return std::unexpected(stage_failed(store.kind == LeaseStoreError::unknown
|
||||
? PublicationError::unknown
|
||||
: PublicationError::rejected,
|
||||
store, {}, prepared));
|
||||
}
|
||||
const auto record = rows->size() == 1 ? parse_stage(rows->front()) : std::nullopt;
|
||||
const auto result = record ? checked_stage(*record, prepared.key, lease) : std::nullopt;
|
||||
if (!result || record->cut.trace != prepared.trace || result->ghost != prepared.ghost)
|
||||
return std::unexpected(stage_failed(PublicationError::unknown, {}, {}, prepared));
|
||||
return *result;
|
||||
}
|
||||
|
||||
std::expected<CellularMigrationStage, CellularMigrationStageFailure>
|
||||
stage_cellular_migration(Postgres& db, const FilesystemObjectStore& source_objects,
|
||||
const FilesystemObjectStore& destination_objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& source, const CommittedCheckpoint& destination,
|
||||
const server::CellularMigrationTrace& trace, milliseconds timeout,
|
||||
const PublishHooks& hooks) {
|
||||
const auto s = decode_manifest(source.manifest), d = decode_manifest(destination.manifest);
|
||||
if (!s || !d || !bound(source, *s) || !bound(destination, *d) ||
|
||||
!cuts(*s, *d, source.head.digest, destination.head.digest, trace) ||
|
||||
lease.frame != d->frame)
|
||||
return std::unexpected(stage_failed(PublicationError::predecessor));
|
||||
auto src = source_objects.get({s->payload_sha256, s->payload_bytes});
|
||||
auto dst = destination_objects.get({d->payload_sha256, d->payload_bytes});
|
||||
if (!src || !dst)
|
||||
return std::unexpected(
|
||||
stage_failed(PublicationError::predecessor, {}, !src ? src.error() : dst.error()));
|
||||
const auto candidate = ghost_candidate(*s, *d, *src, *dst, trace);
|
||||
const auto encoded = server::encode_migration_trace(trace);
|
||||
if (!candidate || !encoded)
|
||||
return std::unexpected(stage_failed(PublicationError::candidate));
|
||||
auto capsule = destination_objects.put(*src);
|
||||
if (!capsule)
|
||||
return std::unexpected(stage_failed(PublicationError::upload, {}, capsule.error()));
|
||||
auto capsule_bytes = destination_objects.get(*capsule);
|
||||
if (!capsule_bytes || *capsule_bytes != *src ||
|
||||
*capsule != VerifiedObject{s->payload_sha256, s->payload_bytes})
|
||||
return std::unexpected(
|
||||
stage_failed(PublicationError::verify, {},
|
||||
capsule_bytes ? ObjectError::corrupt : capsule_bytes.error()));
|
||||
auto ghost = destination_objects.put(*candidate);
|
||||
if (!ghost)
|
||||
return std::unexpected(stage_failed(PublicationError::upload, {}, ghost.error()));
|
||||
auto readback = destination_objects.get(*ghost);
|
||||
if (!readback || *readback != *candidate)
|
||||
return std::unexpected(stage_failed(PublicationError::verify, {},
|
||||
readback ? ObjectError::corrupt : readback.error()));
|
||||
if (hooks.before_transaction)
|
||||
hooks.before_transaction();
|
||||
return retry_cellular_migration_stage(db, lease, {d->key, *encoded, *ghost}, timeout);
|
||||
}
|
||||
|
||||
std::expected<std::optional<CellularMigrationStage>, RecoveryFailure>
|
||||
load_cellular_migration_stage(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
const LeaseGrant& lease, milliseconds timeout) {
|
||||
const auto fail = [](RecoveryError kind) { return std::unexpected(RecoveryFailure{kind, {}}); };
|
||||
const std::array<std::string, 4> args{std::to_string(key.world), std::to_string(key.region),
|
||||
uuid(lease.holder), std::to_string(lease.epoch)};
|
||||
auto rows = db.query(std::string("SELECT ") + stage_columns +
|
||||
" FROM evolution.load_cellular_migration_stage($1::evolution.u64,"
|
||||
"$2::evolution.u64,$3::uuid,$4::evolution.u64)",
|
||||
args, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected(RecoveryFailure{RecoveryError::store, failure(rows.error())});
|
||||
if (rows->empty())
|
||||
return std::optional<CellularMigrationStage>{};
|
||||
const auto record = rows->size() == 1 ? parse_stage(rows->front()) : std::nullopt;
|
||||
auto result = record ? checked_stage(*record, key, lease) : std::nullopt;
|
||||
if (!result || result->freeze.aborted)
|
||||
return fail(RecoveryError::history);
|
||||
const auto s = decode_manifest(record->cut.source),
|
||||
d = decode_manifest(record->cut.destination);
|
||||
if (!s || !d)
|
||||
return fail(RecoveryError::manifest);
|
||||
auto src = objects.get({s->payload_sha256, s->payload_bytes});
|
||||
auto dst = objects.get({d->payload_sha256, d->payload_bytes});
|
||||
auto ghost = objects.get(result->ghost);
|
||||
if (!src || !dst || !ghost)
|
||||
return fail(RecoveryError::tail);
|
||||
const auto candidate = ghost_candidate(*s, *d, *src, *dst, result->freeze.trace);
|
||||
if (!candidate || *candidate != *ghost)
|
||||
return fail(RecoveryError::history);
|
||||
return result;
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -416,7 +416,8 @@ void CellularRegionHost::refresh() {
|
||||
status_.ready = status_.phase == HostPhase::running && state_ && committed_ && grant &&
|
||||
authority_.may_originate(LeaseClock::now()) && !interval_reload_ &&
|
||||
!interval_request_.has_value() && !migration_reload_ && !migration_frozen_ &&
|
||||
!migration_request_.has_value() && !migration_abort_.has_value();
|
||||
!migration_request_.has_value() && !migration_stage_request_.has_value() &&
|
||||
!migration_abort_.has_value();
|
||||
status_.migration_frozen = migration_frozen_;
|
||||
const auto& interval = admitted_interval_;
|
||||
status_.ecological_pending_ticks =
|
||||
@@ -430,7 +431,8 @@ void CellularRegionHost::refresh() {
|
||||
}
|
||||
|
||||
void CellularRegionHost::reconcile_migration_freeze() {
|
||||
if ((!migration_reload_ && !migration_frozen_ && !migration_request_ && !migration_abort_) ||
|
||||
if ((!migration_reload_ && !migration_frozen_ && !migration_request_ &&
|
||||
!migration_stage_request_ && !migration_abort_) ||
|
||||
LeaseClock::now() < migration_poll_at_)
|
||||
return;
|
||||
const auto grant = authority_.grant();
|
||||
@@ -460,6 +462,30 @@ void CellularRegionHost::reconcile_migration_freeze() {
|
||||
}
|
||||
migration_request_.reset();
|
||||
}
|
||||
if (migration_stage_request_) {
|
||||
const auto staged = persistence::retry_cellular_migration_stage(
|
||||
*db_, *grant, *migration_stage_request_, config_.lease.commit_timeout);
|
||||
if (!staged) {
|
||||
if (staged.error().kind == persistence::PublicationError::unknown) {
|
||||
db_.reset();
|
||||
++status_.unknown_outcomes;
|
||||
return;
|
||||
}
|
||||
// A concurrent source abort can win before the uncertain stage.
|
||||
// Reload authoritative pointers before making the region ready.
|
||||
if (staged.error().store.kind != LeaseStoreError::commit_conflict) {
|
||||
authority_.fence();
|
||||
status_.phase = staged.error().store.kind == LeaseStoreError::fenced ||
|
||||
staged.error().store.kind == LeaseStoreError::head_mismatch
|
||||
? HostPhase::acquiring
|
||||
: HostPhase::quarantined;
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
return;
|
||||
}
|
||||
}
|
||||
migration_stage_request_.reset();
|
||||
}
|
||||
if (migration_abort_) {
|
||||
auto result = persistence::abort_cellular_migration_freeze(
|
||||
*db_, config_.key, *grant, *migration_abort_, config_.lease.commit_timeout);
|
||||
@@ -492,9 +518,65 @@ void CellularRegionHost::reconcile_migration_freeze() {
|
||||
return;
|
||||
}
|
||||
migration_frozen_ = frozen->has_value();
|
||||
if (!migration_frozen_) {
|
||||
const auto staged = persistence::load_cellular_migration_stage(
|
||||
*db_, *objects_, config_.key, *grant, config_.lease.commit_timeout);
|
||||
if (!staged) {
|
||||
migration_reload_ = true;
|
||||
if (staged.error().kind == persistence::RecoveryError::store) {
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
} else {
|
||||
authority_.fence();
|
||||
status_.phase = HostPhase::quarantined;
|
||||
}
|
||||
return;
|
||||
}
|
||||
migration_frozen_ = staged->has_value();
|
||||
}
|
||||
migration_reload_ = false;
|
||||
}
|
||||
|
||||
std::expected<void, CellularPublishFailure>
|
||||
CellularRegionHost::stage_migration(const persistence::CommittedCheckpoint& source,
|
||||
const persistence::FilesystemObjectStore& source_objects,
|
||||
const CellularMigrationTrace& trace) {
|
||||
const auto grant = authority_.grant();
|
||||
if (!grant.has_value() || !status_.ready || pending_ || !state_ || !committed_ ||
|
||||
!queued_births_.empty() || !queued_mutations_.empty() || mutations_uncertain_ ||
|
||||
!pending_endings_.empty() || interval_request_ || admitted_interval_ || interval_reload_ ||
|
||||
!population_interval_safe(*state_) || !authority_.may_begin_commit(LeaseClock::now()) ||
|
||||
!database())
|
||||
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
|
||||
migration_reload_ = true;
|
||||
refresh();
|
||||
const auto staged =
|
||||
persistence::stage_cellular_migration(*db_, source_objects, *objects_, *grant, source,
|
||||
*committed_, trace, config_.lease.commit_timeout);
|
||||
if (!staged) {
|
||||
if (staged.error().kind == persistence::PublicationError::unknown) {
|
||||
migration_stage_request_ = staged.error().prepared;
|
||||
db_.reset();
|
||||
++status_.unknown_outcomes;
|
||||
} else if (staged.error().store.kind == LeaseStoreError::fenced ||
|
||||
staged.error().store.kind == LeaseStoreError::head_mismatch) {
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
status_.phase = HostPhase::acquiring;
|
||||
}
|
||||
migration_poll_at_ = {};
|
||||
refresh();
|
||||
return std::unexpected(CellularPublishFailure{CellularHostError::publication,
|
||||
staged.error().kind, staged.error().store,
|
||||
staged.error().object});
|
||||
}
|
||||
migration_poll_at_ = {};
|
||||
reconcile_migration_freeze();
|
||||
refresh();
|
||||
return {};
|
||||
}
|
||||
|
||||
std::expected<void, CellularPublishFailure> CellularRegionHost::prepare_migration_freeze(
|
||||
const persistence::CommittedCheckpoint& destination,
|
||||
const persistence::FilesystemObjectStore& destination_objects,
|
||||
@@ -573,7 +655,8 @@ const HostStatus& CellularRegionHost::step() {
|
||||
acquire();
|
||||
if (!pending_ && state_ && status_.phase == HostPhase::running)
|
||||
reconcile_migration_freeze();
|
||||
if (migration_reload_ || migration_frozen_ || migration_request_ || migration_abort_) {
|
||||
if (migration_reload_ || migration_frozen_ || migration_request_ || migration_stage_request_ ||
|
||||
migration_abort_) {
|
||||
refresh();
|
||||
return status_;
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
#include "evo/server/cellular_region_host.hpp"
|
||||
#include "fixtures/cellular_ecology.hpp"
|
||||
|
||||
#include <barrier>
|
||||
#include <csignal>
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
@@ -868,6 +869,155 @@ void migration_freezes(Fixture& f) {
|
||||
!take(retry_cellular_migration_freeze(*f.db, f.lease, prepared, 5s), "duplicate freeze")
|
||||
.aborted,
|
||||
"duplicate freeze lost its state");
|
||||
const auto stage = [&](const CellularMigrationTrace& trace,
|
||||
std::chrono::milliseconds timeout = 5s) {
|
||||
return stage_cellular_migration(*f.db, *f.objects, *destination_objects,
|
||||
destination_lease, f.head, destination_head, trace,
|
||||
timeout);
|
||||
};
|
||||
// The optimistic destination cut must still be exact at first staging.
|
||||
take(f.db->query("BEGIN", {}, 5s), "move destination before stage transaction");
|
||||
if (representation == 0)
|
||||
take(migrate_cellular_to_population(*f.db, *destination_objects, destination_lease,
|
||||
destination_head, f.id(), f.build, {}, 5s),
|
||||
"temporarily move destination profile");
|
||||
else
|
||||
take(materialize_population_profile(*f.db, *destination_objects, destination_lease,
|
||||
destination_head, f.id(), f.build, 5s),
|
||||
"temporarily move population destination");
|
||||
const auto moved = stage(frozen.trace);
|
||||
take(f.db->query("ROLLBACK", {}, 5s), "rollback moved destination fixture");
|
||||
require(!moved && moved.error().store.kind == LeaseStoreError::head_mismatch,
|
||||
"stage silently rebased a moved destination");
|
||||
if (representation == 1) {
|
||||
take(f.db->query("BEGIN", {}, 5s), "stage pending interval transaction");
|
||||
take(admit_population_interval(*f.db, destination_key, destination_lease,
|
||||
destination_head.head,
|
||||
destination_head.head.end_tick + 20, 5s),
|
||||
"destination admitted work before staging");
|
||||
const auto pending = stage(frozen.trace);
|
||||
take(f.db->query("ROLLBACK", {}, 5s), "rollback destination interval fixture");
|
||||
require(!pending && pending.error().store.kind == LeaseStoreError::commit_conflict,
|
||||
"stage stranded destination interval work");
|
||||
}
|
||||
const auto staged = take(stage(frozen.trace), "durable destination staging");
|
||||
const CellularMigrationStagePrepared stage_request{destination_key, prepared.trace,
|
||||
staged.ghost};
|
||||
require(
|
||||
!staged.freeze.aborted &&
|
||||
take(take(load_cellular_migration_stage(*f.db, *destination_objects,
|
||||
destination_key, destination_lease, 5s),
|
||||
"recover destination ghost"),
|
||||
"missing staged destination")
|
||||
.ghost == staged.ghost &&
|
||||
take(load_region_checkpoint(*f.db, destination_key, 5s),
|
||||
"staged destination still owns original head")
|
||||
.head == destination_before.head &&
|
||||
take(retry_cellular_migration_stage(*f.db, destination_lease, stage_request, 5s),
|
||||
"duplicate stage")
|
||||
.ghost == staged.ghost,
|
||||
"staging activated a ghost or lost exact replay");
|
||||
auto changed_stage = stage_request;
|
||||
++changed_stage.ghost.bytes;
|
||||
const auto changed =
|
||||
retry_cellular_migration_stage(*f.db, destination_lease, changed_stage, 5s);
|
||||
require(!changed && changed.error().store.kind == LeaseStoreError::commit_conflict,
|
||||
"stage accepted conflicting immutable ghost bytes");
|
||||
auto reverse = frozen.trace;
|
||||
reverse.request.request = f.id();
|
||||
std::swap(reverse.request.source, reverse.request.destination);
|
||||
const auto overlapping = retry_cellular_migration_freeze(
|
||||
*f.db, destination_lease,
|
||||
{destination_key, take(encode_migration_trace(reverse), "overlapping source trace")},
|
||||
5s);
|
||||
require(!overlapping && overlapping.error().store.kind == LeaseStoreError::commit_conflict,
|
||||
"staged destination simultaneously prepared an outgoing workflow");
|
||||
take(f.db->query("BEGIN", {}, 5s), "staged destination life reservation transaction");
|
||||
Uuid new_account{};
|
||||
new_account.back() = 5;
|
||||
const auto staged_control =
|
||||
take(claim_account_control(*f.db, destination_key, destination_lease, new_account, 1,
|
||||
9950, 60000ms, 5s),
|
||||
"staged destination reservation current control");
|
||||
const auto staged_life =
|
||||
reserve_primitive_life(*f.db, destination_key, destination_lease, new_account,
|
||||
staged_control.generation, 1, 1, 5s);
|
||||
take(f.db->query("ROLLBACK", {}, 5s), "rollback staged life reservation fixture");
|
||||
require(!staged_life &&
|
||||
std::string(staged_life.error().cause.sqlstate.data(), 5) == "EVL18",
|
||||
"staged destination admitted a new player life reservation");
|
||||
if (representation == 0) {
|
||||
CellularBatch inputs{{f.key.world, 2, 1, destination_head.head.commit,
|
||||
destination_head.head.digest, destination->tick(),
|
||||
destination->tick() + 1},
|
||||
{{}}};
|
||||
const auto advanced = take(replay_cellular_batch(*destination, inputs),
|
||||
"detached staged destination future");
|
||||
const auto denied =
|
||||
publish_cellular_batch(*f.db, *destination_objects, destination_lease,
|
||||
destination_head, inputs, *advanced, f.id(), f.build, 5s);
|
||||
require(!denied &&
|
||||
std::string(denied.error().store.cause.sqlstate.data(), 5) == "EVL18",
|
||||
"staged destination published an ordinary future");
|
||||
} else {
|
||||
require(!admit_population_interval(*f.db, destination_key, destination_lease,
|
||||
destination_head.head,
|
||||
destination_head.head.end_tick + 20, 5s) &&
|
||||
!materialize_population_profile(*f.db, *destination_objects,
|
||||
destination_lease, destination_head, f.id(),
|
||||
f.build, 5s),
|
||||
"staged destination progressed or changed profile");
|
||||
}
|
||||
const auto ghost_path =
|
||||
destination_objects->directory() / hexadecimal(staged.ghost.sha256, false);
|
||||
const auto ghost_backup = ghost_path.string() + ".stage-test";
|
||||
std::filesystem::rename(ghost_path, ghost_backup);
|
||||
const auto missing_ghost = load_cellular_migration_stage(
|
||||
*f.db, *destination_objects, destination_key, destination_lease, 5s);
|
||||
std::filesystem::rename(ghost_backup, ghost_path);
|
||||
require(!missing_ghost && missing_ghost.error().kind == RecoveryError::tail,
|
||||
"missing ghost fell back to the active destination");
|
||||
char ghost_first{};
|
||||
{
|
||||
std::fstream file(ghost_path, std::ios::binary | std::ios::in | std::ios::out);
|
||||
file.get(ghost_first);
|
||||
file.seekp(0);
|
||||
file.put(static_cast<char>(static_cast<unsigned char>(ghost_first) ^ 1U));
|
||||
file.flush();
|
||||
require(file.good(), "same-length ghost corruption fixture");
|
||||
}
|
||||
const auto corrupt_ghost = load_cellular_migration_stage(
|
||||
*f.db, *destination_objects, destination_key, destination_lease, 5s);
|
||||
{
|
||||
std::fstream file(ghost_path, std::ios::binary | std::ios::in | std::ios::out);
|
||||
file.put(ghost_first);
|
||||
file.flush();
|
||||
require(file.good(), "restore corrupt ghost fixture");
|
||||
}
|
||||
require(!corrupt_ghost && corrupt_ghost.error().kind == RecoveryError::tail,
|
||||
"same-length corrupt ghost bypassed digest verification");
|
||||
const auto source_manifest = take(decode_manifest(f.head.manifest), "stage source capsule");
|
||||
const auto source_capsule_path =
|
||||
destination_objects->directory() / hexadecimal(source_manifest.payload_sha256, false);
|
||||
const auto source_capsule_backup = source_capsule_path.string() + ".stage-test";
|
||||
std::filesystem::rename(source_capsule_path, source_capsule_backup);
|
||||
const auto missing_source = load_cellular_migration_stage(
|
||||
*f.db, *destination_objects, destination_key, destination_lease, 5s);
|
||||
std::filesystem::rename(source_capsule_backup, source_capsule_path);
|
||||
require(!missing_source && missing_source.error().kind == RecoveryError::tail,
|
||||
"destination recovery used the live source instead of its retained capsule");
|
||||
const auto old_destination_lease = destination_lease;
|
||||
take(release_region_lease(*f.db, destination_key, destination_lease, 5s),
|
||||
"release staged destination owner");
|
||||
destination_lease = take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s),
|
||||
"replacement staged destination owner");
|
||||
require(
|
||||
take(load_cellular_migration_stage(*f.db, *destination_objects, destination_key,
|
||||
destination_lease, 5s),
|
||||
"new epoch ghost replay")
|
||||
.has_value() &&
|
||||
!retry_cellular_migration_stage(*f.db, old_destination_lease, stage_request, 5s),
|
||||
"replacement activated the ghost or stale destination staged it");
|
||||
auto conflict_trace = frozen.trace;
|
||||
++conflict_trace.transferred.energy.value;
|
||||
auto conflict = retry_cellular_migration_freeze(
|
||||
@@ -960,6 +1110,15 @@ void migration_freezes(Fixture& f) {
|
||||
"aborted pointer load")
|
||||
.has_value(),
|
||||
"abort was not durable or preparation retry reopened it");
|
||||
require(
|
||||
!take(load_cellular_migration_stage(*f.db, *destination_objects, destination_key,
|
||||
destination_lease, 5s),
|
||||
"aborted destination pointer")
|
||||
.has_value() &&
|
||||
take(retry_cellular_migration_stage(*f.db, destination_lease, stage_request, 5s),
|
||||
"retry aborted stage")
|
||||
.freeze.aborted,
|
||||
"source abort stranded the destination or stage retry reopened its pointer");
|
||||
require(take(load_region_checkpoint(*f.db, f.key, 5s), "aborted source head").head ==
|
||||
source_before.head &&
|
||||
f.raw->snapshot() == source_bytes,
|
||||
@@ -1004,6 +1163,76 @@ void migration_freezes(Fixture& f) {
|
||||
5s),
|
||||
"abort reconciled freeze");
|
||||
|
||||
const auto false_stage_freeze = take(prepare(request_for()), "false ghost preparation");
|
||||
const auto original_destination =
|
||||
take(decode_manifest(destination_head.manifest), "false ghost existing object");
|
||||
const CellularMigrationStagePrepared false_ghost{
|
||||
destination_key,
|
||||
take(encode_migration_trace(false_stage_freeze.trace), "false ghost trace"),
|
||||
{original_destination.payload_sha256, original_destination.payload_bytes}};
|
||||
take(retry_cellular_migration_stage(*f.db, destination_lease, false_ghost, 5s),
|
||||
"metadata-only false ghost stage");
|
||||
const auto false_ghost_replay = load_cellular_migration_stage(
|
||||
*f.db, *destination_objects, destination_key, destination_lease, 5s);
|
||||
require(!false_ghost_replay && false_ghost_replay.error().kind == RecoveryError::history,
|
||||
"stage recovery accepted a real object with the wrong complete biology");
|
||||
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease,
|
||||
false_stage_freeze.trace.request.request, 5s),
|
||||
"abort false ghost evidence");
|
||||
|
||||
const auto uncertain_stage_freeze =
|
||||
take(prepare(request_for()), "unknown stage preparation");
|
||||
take(locker->query("BEGIN", {}, 5s), "stage ambiguity lock");
|
||||
take(locker->query("SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND "
|
||||
"region_id=2 FOR UPDATE",
|
||||
{}, 5s),
|
||||
"hold stage reply");
|
||||
auto stage_timed = connection();
|
||||
const auto stage_unknown = stage_cellular_migration(
|
||||
*stage_timed, *f.objects, *destination_objects, destination_lease, f.head,
|
||||
destination_head, uncertain_stage_freeze.trace, 300ms);
|
||||
require(!stage_unknown && stage_unknown.error().kind == PublicationError::unknown &&
|
||||
!stage_unknown.error().prepared.trace.empty(),
|
||||
"stage ambiguity lost exact bytes");
|
||||
take(locker->query("ROLLBACK", {}, 5s), "release stage ambiguity");
|
||||
take(retry_cellular_migration_stage(*f.db, destination_lease,
|
||||
stage_unknown.error().prepared, 5s),
|
||||
"resolve unknown stage");
|
||||
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease,
|
||||
uncertain_stage_freeze.trace.request.request, 5s),
|
||||
"abort reconciled stage");
|
||||
|
||||
for (unsigned race = 0; race < 4; ++race) {
|
||||
const auto race_freeze = take(prepare(request_for()), "stage race preparation");
|
||||
auto stage_db = connection(), abort_db = connection();
|
||||
std::barrier start(2);
|
||||
std::optional<std::expected<CellularMigrationStage, CellularMigrationStageFailure>>
|
||||
staged_race;
|
||||
std::optional<std::expected<CellularMigrationFreeze, LeaseStoreFailure>> aborted_race;
|
||||
std::thread staging([&] {
|
||||
start.arrive_and_wait();
|
||||
staged_race = stage_cellular_migration(*stage_db, *f.objects, *destination_objects,
|
||||
destination_lease, f.head, destination_head,
|
||||
race_freeze.trace, 5s);
|
||||
});
|
||||
std::thread aborting([&] {
|
||||
start.arrive_and_wait();
|
||||
aborted_race = abort_cellular_migration_freeze(
|
||||
*abort_db, f.key, f.lease, race_freeze.trace.request.request, 5s);
|
||||
});
|
||||
staging.join();
|
||||
aborting.join();
|
||||
require(aborted_race.has_value() && aborted_race->has_value() &&
|
||||
staged_race.has_value() &&
|
||||
(staged_race->has_value() ||
|
||||
staged_race->error().store.kind == LeaseStoreError::commit_conflict) &&
|
||||
!take(load_cellular_migration_stage(*f.db, *destination_objects,
|
||||
destination_key, destination_lease, 5s),
|
||||
"stage race final pointer")
|
||||
.has_value(),
|
||||
"stage/abort race left an active ghost or stranded pause");
|
||||
}
|
||||
|
||||
// Both sides of preparation's transaction boundary, with real process death.
|
||||
for (unsigned point = 0; point < 2; ++point) {
|
||||
const auto killed_request = request_for();
|
||||
@@ -1037,6 +1266,38 @@ void migration_freezes(Fixture& f) {
|
||||
5s),
|
||||
"abort killed preparation");
|
||||
}
|
||||
for (unsigned point = 0; point < 2; ++point) {
|
||||
const auto killed = take(prepare(request_for()), "stage process-loss preparation");
|
||||
const auto child = ::fork();
|
||||
require(child >= 0, "stage publisher fork");
|
||||
if (child == 0) {
|
||||
try {
|
||||
auto db = connection();
|
||||
PublishHooks hooks;
|
||||
if (point == 0)
|
||||
hooks.before_transaction = [] { ::kill(::getpid(), SIGKILL); };
|
||||
if (!stage_cellular_migration(*db, *f.objects, *destination_objects,
|
||||
destination_lease, f.head, destination_head,
|
||||
killed.trace, 5s, hooks))
|
||||
::_exit(44);
|
||||
::kill(::getpid(), SIGKILL);
|
||||
} catch (...) {
|
||||
::_exit(45);
|
||||
}
|
||||
::_exit(46);
|
||||
}
|
||||
int status{};
|
||||
require(::waitpid(child, &status, 0) == child && WIFSIGNALED(status) &&
|
||||
WTERMSIG(status) == SIGKILL &&
|
||||
take(load_cellular_migration_stage(*f.db, *destination_objects,
|
||||
destination_key, destination_lease, 5s),
|
||||
"recover killed stage")
|
||||
.has_value() == (point == 1),
|
||||
"stage process loss changed ownership or lost the durable pause");
|
||||
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease,
|
||||
killed.trace.request.request, 5s),
|
||||
"abort killed stage");
|
||||
}
|
||||
}
|
||||
// The owning host must retain the freeze through drain/recreation with entry off.
|
||||
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release fixture to frozen host");
|
||||
@@ -1107,6 +1368,64 @@ void migration_freezes(Fixture& f) {
|
||||
host->drain();
|
||||
f.lease = take(acquire_region_lease(*f.db, f.key, f.build, 60000ms, 5s),
|
||||
"return freeze fixture authority");
|
||||
const auto destination_host_freeze =
|
||||
take(prepare_cellular_migration_freeze(*f.db, *f.objects, *destination_objects, f.lease,
|
||||
f.head, destination_head, request_for(), 5s),
|
||||
"destination host source preparation");
|
||||
take(release_region_lease(*f.db, destination_key, destination_lease, 5s),
|
||||
"release destination to owning host");
|
||||
const auto make_destination_host = [&] {
|
||||
auto objects = take(FilesystemObjectStore::open(environment("PGHOST"), destination_key,
|
||||
max_cellular_population_bytes),
|
||||
"destination host objects");
|
||||
CellularHostConfig config{destination_key, 1, {}, f.build, {f.build}};
|
||||
config.lease.commit_timeout = 300ms;
|
||||
return take(
|
||||
CellularRegionHost::create(
|
||||
config,
|
||||
[] { return std::expected<std::unique_ptr<Postgres>, PgFailure>{connection()}; },
|
||||
std::move(objects)),
|
||||
"destination staging host");
|
||||
};
|
||||
auto destination_host = make_destination_host();
|
||||
require(destination_host->step().ready, "destination host before staging");
|
||||
const auto destination_bytes = destination_host->state()->snapshot();
|
||||
take(locker->query("BEGIN", {}, 5s), "host stage ambiguity transaction");
|
||||
take(locker->query("SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND "
|
||||
"region_id=1 FOR UPDATE",
|
||||
{}, 5s),
|
||||
"hold owning destination stage");
|
||||
const auto host_stage_unknown =
|
||||
destination_host->stage_migration(f.head, *f.objects, destination_host_freeze.trace);
|
||||
require(!host_stage_unknown && !destination_host->status().ready &&
|
||||
destination_host->state()->snapshot() == destination_bytes,
|
||||
"owning destination advanced during ambiguous staging");
|
||||
take(locker->query("ROLLBACK", {}, 5s), "release owning stage reply");
|
||||
for (unsigned attempt = 0; attempt < 200 && !destination_host->status().migration_frozen;
|
||||
++attempt) {
|
||||
destination_host->step();
|
||||
std::this_thread::sleep_for(25ms);
|
||||
}
|
||||
require(destination_host->status().migration_frozen && !destination_host->status().ready &&
|
||||
!destination_host->advance_unattended(64),
|
||||
"owning destination activated its ghost");
|
||||
destination_host->drain();
|
||||
destination_host = make_destination_host();
|
||||
require(!destination_host->step().ready && destination_host->status().migration_frozen &&
|
||||
destination_host->state()->snapshot() == destination_bytes &&
|
||||
!destination_host->authority().has_value(),
|
||||
"replacement destination lost staging or adopted the uncommitted ghost");
|
||||
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease,
|
||||
destination_host_freeze.trace.request.request, 5s),
|
||||
"source-authorized abort releases destination host");
|
||||
for (unsigned attempt = 0; attempt < 200 && !destination_host->step().ready; ++attempt)
|
||||
std::this_thread::sleep_for(25ms);
|
||||
require(destination_host->status().ready && !destination_host->status().migration_frozen &&
|
||||
destination_host->state()->snapshot() == destination_bytes,
|
||||
"destination failed to resume its unchanged original state after source abort");
|
||||
destination_host->drain();
|
||||
destination_lease = take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s),
|
||||
"return staged destination fixture authority");
|
||||
f.head = take(
|
||||
materialize_population_profile(*f.db, *f.objects, f.lease, f.head, f.id(), f.build, 5s),
|
||||
"resume frozen population as raw owner");
|
||||
|
||||
@@ -213,6 +213,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
" -v migration_sha_45=$(s 0045_owned_population_intervals.sql)"
|
||||
" -v migration_sha_46=$(s 0046_boundary_accounting_upgrade.sql)"
|
||||
" -v migration_sha_47=$(s 0047_cellular_migration_freeze.sql)"
|
||||
" -v migration_sha_48=$(s 0048_cellular_migration_staging.sql)"
|
||||
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
|
||||
" > /dev/null 2>&1") != 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
|
||||
@@ -230,7 +231,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
"tools/migrate-postgres.sh\" > /dev/null") == 0);
|
||||
CHECK(history(*owner_db) ==
|
||||
"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,"
|
||||
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47");
|
||||
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48");
|
||||
CHECK(scalar(*owner_db, "SELECT (c.condeferrable AND c.condeferred AND "
|
||||
"c.confrelid = 'evolution.region_commits'::regclass)::text "
|
||||
"FROM pg_catalog.pg_constraint c WHERE c.conrelid = "
|
||||
|
||||
@@ -129,7 +129,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
|
||||
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
|
||||
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
|
||||
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
|
||||
v2_sha="${v2_sha%% *}"
|
||||
@@ -150,7 +150,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 47:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 48:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
|
||||
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
|
||||
@@ -204,7 +204,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
|
||||
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
|
||||
phase permission-audit
|
||||
@@ -257,9 +257,9 @@ expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control
|
||||
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
|
||||
phase checksum-drift
|
||||
# Unknown future versions and checksum drift must not be repaired silently.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=48 WHERE version=47'
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=49 WHERE version=48'
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=47 WHERE version=48"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=48 WHERE version=49"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13"
|
||||
|
||||
@@ -106,6 +106,8 @@ digest_46="$(sha256sum "$root/migrations/0046_boundary_accounting_upgrade.sql")"
|
||||
digest_46="${digest_46%% *}"
|
||||
digest_47="$(sha256sum "$root/migrations/0047_cellular_migration_freeze.sql")"
|
||||
digest_47="${digest_47%% *}"
|
||||
digest_48="$(sha256sum "$root/migrations/0048_cellular_migration_staging.sql")"
|
||||
digest_48="${digest_48%% *}"
|
||||
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
|
||||
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
|
||||
@@ -146,5 +148,6 @@ psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_45=$digest_45" \
|
||||
--set="migration_sha_46=$digest_46" \
|
||||
--set="migration_sha_47=$digest_47" \
|
||||
--set="migration_sha_48=$digest_48" \
|
||||
--set="runtime_role=$runtime_role" \
|
||||
--file="$root/migrations/apply.sql"
|
||||
|
||||
Reference in New Issue
Block a user