M12.4: retain migration source freezes and explicit aborts
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m24s
Foundation / checks (network) (push) Successful in 20m44s
Foundation / checks (client) (push) Successful in 22m3s
Foundation / checks (tsan) (push) Successful in 23m1s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (checks) (push) Canceled after 24m7s
Foundation / checks (postgres) (push) Canceled after 24m3s

This commit is contained in:
dingo
2026-10-18 09:08:50 -04:00
parent 1b5c0cdf7c
commit c625a4cb68
25 changed files with 1342 additions and 18 deletions
+1 -1
View File
@@ -255,7 +255,7 @@ if(EVO_BUILD_POSTGRES)
add_library(evo_region_store STATIC src/persistence/sha256.cpp src/persistence/object_store.cpp
src/persistence/checkpoint_store.cpp src/persistence/primitive_checkpoint_store.cpp
src/persistence/cellular_checkpoint_store.cpp src/persistence/population_checkpoint_store.cpp
src/persistence/region_clone.cpp)
src/persistence/region_clone.cpp src/persistence/cellular_migration_store.cpp)
add_library(evo::region_store ALIAS evo_region_store)
target_link_libraries(evo_region_store PUBLIC evo::postgres evo::region_checkpoint
PRIVATE OpenSSL::Crypto)
+17
View File
@@ -1,5 +1,22 @@
# Project status and next steps
## M12.4h durable source freeze and explicit abort - 2026-10-18
ADR-0147/schema 47 retains exact biological migration preparation, replay capsules
and a durable source freeze across worker replacement. Source publication and
new interval/player reservations are refused while pending; source stock and both
heads remain unchanged. Explicit current-source abort appends its outcome and
clears the pause, with exact retries. The host pauses readiness/advancement,
retains ambiguous requests and verifies complete biology before resuming.
Native Release passes 78/78; focused private PostgreSQL, ASan/UBSan, Clang
analysis, format and shell syntax checks pass. The complete private persistence,
process-fault, backup/DR and migration/permission suite passes in 502 s.
[Evidence and limits](engineering/VALIDATION.md)
retain destination staging, atomic paired commit, commit/abort races, activation
and linked-boundary contracts as subsequent M12.4 work. All of M12 remains in
progress; no deployment or live region changed.
## M12.4g durable accounting upgrade - 2026-10-18
ADR-0146/schema 46 extends the immutable version-upgrade journal to same-profile
@@ -0,0 +1,62 @@
# ADR-0147: Durable biological migration preparation and source freeze
## Status
Accepted and qualified for scoped M12.4h implementation, 2026-10-18. Executed
evidence and limits are in [VALIDATION](../engineering/VALIDATION.md).
Destination staging, atomic paired publication and activation
remain subsequent segments. No deployment or automatic migration is selected.
## Decision
Prepare ADR-0144's exact raw/population pair against two digest-verified bundle-10
committed cuts, with the same representation, world/frame/tick and compatible
biology/geometry. Retain canonical migration trace 1 under its stable request
UUID. The source's current lease and exact head fence preparation. The destination
head is checked optimistically during preparation; it is neither locked nor
activated. This does not establish a simultaneous global cut at freeze commit.
Before preparation, copy and verify the destination predecessor payload into the
source's immutable object namespace. This is a read-only replay capsule, not an
owned population or additional spendable stock. Both manifests stay retained in
the shared commit journal. Recovery can verify the entire recorded pair from the
source namespace even if the destination later advances or is unavailable.
Schema 47 records immutable preparations and explicit source-authorized aborts,
plus one bounded pending pointer per source region. A prepared source retains all
its cells, inventory and history at its existing head. Coarsely freeze the whole
region: refuse new checkpoints, profile/version changes, interval admissions and
player mutation reservations until settlement. Preparation refuses pending
intervals and outstanding player choices; C++ requires fully suspended controlled
cells and an eligible complete wildlife group. No biological tick or transfer
occurs. Destination ghosts own nothing; this segment creates none.
The owning host pauses readiness and advancement during preparation uncertainty
or a recovered freeze. It retries the exact retained request, renews authority,
and restores/validates the capsule before resuming any work. Replacing a worker
or expiring a lease never releases the freeze. Explicit abort under current source
authority appends its immutable outcome and removes the pending pointer in one
transaction, leaving both biological heads unchanged. Exact preparation/abort
retries retain their outcome; retrying an aborted preparation never reopens it.
Timeout is not an abort. A replacement owner can explicitly abort without needing
destination authority, since no ownership has transferred.
Subsequent destination staging must recheck its exact captured head. A moved
destination requires an explicit source abort and a fresh workflow; do not rebase
the old trace silently. Future two-head commit must settle the source pointer
atomically with both exact results and immutable shared evidence. This segment
admits no migration result manifest and no destination activation.
## Alternatives, costs and qualification
An in-memory pause disappears after process death. Lease expiry is not an abort
decision. Copying cells into an active destination before paired commit duplicates
ownership. Keep the source authoritative and durably frozen instead. A whole
region pause is deliberately coarse; it is not a final ecological movement or
latency policy. Source-side capsule retention adds bounded storage and replay work.
Require real player-origin wildlife, both representations, complete replay and
stock equality, pending-work refusal, publication/admission bypass rejection,
duplicate/conflicting preparation, explicit abort, restart/new-epoch recovery,
ambiguous replies, process loss, missing/corrupt capsules and host readiness.
No elapsed-day, atlas, environmental coupling or multigeneration claim follows.
+2
View File
@@ -1,5 +1,7 @@
# Architecture decision records
- [ADR-0147: Durable biological preparation and source freeze](0147-durable-cellular-migration-freeze.md) - schema 47 retains exact preparation/capsules and explicit abort; the source stays authoritative and paused across worker replacement. Destination staging and paired activation remain open.
- [ADR-0146: Durable boundary-accounting upgrade admission](0146-durable-boundary-accounting-upgrade.md) - schema 46 extends immutable version-upgrade evidence to same-profile bundle 9->10, full upgrade replay, population batch 2 and preserved representation/interval guards; paired migration escrow remains open.
- [ADR-0145: Explicit boundary-accounting checkpoint upgrade](0145-boundary-accounting-checkpoint-upgrade.md) - portable bundle 10, exact no-tick retained-source upgrade and representation-preserving continuation; SQL admission and durable upgrade evidence remain open.
@@ -10,6 +10,15 @@ upgrade automatically. Atomic paired transfer publication remains required.
See [ADR-0145](../adr/0145-boundary-accounting-checkpoint-upgrade.md) and
[ADR-0146](../adr/0146-durable-boundary-accounting-upgrade.md).
M12.4h/schema 47 retains source migration preparation, immutable explicit abort
and a bounded pending freeze across worker replacement. Complete recovery replay
uses both retained manifests and a verified destination capsule in the source's
object namespace. Both biological heads stay unchanged; destination inspection
is optimistic and grants no destination authority. Source publication and new
interval/player reservations are blocked until explicit settlement. See
[ADR-0147](../adr/0147-durable-cellular-migration-freeze.md). Destination staging,
atomic paired result publication and activation remain required.
M7.1 now scopes the first implementation in [ADR-0058](../adr/0058-region-durability-contract.md)
and [REGION_DURABILITY.md](REGION_DURABILITY.md): full checkpoint batches, not
recovery from semantic events or an unimplemented input journal. Candidate uploads
+17 -5
View File
@@ -49,17 +49,29 @@ readiness. New work requires fully suspended controlled cells, leaving vulnerabl
linger/death on profile 4. Earlier trace semantics remain readable. See the
[interval contract](../../schemas/ecology/population-interval-v1.md).
ADR-0144 adds a portable exact transfer pair with explicit regional boundary
flows/nonresident history (owner 7, population envelope 2). These versions have
no admitted durable manifest bundle yet. Existing profile-4/5 publishers retain
their tuples; population trace 1 refuses envelope 2. A future adapter must freeze
the source and atomically commit both fenced heads before destination activation.
ADR-0144 adds an exact transfer pair with explicit regional boundary flows and
nonresident history (owner 7, population envelope 2). ADR-0145/0146 and schema 46
admit bundle 10 through an explicit same-profile version upgrade. Population trace
1 retains bundle 9/envelope 1; trace 2 requires bundle 10/envelope 2. No worker
upgrades automatically. A paired publisher must commit both fenced heads before
destination activation.
See [scope and remaining proof](../engineering/M12_EXACT_BIOLOGICAL_TRANSFER.md).
Its bounded [paired migration trace](../../schemas/cellular-migration-trace-v1.md)
replays from both source cuts and an independently trusted request, including
complete cohort tags. Independent-process replay is qualified; no existing
durable publisher admits that trace or treats it as a transfer receipt.
ADR-0147/schema 47 retains that trace as source preparation evidence, without
admitting either migration result. The source stays authoritative at its exact
head; a durable pending pointer blocks its new publications/admissions and
survives lease expiry or replacement. Explicit fenced abort appends its outcome
and releases the pointer with both biological heads unchanged. A verified
destination predecessor capsule in the source namespace allows complete pair
replay without consulting destination live state. The owning host pauses
readiness, retains unknown requests and never treats timeout/drain as abort.
See [the freeze contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
Destination staging, paired commit/abort races and activation remain open.
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them.
@@ -120,3 +120,23 @@ This admits the accounting format and its ordinary continuation. Paired migratio
trace 1 still has no SQL publisher, source freeze, staged ghost or activation
rights. Durable two-head escrow, abort/partition/duplicate settlement, linked
geometry and ecology remain open. No deployment, live upgrade or golden changed.
## M12.4h durable source preparation and abort
ADR-0147/schema 47 retains canonical paired preparation under the source's current
lease and exact head. A verified destination predecessor capsule in the source
namespace supports complete recovery replay. The source keeps all stock at its
unchanged head; the destination is neither staged nor activated. New source
checkpoints/profile changes and interval/player reservations are refused while
pending. A replacement owner recovers the same freeze, without a timeout-based
release. Explicit current-source abort appends its immutable outcome and removes
the pending pointer with both biological heads unchanged. Retrying an aborted
preparation never reopens it.
The owning host withholds readiness and advancement while frozen or uncertain,
retains exact requests, renews authority and polls paused work once per second.
A fixed migration-freeze gauge identifies known pauses. Missing/corrupt capsules
and canonically encoded false member/material declarations fail complete replay.
See [the contract](../../schemas/ecology/cellular-migration-freeze-v1.md).
Destination staging, atomic paired head publication, commit/abort races and
activation remain the next segment; no route or automatic migration is enabled.
+6
View File
@@ -61,6 +61,12 @@ version-upgrade evidence and complete replayed recovery. Both profile upgrades,
population batch 2 and bundle-preserving profile changes are explicit; no worker
upgrades automatically. This format admission does not publish a migration pair
or close source-freeze/destination-activation and escrow qualification.
ADR-0147/schema 47 now adds durable source preparation/freeze, retained replay
capsules, explicit abort and host pause/recovery. Source stock remains owned at
its exact head through worker replacement; no timeout or drain releases it.
Destination staging, atomic paired commit, commit/abort races and activation
remain open. The destination head is only checked optimistically at preparation
and must be rechecked before any future stage/commit.
The transfer tests also expose the current rules-8 maturation gap: buds retain
their subadult realization while reproduction requires a fully realized parent.
Finite extra food alone does not close repeated wildlife generations. A versioned
+12
View File
@@ -20,6 +20,18 @@ Batch-1/2 cross-binding is refused in both directions. The process-loss and
host-recreation/materialization matrix also runs with batch 2. Full historical
migration/backup/DR/permission regression remains required for schema admission.
M12.4h's private freeze matrix uses funded player-derived wildlife and both
bundle-10 representations. It requires unchanged source/destination heads and
stock, exact request retries, conflicting/new-work refusal, pending-choice/
interval guards, new-epoch recovery, explicit abort without reopening, and real
SIGKILL before/after preparation. Unknown preparation/abort replies pause the
owning host through exact reconciliation; drain/recreation retains the freeze.
Missing and same-length corrupt capsules fail, and canonical false material
fails full biological replay. SQL checksum rejection is checked independently of
the C++ decoder. The metric has one fixed unlabelled series and remains bounded.
These are preparation/abort checks, not proof of destination activation or atomic
paired publication; full historical persistence regression is also required.
Tests defend contracts, not directory coverage. All checks remain active in
Release builds. Current proof has no actual network, biology or distributed
recovery, so those categories below are milestone gates rather than passing
+54
View File
@@ -1,5 +1,59 @@
# Validation report
## 2026-10-18 - M12.4h durable migration source freeze and explicit abort
ADR-0147/schema 47 retains exact source-authorized preparation, both predecessor
manifests and a verified destination payload capsule in the source namespace.
Source stock remains at its unchanged head. Immutable explicit abort removes the
pending pointer without transferring ownership or advancing either head. The
owning host pauses readiness/advancement across uncertainty and replacement,
renews authority and verifies the complete retained biological pair. Destination
staging and paired result publication are subsequent work.
Executed locally:
- Native MSVC Release **78/78 passed**, 14.66 s.
- Actual GCC 14 optional persistence/server targets rebuilt; final private
PostgreSQL population/freeze suite passed, 9 s total (6 s test phase).
- Linux Clang 19 ASan/UBSan final private population/freeze suite passed,
**30 s** total (28 s test phase), with no sanitizer reports.
- Explicit clang-tidy 19 on the new persistence adapter, owning host and
PostgreSQL population tests passed with no owned warnings. Repository
clang-format 19 and both changed shell scripts' syntax checks passed.
- Complete frozen-binary private PostgreSQL qualification passed with **exit 0**,
**502 s**: C++ integration/process/load/history (424 s), life reservations,
backup/restore, host-loss DR, upgrade, migration history/rollback, runtime
permission audit and checksum/future-version rejection. The detached runner
recorded its completion code; no interrupted run is counted as a pass.
Both raw and population representations exercise actual player-origin wildlife,
complete capsule replay, unchanged heads/material, duplicate/conflicting
requests, pending-work refusal and publication/admission bypass guards. Replacement
epochs retain freezes; stale aborts fail, explicit abort is idempotent and aborted
preparation retries never reopen a pointer. Real lock-timeout ambiguity retains
exact requests. Process kills before preparation and after commit distinguish
absence from a retained freeze. Missing and same-length corrupt capsules fail
closed even while the original destination object exists; canonical false
inventory declarations fail full biological replay. The owning-host matrix
covers restart/drain, ambiguous prepare/abort and resumption of identical biology.
The fixed unlabelled freeze metric is included in existing cardinality bounds.
Initial compile/analysis issues (qualified lease-clock type, optional handling)
were corrected. The first new reservation fixture used an obsolete account
control generation and reached the existing epoch fence first. The successful
fixture claims current control inside a rolled-back transaction and reaches the
new freeze guard without weakening either fence.
Limits: source preparation checks the destination head optimistically; it is not
a simultaneous world cut. There is no destination ghost, paired commit,
activation, linked geometry, elapsed-day or multigeneration qualification. Backup
inspection confirms all immutable namespace objects, including capsules, are
copied; existing backup/DR fixtures do not qualify cellular history cloning or a
new cellular restore procedure. No live database, World0, deployment, dependency,
biological rule or golden changed. The supplied `Build_Errors.txt` still contains
the earlier captured CI failures; remote confirmation of their committed runner
repair remains separate. That file and unrelated local work remain untracked.
## 2026-10-18 - M12.4g durable boundary-accounting upgrade
ADR-0146/schema 46 admits the explicit same-profile bundle-9 to 10 upgrade through
@@ -0,0 +1,45 @@
#pragma once
#include "evo/persistence/checkpoint_store.hpp"
#include "evo/server/lifecycle/cellular_migration_trace.hpp"
namespace evo::persistence {
struct CellularMigrationPrepared {
RegionKey key{}; // Source region.
std::vector<std::uint8_t> trace;
};
struct CellularMigrationFreeze {
server::CellularMigrationTrace trace;
bool aborted{};
};
struct CellularMigrationFailure {
PublicationError kind{};
LeaseStoreFailure store{};
ObjectError object{};
CellularMigrationPrepared prepared; // Exact request after unknown/rejection.
};
// Verify both complete cuts and the paired biology, retain a verified destination
// replay capsule in the source namespace, then freeze the exact source head.
// Neither candidate is published or activated by this operation.
[[nodiscard]] std::expected<CellularMigrationFreeze, CellularMigrationFailure>
prepare_cellular_migration_freeze(Postgres&, const FilesystemObjectStore& source_objects,
const FilesystemObjectStore& destination_objects,
const LeaseGrant& source_lease, const CommittedCheckpoint& source,
const CommittedCheckpoint& destination,
const server::CellularMigrationRequest&,
std::chrono::milliseconds timeout,
const PublishHooks& hooks = {});
[[nodiscard]] std::expected<CellularMigrationFreeze, CellularMigrationFailure>
retry_cellular_migration_freeze(Postgres&, const LeaseGrant&, const CellularMigrationPrepared&,
std::chrono::milliseconds timeout);
// Current source authority only. No timeout/lease-expiry auto-abort. The result
// repeats a retained abort and cannot reopen an old preparation.
[[nodiscard]] std::expected<CellularMigrationFreeze, LeaseStoreFailure>
abort_cellular_migration_freeze(Postgres&, RegionKey, const LeaseGrant&, const Uuid& request,
std::chrono::milliseconds timeout);
// Fenced lookup AND complete capsule replay. An empty result means no freeze;
// missing/corrupt evidence is an error, never implicit permission to advance.
[[nodiscard]] std::expected<std::optional<CellularMigrationFreeze>, RecoveryFailure>
load_cellular_migration_freeze(Postgres&, const FilesystemObjectStore&, RegionKey,
const LeaseGrant&, std::chrono::milliseconds timeout);
} // namespace evo::persistence
@@ -1,6 +1,7 @@
#pragma once
#include "evo/network/session_protocol.hpp"
#include "evo/persistence/cellular_migration_store.hpp"
#include "evo/persistence/checkpoint_store.hpp"
#include "evo/persistence/life_reservation_store.hpp"
#include "evo/persistence/population_checkpoint_store.hpp"
@@ -91,6 +92,14 @@ class CellularRegionHost {
// Acquires/renews authority, recovers fully, or resolves the exact pending commit.
const HostStatus& step();
// Explicit source-only preparation/abort. Preparation pauses the whole region;
// restart and lease expiry never imply abort or destination activation.
[[nodiscard]] std::expected<void, CellularPublishFailure>
prepare_migration_freeze(const persistence::CommittedCheckpoint& destination,
const persistence::FilesystemObjectStore& destination_objects,
const CellularMigrationRequest&);
[[nodiscard]] std::expected<void, CellularPublishFailure>
abort_migration_freeze(const persistence::Uuid& request);
// Explicit no-tick fenced transition. Uncertain publication pauses the host.
[[nodiscard]] std::expected<void, CellularPublishFailure> set_population_mode(bool enabled);
[[nodiscard]] bool population_mode() const noexcept;
@@ -154,6 +163,7 @@ class CellularRegionHost {
void refresh();
bool recover_state(const persistence::LeaseGrant&);
void reconcile_interval();
void reconcile_migration_freeze();
void finish_interval();
[[nodiscard]] std::expected<persistence::PublishedCellularBatch, CellularPublishFailure>
publish_population(const CellularBatch&);
@@ -215,6 +225,10 @@ class CellularRegionHost {
std::expected<persistence::CommittedCheckpoint, persistence::PopulationPublicationFailure>,
OwnedState candidate);
OwnedState state_;
bool migration_reload_{}, migration_frozen_{};
persistence::LeaseClock::time_point migration_poll_at_{};
std::optional<persistence::CellularMigrationPrepared> migration_request_;
std::optional<persistence::Uuid> migration_abort_;
bool interval_reload_{};
std::optional<persistence::PopulationInterval> admitted_interval_;
// An uncertain admission must retry its exact source/target, never select again.
+1
View File
@@ -38,6 +38,7 @@ struct HostStatus {
std::uint64_t commits{}, store_failures{}, unknown_outcomes{}, fences{};
// Local lease safety: time until this host must stop originating (0 = none).
std::uint64_t lease_remaining_ms{};
bool migration_frozen{}; // Known durable source freeze; expiry is not an abort.
};
// Process-level counters and bounds the exposition adds to the host status.
@@ -0,0 +1,207 @@
-- ADR-0147: retain exact biological preparation; source owns all stock until paired commit.
CREATE TABLE evolution.cellular_migration_preparations (
request_id uuid PRIMARY KEY,
world_id evolution.u64 NOT NULL,
source_region evolution.u64 NOT NULL,
destination_region evolution.u64 NOT NULL CHECK (destination_region <> source_region),
source_commit uuid NOT NULL REFERENCES evolution.region_commits(commit_id),
destination_commit uuid NOT NULL REFERENCES evolution.region_commits(commit_id),
trace bytea NOT NULL CHECK (octet_length(trace) BETWEEN 272 AND 2312),
prepared_holder uuid NOT NULL,
prepared_epoch evolution.u64 NOT NULL CHECK (prepared_epoch > 0),
FOREIGN KEY(world_id,source_region) REFERENCES evolution.region_leases,
FOREIGN KEY(world_id,destination_region) REFERENCES evolution.region_leases
);
CREATE TABLE evolution.cellular_migration_aborts (
request_id uuid PRIMARY KEY REFERENCES evolution.cellular_migration_preparations,
holder uuid NOT NULL, epoch evolution.u64 NOT NULL CHECK (epoch > 0)
);
CREATE TABLE evolution.cellular_migration_pending (
world_id evolution.u64 NOT NULL, region_id evolution.u64 NOT NULL,
request_id uuid UNIQUE NOT NULL REFERENCES evolution.cellular_migration_preparations,
PRIMARY KEY(world_id,region_id)
);
REVOKE ALL ON evolution.cellular_migration_preparations,evolution.cellular_migration_aborts,
evolution.cellular_migration_pending FROM PUBLIC;
CREATE TRIGGER cellular_migration_preparations_immutable BEFORE UPDATE OR DELETE ON evolution.cellular_migration_preparations
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER cellular_migration_preparations_no_truncate BEFORE TRUNCATE ON evolution.cellular_migration_preparations
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER cellular_migration_aborts_immutable BEFORE UPDATE OR DELETE ON evolution.cellular_migration_aborts
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER cellular_migration_aborts_no_truncate BEFORE TRUNCATE ON evolution.cellular_migration_aborts
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE FUNCTION evolution.check_cellular_migration_trace(t bytea) RETURNS void
LANGUAGE plpgsql IMMUTABLE SET search_path = pg_catalog, pg_temp AS $$
DECLARE n integer; members numeric; previous numeric := 0; life numeric;
hash numeric := 14695981039346656037; ids bytea[];
BEGIN
IF t IS NULL OR octet_length(t) < 272 OR octet_length(t) > 2312 THEN
RAISE EXCEPTION 'invalid migration trace length' USING ERRCODE = '22023';
END IF;
n := octet_length(t); members := evolution.manifest_uint(t,248,4);
ids := ARRAY[substring(t FROM 73 FOR 16),substring(t FROM 121 FOR 16),
substring(t FROM 145 FOR 16),substring(t FROM 193 FOR 16)];
IF substring(t FROM 1 FOR 8) <> '\x4145564d49473031'::bytea OR evolution.manifest_uint(t,8,4) <> 1
OR evolution.manifest_uint(t,12,4) NOT IN (1,2) OR evolution.manifest_uint(t,16,8) = 0
OR evolution.manifest_uint(t,24,8) = 0 OR evolution.manifest_uint(t,32,8) = 0
OR evolution.manifest_uint(t,40,8) = 0 OR substring(t FROM 49 FOR 16) = decode(repeat('00',16),'hex')
OR evolution.manifest_uint(t,64,8) NOT BETWEEN 1 AND 2147483647
OR evolution.manifest_uint(t,136,8) NOT BETWEEN 1 AND 2147483647
OR evolution.manifest_uint(t,64,8) = evolution.manifest_uint(t,136,8)
OR substring(t FROM 89 FOR 32) = decode(repeat('00',32),'hex')
OR substring(t FROM 161 FOR 32) = decode(repeat('00',32),'hex')
OR substring(t FROM 209 FOR 40) = decode(repeat('00',40),'hex')
OR members NOT BETWEEN 1 AND 256 OR n <> 264 + 8 * members
OR evolution.manifest_uint(t,252,4) <> 0
OR (SELECT count(DISTINCT x) FROM unnest(ids) x) <> 4
OR decode(repeat('00',16),'hex') = ANY(ids) THEN
RAISE EXCEPTION 'invalid migration trace shape' USING ERRCODE = '22023';
END IF;
FOR i IN 0..members::integer-1 LOOP
life := evolution.manifest_uint(t,256+8*i,8);
IF life <= previous OR life < 9223372036854775808
OR trunc((life-9223372036854775808)/4294967296) = 0 OR mod(life,4294967296) = 0 THEN
RAISE EXCEPTION 'invalid migration member' USING ERRCODE = '22023';
END IF;
previous := life;
END LOOP;
-- Exact bounded FNV-1a64, corruption detection only; manifests/payloads use SHA-256.
FOR i IN 0..n-9 LOOP
hash := mod((hash-mod(hash,256)+(mod(hash,256)::integer # get_byte(t,i)))*1099511628211,
18446744073709551616);
END LOOP;
IF hash <> evolution.manifest_uint(t,n-8,8) THEN
RAISE EXCEPTION 'migration trace checksum differs' USING ERRCODE = '22023';
END IF;
END $$;
REVOKE ALL ON FUNCTION evolution.check_cellular_migration_trace(bytea) FROM PUBLIC;
CREATE FUNCTION evolution.cellular_migration_record(w evolution.u64,r evolution.u64,id uuid)
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean)
LANGUAGE sql STABLE SET search_path = pg_catalog, pg_temp AS $$
SELECT p.trace,s.manifest,d.manifest,EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=p.request_id)
FROM evolution.cellular_migration_preparations p
JOIN evolution.region_commits s ON s.commit_id=p.source_commit
JOIN evolution.region_commits d ON d.commit_id=p.destination_commit
WHERE p.world_id=w AND p.source_region=r AND p.request_id=id
$$;
REVOKE ALL ON FUNCTION evolution.cellular_migration_record(evolution.u64,evolution.u64,uuid) FROM PUBLIC;
CREATE FUNCTION evolution.prepare_cellular_migration_freeze(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,t bytea)
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE id uuid; src uuid; dst uuid; destination evolution.u64; h evolution.region_heads%ROWTYPE;
sm bytea; dm bytea; s evolution.checkpoint_manifest; d evolution.checkpoint_manifest;
old evolution.cellular_migration_preparations%ROWTYPE;
bundle10 constant bytea := '\x080000000700000002000000010000000100000004000000030000000200000001000000';
BEGIN
PERFORM evolution.check_cellular_migration_trace(t);
IF evolution.manifest_uint(t,16,8) <> w OR evolution.manifest_uint(t,64,8) <> r THEN
RAISE EXCEPTION 'foreign migration source' USING ERRCODE = '22023';
END IF;
id := encode(substring(t FROM 49 FOR 16),'hex')::uuid;
src := encode(substring(t FROM 73 FOR 16),'hex')::uuid;
dst := encode(substring(t FROM 145 FOR 16),'hex')::uuid;
destination := evolution.manifest_uint(t,136,8);
PERFORM evolution.check_region_lease(w,r,holder,epoch);
SELECT * INTO old FROM evolution.cellular_migration_preparations p WHERE p.request_id=id;
IF FOUND THEN
IF old.world_id<>w OR old.source_region<>r OR old.trace<>t THEN
RAISE EXCEPTION 'migration request reused' USING ERRCODE = 'EVL07';
END IF;
RETURN QUERY SELECT * FROM evolution.cellular_migration_record(w,r,id);
RETURN;
END IF;
SELECT * INTO h FROM evolution.region_heads x WHERE x.world_id=w AND x.region_id=r FOR UPDATE;
IF NOT FOUND OR h.commit_id<>src OR h.digest<>substring(t FROM 89 FOR 32)
OR h.end_tick<>evolution.manifest_uint(t,32,8) THEN
RAISE EXCEPTION 'migration source moved' USING ERRCODE = 'EVL03';
END IF;
SELECT c.manifest INTO sm FROM evolution.region_commits c WHERE c.commit_id=src;
-- Destination remains unowned by this workflow. A later stage must recheck its head.
SELECT c.manifest INTO dm FROM evolution.region_heads x JOIN evolution.region_commits c ON c.commit_id=x.commit_id
WHERE x.world_id=w AND x.region_id=destination AND x.commit_id=dst AND x.digest=substring(t FROM 161 FOR 32);
IF dm IS NULL THEN RAISE EXCEPTION 'migration destination moved' USING ERRCODE = 'EVL03'; END IF;
s := evolution.parse_checkpoint_manifest(sm); d := evolution.parse_checkpoint_manifest(dm);
IF s.profile NOT IN (4,5) OR d.profile<>s.profile OR s.versions<>bundle10 OR d.versions<>bundle10
OR s.commit_sequence=0 OR d.commit_sequence=0 OR s.frame_id<>d.frame_id
OR s.frame_id<>evolution.manifest_uint(t,24,8) OR s.end_tick<>d.end_tick
OR s.tick_microseconds<>d.tick_microseconds
OR s.profile<>(CASE evolution.manifest_uint(t,12,4) WHEN 1 THEN 4 ELSE 5 END) THEN
RAISE EXCEPTION 'incompatible migration cuts' USING ERRCODE = 'EVL06';
END IF;
IF EXISTS(SELECT 1 FROM evolution.cellular_migration_pending p WHERE p.world_id=w AND p.region_id=r)
OR EXISTS(SELECT 1 FROM evolution.population_interval_pending p WHERE p.world_id=w AND p.region_id=r)
OR EXISTS(SELECT 1 FROM evolution.cellular_mutation_reservations x WHERE x.world_id=w AND x.region_id=r
AND NOT EXISTS(SELECT 1 FROM evolution.cellular_mutation_action_publications a
WHERE a.world_id=x.world_id AND a.account_id=x.account_id AND a.action_id=x.action_id))
OR EXISTS(SELECT 1 FROM evolution.life_reservations x WHERE x.world_id=w AND x.region_id=r AND x.rejection=0 AND x.ended_commit IS NULL
AND NOT EXISTS(SELECT 1 FROM evolution.primitive_life_bindings b
WHERE b.world_id=x.world_id AND b.account_id=x.account_id AND b.request_id=x.request_id)) THEN
RAISE EXCEPTION 'migration source has pending work' USING ERRCODE = 'EVL18';
END IF;
INSERT INTO evolution.cellular_migration_preparations VALUES(id,w,r,destination,src,dst,t,holder,epoch);
INSERT INTO evolution.cellular_migration_pending VALUES(w,r,id);
RETURN QUERY SELECT * FROM evolution.cellular_migration_record(w,r,id);
END $$;
REVOKE ALL ON FUNCTION evolution.prepare_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,bytea) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.prepare_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,bytea) TO :"runtime_role";
CREATE FUNCTION evolution.abort_cellular_migration_freeze(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,id uuid)
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE p evolution.cellular_migration_preparations%ROWTYPE;
BEGIN
PERFORM evolution.check_region_lease(w,r,holder,epoch);
SELECT * INTO p FROM evolution.cellular_migration_preparations x WHERE x.request_id=id AND x.world_id=w AND x.source_region=r;
IF NOT FOUND THEN RAISE EXCEPTION 'unknown source migration' USING ERRCODE = 'EVL04'; END IF;
IF NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=id) THEN
IF NOT EXISTS(SELECT 1 FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=r AND x.request_id=id)
OR NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=r AND h.commit_id=p.source_commit
AND h.digest=substring(p.trace FROM 89 FOR 32) AND h.end_tick=evolution.manifest_uint(p.trace,32,8)) THEN
RAISE EXCEPTION 'migration freeze inconsistent' USING ERRCODE = 'EVL03';
END IF;
INSERT INTO evolution.cellular_migration_aborts VALUES(id,holder,epoch);
DELETE FROM evolution.cellular_migration_pending x WHERE x.world_id=w AND x.region_id=r AND x.request_id=id;
END IF;
RETURN QUERY SELECT * FROM evolution.cellular_migration_record(w,r,id);
END $$;
REVOKE ALL ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid) TO :"runtime_role";
CREATE FUNCTION evolution.load_cellular_migration_freeze(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64)
RETURNS TABLE(trace bytea,source_manifest bytea,destination_manifest bytea,aborted boolean)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE p evolution.cellular_migration_preparations%ROWTYPE;
BEGIN
PERFORM evolution.check_region_lease(w,r,holder,epoch);
SELECT a.* INTO p FROM evolution.cellular_migration_pending x JOIN evolution.cellular_migration_preparations a ON a.request_id=x.request_id
WHERE x.world_id=w AND x.region_id=r;
IF NOT FOUND THEN RETURN; END IF;
IF p.world_id<>w OR p.source_region<>r
OR EXISTS(SELECT 1 FROM evolution.cellular_migration_aborts a WHERE a.request_id=p.request_id)
OR NOT EXISTS(SELECT 1 FROM evolution.region_heads h WHERE h.world_id=w AND h.region_id=r AND h.commit_id=p.source_commit
AND h.digest=substring(p.trace FROM 89 FOR 32) AND h.end_tick=evolution.manifest_uint(p.trace,32,8)) THEN
RAISE EXCEPTION 'migration freeze inconsistent' USING ERRCODE = 'XX000';
END IF;
RETURN QUERY SELECT * FROM evolution.cellular_migration_record(w,r,p.request_id);
END $$;
REVOKE ALL ON FUNCTION evolution.load_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.load_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64) TO :"runtime_role";
CREATE FUNCTION evolution.refuse_frozen_cellular_work() RETURNS trigger
LANGUAGE plpgsql SET search_path = pg_catalog, pg_temp AS $$
BEGIN
IF EXISTS(SELECT 1 FROM evolution.cellular_migration_pending p WHERE p.world_id=NEW.world_id AND p.region_id=NEW.region_id) THEN
RAISE EXCEPTION 'cellular source frozen for migration' USING ERRCODE = 'EVL18';
END IF;
RETURN NEW;
END $$;
REVOKE ALL ON FUNCTION evolution.refuse_frozen_cellular_work() FROM PUBLIC;
CREATE TRIGGER region_cellular_migration_freeze BEFORE INSERT ON evolution.region_commits FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work();
CREATE TRIGGER interval_cellular_migration_freeze BEFORE INSERT ON evolution.population_interval_admissions FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work();
CREATE TRIGGER mutation_cellular_migration_freeze BEFORE INSERT ON evolution.cellular_mutation_reservations FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work();
CREATE TRIGGER birth_cellular_migration_freeze BEFORE INSERT ON evolution.life_reservations FOR EACH ROW EXECUTE FUNCTION evolution.refuse_frozen_cellular_work();
INSERT INTO evolution.schema_migrations(version,sha256) VALUES(47,:'migration_sha_47');
+15 -3
View File
@@ -122,6 +122,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
SELECT :'history_v43' || ',44:' || :'migration_sha_44' AS history_v44 \gset
SELECT :'history_v44' || ',45:' || :'migration_sha_45' AS history_v45 \gset
SELECT :'history_v45' || ',46:' || :'migration_sha_46' AS history_v46 \gset
SELECT :'history_v46' || ',47:' || :'migration_sha_47' AS history_v47 \gset
SELECT :'history' = :'history_v22' AS at_v22,
:'history' = :'history_v23' AS at_v23,
:'history' = :'history_v24' AS at_v24,
@@ -146,8 +147,10 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
:'history' = :'history_v43' AS at_v43,
:'history' = :'history_v44' AS at_v44,
:'history' = :'history_v45' AS at_v45,
:'history' = :'history_v46' AS at_v46 \gset
\if :at_v46
:'history' = :'history_v46' AS at_v46,
:'history' = :'history_v47' AS at_v47 \gset
\if :at_v47
\elif :at_v46
\elif :at_v45
\elif :at_v44
\elif :at_v43
@@ -533,7 +536,8 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
coalesce(max(version), 0) = 43 AS at_v43,
coalesce(max(version), 0) = 44 AS at_v44,
coalesce(max(version), 0) = 45 AS at_v45,
coalesce(max(version), 0) = 46 AS at_v46
coalesce(max(version), 0) = 46 AS at_v46,
coalesce(max(version), 0) = 47 AS at_v47
FROM evolution.schema_migrations \gset
\if :at_v24
\ir 0025_cellular_v2_bundle.sql
@@ -748,6 +752,7 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
\elif :at_v44
\elif :at_v45
\elif :at_v46
\elif :at_v47
\else
\echo 'Unknown or changed migration history after v43'
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
@@ -764,6 +769,10 @@ SELECT coalesce(max(version), 0) = 45 AS needs_v46 FROM evolution.schema_migrati
\if :needs_v46
\ir 0046_boundary_accounting_upgrade.sql
\endif
SELECT coalesce(max(version), 0) = 46 AS needs_v47 FROM evolution.schema_migrations \gset
\if :needs_v47
\ir 0047_cellular_migration_freeze.sql
\endif
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
-- grants or SET-able memberships would bypass the fenced functions. Checks the
@@ -783,6 +792,9 @@ BEGIN
UNION
SELECT m.member FROM pg_catalog.pg_auth_members m JOIN closure c ON m.roleid = c.oid
), fenced(oid) AS (VALUES
('evolution.prepare_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,bytea)'::regprocedure::oid),
('evolution.abort_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64,uuid)'::regprocedure::oid),
('evolution.load_cellular_migration_freeze(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
('evolution.admit_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,bytea,evolution.u64)'::regprocedure::oid),
('evolution.load_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
('evolution.publish_population_checkpoint(bytea,bytea)'::regprocedure::oid),
+7
View File
@@ -57,3 +57,10 @@ terminates, restores in a fresh process, replays and compares all four complete
output snapshots. This qualifies portable continuation, not database atomicity,
source freeze, ghost staging, abort, duplicate-delivery settlement or crash-safe
activation. No schema 45 publisher admits this trace or the new owner tuple.
M12.4h/ADR-0147/schema 47 subsequently retains trace 1 as durable source-freeze
preparation evidence. Both predecessor cuts are SHA-256 bound and replayed from
the source's retained capsule; SQL also checks canonical shape and FNV. The
source still owns every cell and the destination owns no ghost. Explicit abort
releases the freeze without changing either head. This is not result-manifest
admission or a paired ownership receipt. See [the freeze contract](ecology/cellular-migration-freeze-v1.md).
@@ -0,0 +1,64 @@
# Biological migration preparation and source freeze
Authority: ADR-0147, SQL schema 47. This retains preparation and explicit abort,
not a migration result or destination activation receipt. Checkpoint profiles,
rules 8, owner 7 and population batch 2 retain their existing bytes and meanings.
One source-authorized request binds canonical
[migration trace 1](../cellular-migration-trace-v1.md), both retained predecessor
commits/manifests and their exact payloads. Both cuts use bundle 10, the same
representation, world/frame/completed tick and compatible geometry/biology.
The source lease/head is fenced; the destination head is checked optimistically
and remains free to advance. Later staging must check it again. Preparation
does not establish a simultaneous world cut or a physical route.
The adapter restores both SHA-256-verified payloads and derives the complete
biological pair before publication. It copies/verifies the destination cut into
the source's immutable object namespace as a replay capsule. This copy owns no
stock. The source keeps every organism, reserve, body, nursery/habitat compartment,
heat and historical fact at its unchanged committed head. The destination also
keeps its existing head. Proposed transferred material in the trace is metadata,
not an additional account or an applied boundary flow.
SQL retains:
- Immutable `cellular_migration_preparations`, keyed by globally unique request
UUID: source/destination region and predecessor commit references, canonical
trace (272..2,312 bytes), preparing holder and epoch.
- Immutable `cellular_migration_aborts`, keyed by request, with the deciding
current source holder/epoch.
- One operational `cellular_migration_pending` pointer per source region.
Preparation refuses an active source workflow, pending ecological interval,
unpublished player choice or outstanding successful life reservation. C++ also
requires fully suspended controlled cells and an eligible whole wildlife group.
SQL independently validates trace identity/length/member bounds, exact FNV-1a64,
both committed manifest bindings and the declared representation/tuple.
Checksum/UUID metadata alone does not qualify the biology: recovery restores both
cuts from the source namespace and rederives every recorded member/channel.
While pending, indexed trigger guards refuse new region commits, interval
admissions, mutation reservations and life reservations at that source. Historical
exact operation retries keep their old meanings. The whole region is paused;
there is no per-member advancement or passive healing. The host withholds
readiness during preparation uncertainty and a recovered freeze, renews authority,
and polls paused work at most once per second. Normal unpaused stepping adds no
per-tick migration query. `evo_region_migration_frozen` is a fixed, unlabelled
0/1 gauge for a known durable freeze; it does not grant authority.
Explicit source-authorized abort appends its outcome and removes the pending
pointer atomically, without changing either biological head. Exact preparation
and abort retries return the recorded outcome. An aborted preparation cannot
reopen its pointer. Process death, lease expiry, drain, missing evidence and
timeout never imply abort. Missing/corrupt capsules fail closed, without falling
back to the destination's live state. A new source holder may explicitly abort;
destination authority is unnecessary because no ownership has moved.
Keep both predecessor histories and source-side capsule objects, including after
abort. The current backup copies all immutable namespace objects and the shared
database snapshot; no GC or pruning is authorized. This does not qualify cellular
history cloning/restore beyond the existing supported fixtures.
Destination ghost staging, atomic two-head publication, commit/abort races and
activation remain subsequent work. This segment adds no automatic migration,
atlas, cross-edge movement, environmental coupling or census/extinction rule.
@@ -0,0 +1,255 @@
#include "evo/persistence/cellular_migration_store.hpp"
#include "evo/persistence/sha256.hpp"
#include <algorithm>
#include <charconv>
#include <string_view>
namespace evo::persistence {
namespace {
using std::chrono::milliseconds;
LeaseStoreFailure failure(const PgFailure& cause) {
const std::string_view state(cause.sqlstate.data(), 5);
auto kind = LeaseStoreError::store;
if (cause.kind == PgError::unknown)
kind = LeaseStoreError::unknown;
else if (state == "22023" || cause.kind == PgError::bounds || cause.kind == PgError::config)
kind = LeaseStoreError::invalid;
else if (state == "EVL02")
kind = LeaseStoreError::fenced;
else if (state == "EVL03")
kind = LeaseStoreError::head_mismatch;
else if (state == "EVL04")
kind = LeaseStoreError::not_provisioned;
else if (state == "EVL06")
kind = LeaseStoreError::unsupported;
else if (state == "EVL07" || state == "EVL18")
kind = LeaseStoreError::commit_conflict;
return {kind, cause};
}
std::string hex(std::span<const std::uint8_t> bytes) {
constexpr std::string_view digits = "0123456789abcdef";
std::string out;
for (const auto byte : bytes) {
out += digits[byte >> 4U];
out += digits[byte & 15U];
}
return out;
}
std::string uuid(const Uuid& id) {
const auto h = hex(id);
return h.substr(0, 8) + '-' + h.substr(8, 4) + '-' + h.substr(12, 4) + '-' + h.substr(16, 4) +
'-' + h.substr(20);
}
std::optional<std::vector<std::uint8_t>> unhex(std::string_view value, std::size_t cap) {
if (value.size() % 2 != 0 || value.size() / 2 > cap)
return std::nullopt;
std::vector<std::uint8_t> out;
out.reserve(value.size() / 2);
for (std::size_t at = 0; at < value.size(); at += 2) {
unsigned byte{};
const auto [end, error] =
std::from_chars(value.data() + at, value.data() + at + 2, byte, 16);
if (error != std::errc{} || end != value.data() + at + 2 || byte > 255)
return std::nullopt;
out.push_back(static_cast<std::uint8_t>(byte));
}
return out;
}
struct Record {
std::vector<std::uint8_t> trace, source, destination;
bool aborted{};
};
std::optional<Record> parse(const PgRows::value_type& row) {
if (row.size() != 4 || (row[3].value_or("") != "true" && row[3].value_or("") != "false"))
return std::nullopt;
auto trace = unhex(row[0].value_or(""), server::max_migration_trace_bytes);
auto source = unhex(row[1].value_or(""), checkpoint_manifest_bytes);
auto destination = unhex(row[2].value_or(""), checkpoint_manifest_bytes);
if (!trace || !source || !destination)
return std::nullopt;
return Record{std::move(*trace), std::move(*source), std::move(*destination),
row[3].value_or("") == "true"};
}
bool bound(const CommittedCheckpoint& c, const CheckpointManifest& m) {
const auto digest = sha256(c.manifest);
return digest && *digest == c.head.digest && m.commit == c.head.commit &&
m.frame == c.head.frame && m.sequence == c.head.sequence &&
m.end_tick == c.head.end_tick;
}
bool cuts(const CheckpointManifest& s, const CheckpointManifest& d,
const Sha256Digest& source_digest, const Sha256Digest& destination_digest,
const server::CellularMigrationTrace& t) {
const auto profile = t.representation == server::CellularMigrationRepresentation::individual
? CheckpointProfile::cellular_lives_v1
: CheckpointProfile::cellular_population_v1;
return supported(s) && supported(d) && encode_manifest(s) && encode_manifest(d) &&
s.cellular_versions == cellular_v10_versions &&
d.cellular_versions == cellular_v10_versions && s.profile == profile &&
d.profile == profile && s.sequence > 0 && d.sequence > 0 &&
s.key == RegionKey{t.world, t.request.source.region} &&
d.key == RegionKey{t.world, t.request.destination.region} && s.frame == t.frame &&
d.frame == t.frame && s.end_tick == t.tick && d.end_tick == t.tick &&
s.commit == t.request.source.predecessor &&
d.commit == t.request.destination.predecessor &&
source_digest == t.request.source.predecessor_digest &&
destination_digest == t.request.destination.predecessor_digest;
}
std::expected<server::CellularMigrationTrace, CheckpointError>
derive(const CheckpointManifest& s, const CheckpointManifest& d,
std::span<const std::uint8_t> source, std::span<const std::uint8_t> destination,
const server::CellularMigrationRequest& request) {
if (s.profile == CheckpointProfile::cellular_lives_v1) {
auto src = restore_cellular_payload(source, s),
dst = restore_cellular_payload(destination, d);
if (!src || !dst)
return std::unexpected(CheckpointError::payload);
auto pair = server::prepare_migration_trace(**src, **dst, request);
if (!pair)
return std::unexpected(CheckpointError::payload);
return std::move(pair->trace);
}
auto src = restore_population_payload(source, s),
dst = restore_population_payload(destination, d);
if (!src || !dst)
return std::unexpected(CheckpointError::payload);
auto pair = server::prepare_migration_trace(**src, **dst, request);
if (!pair)
return std::unexpected(CheckpointError::payload);
return std::move(pair->trace);
}
std::optional<CellularMigrationFreeze> checked_record(const Record& record, RegionKey key,
const LeaseGrant& lease) {
auto trace = server::decode_migration_trace(record.trace);
const auto source = decode_manifest(record.source),
destination = decode_manifest(record.destination);
const auto s = sha256(record.source), d = sha256(record.destination);
if (!trace || !source || !destination || !s || !d ||
key != RegionKey{trace->world, trace->request.source.region} ||
lease.frame != trace->frame || !cuts(*source, *destination, *s, *d, *trace))
return std::nullopt;
return CellularMigrationFreeze{std::move(*trace), record.aborted};
}
constexpr auto columns = "encode(trace,'hex'),encode(source_manifest,'hex'),"
"encode(destination_manifest,'hex'),aborted::text";
CellularMigrationFailure failed(PublicationError kind, LeaseStoreFailure store = {},
ObjectError object = {}, CellularMigrationPrepared prepared = {}) {
return {kind, store, object, std::move(prepared)};
}
} // namespace
std::expected<CellularMigrationFreeze, CellularMigrationFailure>
retry_cellular_migration_freeze(Postgres& db, const LeaseGrant& lease,
const CellularMigrationPrepared& prepared, milliseconds timeout) {
const auto trace = server::decode_migration_trace(prepared.trace);
if (!trace || prepared.key != RegionKey{trace->world, trace->request.source.region} ||
lease.frame != trace->frame)
return std::unexpected(failed(PublicationError::candidate));
const std::array<std::string, 5> args{std::to_string(prepared.key.world),
std::to_string(prepared.key.region), uuid(lease.holder),
std::to_string(lease.epoch), "\\x" + hex(prepared.trace)};
auto rows =
db.query(std::string("SELECT ") + columns +
" FROM evolution.prepare_cellular_migration_freeze("
"$1::evolution.u64,$2::evolution.u64,$3::uuid,$4::evolution.u64,$5::bytea)",
args, timeout);
if (!rows) {
const auto store = failure(rows.error());
return std::unexpected(failed(store.kind == LeaseStoreError::unknown
? PublicationError::unknown
: PublicationError::rejected,
store, {}, prepared));
}
const auto record = rows->size() == 1 ? parse(rows->front()) : std::nullopt;
const auto result = record ? checked_record(*record, prepared.key, lease) : std::nullopt;
if (!result || record->trace != prepared.trace)
return std::unexpected(failed(PublicationError::unknown, {}, {}, prepared));
return *result;
}
std::expected<CellularMigrationFreeze, CellularMigrationFailure>
prepare_cellular_migration_freeze(Postgres& db, const FilesystemObjectStore& source_objects,
const FilesystemObjectStore& destination_objects,
const LeaseGrant& lease, const CommittedCheckpoint& source,
const CommittedCheckpoint& destination,
const server::CellularMigrationRequest& request,
milliseconds timeout, const PublishHooks& hooks) {
const auto s = decode_manifest(source.manifest), d = decode_manifest(destination.manifest);
if (!s || !d || !bound(source, *s) || !bound(destination, *d))
return std::unexpected(failed(PublicationError::predecessor));
auto src = source_objects.get({s->payload_sha256, s->payload_bytes});
auto dst = destination_objects.get({d->payload_sha256, d->payload_bytes});
if (!src || !dst)
return std::unexpected(
failed(PublicationError::predecessor, {}, !src ? src.error() : dst.error()));
auto trace = derive(*s, *d, *src, *dst, request);
if (!trace || !cuts(*s, *d, source.head.digest, destination.head.digest, *trace) ||
lease.frame != s->frame)
return std::unexpected(failed(PublicationError::candidate));
auto encoded = server::encode_migration_trace(*trace);
if (!encoded)
return std::unexpected(failed(PublicationError::candidate));
auto capsule = source_objects.put(*dst);
if (!capsule)
return std::unexpected(failed(PublicationError::upload, {}, capsule.error()));
auto readback = source_objects.get(*capsule);
if (!readback || *readback != *dst ||
*capsule != VerifiedObject{d->payload_sha256, d->payload_bytes})
return std::unexpected(failed(PublicationError::verify, {},
readback ? ObjectError::corrupt : readback.error()));
if (hooks.before_transaction)
hooks.before_transaction();
return retry_cellular_migration_freeze(db, lease, {s->key, std::move(*encoded)}, timeout);
}
std::expected<CellularMigrationFreeze, LeaseStoreFailure>
abort_cellular_migration_freeze(Postgres& db, RegionKey key, const LeaseGrant& lease,
const Uuid& request, milliseconds timeout) {
const std::array<std::string, 5> args{std::to_string(key.world), std::to_string(key.region),
uuid(lease.holder), std::to_string(lease.epoch),
uuid(request)};
auto rows =
db.query(std::string("SELECT ") + columns +
" FROM evolution.abort_cellular_migration_freeze("
"$1::evolution.u64,$2::evolution.u64,$3::uuid,$4::evolution.u64,$5::uuid)",
args, timeout);
if (!rows)
return std::unexpected(failure(rows.error()));
const auto record = rows->size() == 1 ? parse(rows->front()) : std::nullopt;
auto result = record ? checked_record(*record, key, lease) : std::nullopt;
if (!result || !result->aborted || result->trace.request.request != request)
return std::unexpected(LeaseStoreFailure{LeaseStoreError::unknown, {}});
return *result;
}
std::expected<std::optional<CellularMigrationFreeze>, RecoveryFailure>
load_cellular_migration_freeze(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
const LeaseGrant& lease, milliseconds timeout) {
const auto fail = [](RecoveryError kind) { return std::unexpected(RecoveryFailure{kind, {}}); };
const std::array<std::string, 4> args{std::to_string(key.world), std::to_string(key.region),
uuid(lease.holder), std::to_string(lease.epoch)};
auto rows = db.query(std::string("SELECT ") + columns +
" FROM evolution.load_cellular_migration_freeze("
"$1::evolution.u64,$2::evolution.u64,$3::uuid,$4::evolution.u64)",
args, timeout);
if (!rows)
return std::unexpected(RecoveryFailure{RecoveryError::store, failure(rows.error())});
if (rows->empty())
return std::optional<CellularMigrationFreeze>{};
const auto record = rows->size() == 1 ? parse(rows->front()) : std::nullopt;
auto result = record ? checked_record(*record, key, lease) : std::nullopt;
if (!result || result->aborted)
return fail(RecoveryError::history);
const auto s = decode_manifest(record->source), d = decode_manifest(record->destination);
if (!s || !d)
return fail(RecoveryError::manifest);
auto src = objects.get({s->payload_sha256, s->payload_bytes});
auto dst = objects.get({d->payload_sha256, d->payload_bytes});
if (!src || !dst)
return fail(RecoveryError::tail);
const auto replay = derive(*s, *d, *src, *dst, result->trace.request);
if (!replay || *replay != result->trace)
return fail(RecoveryError::history);
return result;
}
} // namespace evo::persistence
+145 -1
View File
@@ -199,6 +199,9 @@ void CellularRegionHost::acquire() {
}
if (!recover_state(*lease) || !committed_.has_value() || !state_)
return;
migration_reload_ = true;
migration_frozen_ = false;
migration_poll_at_ = {};
interval_reload_ = population_mode();
admitted_interval_.reset();
interval_request_.reset();
@@ -412,7 +415,9 @@ void CellularRegionHost::refresh() {
const auto& grant = authority_.grant();
status_.ready = status_.phase == HostPhase::running && state_ && committed_ && grant &&
authority_.may_originate(LeaseClock::now()) && !interval_reload_ &&
!interval_request_.has_value();
!interval_request_.has_value() && !migration_reload_ && !migration_frozen_ &&
!migration_request_.has_value() && !migration_abort_.has_value();
status_.migration_frozen = migration_frozen_;
const auto& interval = admitted_interval_;
status_.ecological_pending_ticks =
interval.has_value() ? interval->end_tick - interval->begin_tick : 0;
@@ -424,6 +429,139 @@ void CellularRegionHost::refresh() {
: 0;
}
void CellularRegionHost::reconcile_migration_freeze() {
if ((!migration_reload_ && !migration_frozen_ && !migration_request_ && !migration_abort_) ||
LeaseClock::now() < migration_poll_at_)
return;
const auto grant = authority_.grant();
if (!grant || !authority_.may_begin_commit(LeaseClock::now()) || !database())
return;
migration_poll_at_ = LeaseClock::now() + std::chrono::seconds{1};
if (migration_request_) {
auto result = persistence::retry_cellular_migration_freeze(
*db_, *grant, *migration_request_, config_.lease.commit_timeout);
if (!result) {
if (result.error().kind == persistence::PublicationError::unknown) {
db_.reset();
++status_.unknown_outcomes;
return;
}
if (result.error().store.kind == LeaseStoreError::fenced ||
result.error().store.kind == LeaseStoreError::head_mismatch) {
authority_.fence();
state_.reset();
committed_.reset();
status_.phase = HostPhase::acquiring;
} else {
authority_.fence();
status_.phase = HostPhase::quarantined;
}
return;
}
migration_request_.reset();
}
if (migration_abort_) {
auto result = persistence::abort_cellular_migration_freeze(
*db_, config_.key, *grant, *migration_abort_, config_.lease.commit_timeout);
if (!result) {
if (result.error().kind == LeaseStoreError::unknown) {
db_.reset();
++status_.unknown_outcomes;
return;
}
authority_.fence();
status_.phase = result.error().kind == LeaseStoreError::fenced ? HostPhase::acquiring
: HostPhase::quarantined;
state_.reset();
committed_.reset();
return;
}
migration_abort_.reset();
}
const auto frozen = persistence::load_cellular_migration_freeze(
*db_, *objects_, config_.key, *grant, config_.lease.commit_timeout);
if (!frozen) {
migration_reload_ = true;
if (frozen.error().kind == persistence::RecoveryError::store) {
db_.reset();
++status_.store_failures;
} else {
authority_.fence();
status_.phase = HostPhase::quarantined;
}
return;
}
migration_frozen_ = frozen->has_value();
migration_reload_ = false;
}
std::expected<void, CellularPublishFailure> CellularRegionHost::prepare_migration_freeze(
const persistence::CommittedCheckpoint& destination,
const persistence::FilesystemObjectStore& destination_objects,
const CellularMigrationRequest& request) {
const auto grant = authority_.grant();
if (!grant.has_value() || !status_.ready || pending_ || !state_ || !committed_ ||
!queued_births_.empty() || !queued_mutations_.empty() || mutations_uncertain_ ||
!pending_endings_.empty() || interval_request_ || admitted_interval_ || interval_reload_ ||
!population_interval_safe(*state_) || !authority_.may_begin_commit(LeaseClock::now()) ||
!database())
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
migration_reload_ = true;
refresh();
const auto result = persistence::prepare_cellular_migration_freeze(
*db_, *objects_, destination_objects, *grant, *committed_, destination, request,
config_.lease.commit_timeout);
if (!result) {
if (result.error().kind == persistence::PublicationError::unknown) {
migration_request_ = result.error().prepared;
db_.reset();
++status_.unknown_outcomes;
} else if (result.error().store.kind == LeaseStoreError::head_mismatch ||
result.error().store.kind == LeaseStoreError::fenced) {
authority_.fence();
state_.reset();
committed_.reset();
status_.phase = HostPhase::acquiring;
}
migration_poll_at_ = {};
refresh();
return std::unexpected(CellularPublishFailure{CellularHostError::publication,
result.error().kind, result.error().store,
result.error().object});
}
migration_poll_at_ = {};
reconcile_migration_freeze();
refresh();
return {};
}
std::expected<void, CellularPublishFailure>
CellularRegionHost::abort_migration_freeze(const persistence::Uuid& request) {
const auto grant = authority_.grant();
if (!grant || !state_ || pending_ || status_.phase != HostPhase::running ||
!authority_.may_begin_commit(LeaseClock::now()) || !database())
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
migration_reload_ = true;
refresh();
const auto result = persistence::abort_cellular_migration_freeze(
*db_, config_.key, *grant, request, config_.lease.commit_timeout);
if (!result) {
if (result.error().kind == LeaseStoreError::unknown) {
migration_abort_ = request;
db_.reset();
++status_.unknown_outcomes;
}
migration_poll_at_ = {};
refresh();
return std::unexpected(
CellularPublishFailure{CellularHostError::publication, {}, result.error(), {}});
}
migration_poll_at_ = {};
reconcile_migration_freeze();
refresh();
return {};
}
const HostStatus& CellularRegionHost::step() {
if (status_.phase == HostPhase::quarantined || status_.phase == HostPhase::stopped)
return status_;
@@ -433,6 +571,12 @@ const HostStatus& CellularRegionHost::step() {
resolve();
else if (!state_)
acquire();
if (!pending_ && state_ && status_.phase == HostPhase::running)
reconcile_migration_freeze();
if (migration_reload_ || migration_frozen_ || migration_request_ || migration_abort_) {
refresh();
return status_;
}
if (!pending_ && state_ && status_.phase == HostPhase::running)
reconcile_interval();
if (!pending_ && state_ && mutations_uncertain_ && status_.phase == HostPhase::running)
+2
View File
@@ -74,6 +74,8 @@ std::string render_metrics(const HostStatus& s, const OpsCounters& c) {
metric(out, "evo_region_tick_debt", "gauge", "Due ticks not yet simulated", s.tick_debt);
metric(out, "evo_region_ecological_pending_ticks", "gauge", "Durable admitted ecological ticks",
s.ecological_pending_ticks);
metric(out, "evo_region_migration_frozen", "gauge", "Known durable biological source freeze",
s.migration_frozen ? 1 : 0);
metric(out, "evo_region_tick_debt_max", "gauge", "Largest tick debt since recovery",
s.max_tick_debt);
metric(out, "evo_region_queued_inputs", "gauge", "Admitted inputs awaiting a tick", s.queued);
+4 -2
View File
@@ -72,6 +72,7 @@ void exposition_is_bounded_and_well_formed() {
s.committed_tick = 41;
s.lease_remaining_ms = 1234;
s.ecological_pending_ticks = 64;
s.migration_frozen = true;
OpsCounters c;
c.queue_capacity_inputs = 64;
c.queue_capacity_bytes = 65536;
@@ -93,10 +94,11 @@ void exposition_is_bounded_and_well_formed() {
CHECK(text.find("evo_region_ready 1\n") != std::string::npos);
CHECK(text.find("evo_region_commit_lag_ticks 9\n") != std::string::npos);
CHECK(text.find("evo_region_ecological_pending_ticks 64\n") != std::string::npos);
CHECK(text.find("evo_region_migration_frozen 1\n") != std::string::npos);
CHECK(text.find("evo_region_lease_remaining_milliseconds 1234\n") != std::string::npos);
CHECK(text.find("evo_region_dropped_total{reason=\"fenced\"} 5\n") != std::string::npos);
// Series count is fixed: 6 phases + 4 reasons + 23 unlabelled families.
CHECK(samples.size() == 6 + 4 + 23);
// Series count is fixed: 6 phases + 4 reasons + 24 unlabelled families.
CHECK(samples.size() == 6 + 4 + 24);
// Worst-case values keep the size bounded and never underflow the lag.
HostStatus big;
+373
View File
@@ -745,6 +745,378 @@ void boundary_upgrades(Fixture& f) {
require(f.raw->format_version() == cellular_lives_migration_format,
"reverse transition downgraded accounting owner");
}
void migration_freezes(Fixture& f) {
const RegionKey destination_key{f.key.world, 2};
auto destination_objects =
take(FilesystemObjectStore::open(environment("PGHOST"), destination_key,
max_cellular_population_bytes),
"migration destination objects");
auto destination = opening(30001, true, 4, false, 2);
auto destination_head =
take(provision_cellular_region(*f.owner, *destination_objects, destination_key, 1,
*destination, f.id(), f.build, 5s),
"migration empty destination genesis");
auto destination_lease =
take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s),
"migration destination authority");
while (destination->tick() < f.raw->tick()) {
const auto count = std::min(std::uint64_t{64}, f.raw->tick() - destination->tick());
CellularBatch inputs{{f.key.world, 2, 1, destination_head.head.commit,
destination_head.head.digest, destination->tick(),
destination->tick() + count},
{}};
inputs.ticks.resize(static_cast<std::size_t>(count));
destination =
take(replay_cellular_batch(*destination, inputs), "empty destination biology");
destination_head = take(publish_cellular_batch(*f.db, *destination_objects,
destination_lease, destination_head, inputs,
*destination, f.id(), f.build, 5s),
"empty destination tick publication")
.committed;
}
destination_head =
take(migrate_cellular_boundary_profile(*f.db, *destination_objects, destination_lease,
destination_head, f.id(), f.build, 5s),
"destination accounting upgrade");
destination = take(destination->migration_upgrade_candidate(), "destination upgraded oracle");
const std::array<Sha256Digest, 1> builds{f.build};
const auto request_for = [&] {
const auto member = std::ranges::find_if(
f.raw->lives(), [](const auto& life) { return life.account == LifeAccount{}; });
require(member != f.raw->lives().end(), "freeze fixture lacks real wildlife");
return CellularMigrationRequest{
f.id(),
{1, f.head.head.commit, f.id(), f.head.head.digest},
{2, destination_head.head.commit, f.id(), destination_head.head.digest},
member->identity.genome};
};
for (unsigned representation = 0; representation < 2; ++representation) {
if (representation == 1) {
f.head = take(migrate_cellular_to_population(*f.db, *f.objects, f.lease, f.head, f.id(),
f.build, {true, 1200, 1800}, 5s),
"freeze cohort source entry");
recover(f);
destination_head = take(migrate_cellular_to_population(
*f.db, *destination_objects, destination_lease,
destination_head, f.id(), f.build, {true, 1200, 1800}, 5s),
"freeze cohort destination entry");
}
const auto source_before = f.head;
const auto source_bytes = f.raw->snapshot();
const auto destination_before = destination_head;
const auto request = request_for();
const auto prepare = [&](const CellularMigrationRequest& requested,
std::chrono::milliseconds timeout = 5s) {
return prepare_cellular_migration_freeze(*f.db, *f.objects, *destination_objects,
f.lease, f.head, destination_head, requested,
timeout);
};
if (representation == 0) {
Uuid account{};
account.back() = 1;
take(f.db->query("BEGIN", {}, 5s), "pending player choice before freeze");
const auto control =
take(claim_account_control(*f.db, f.key, f.lease, account, 2, 9900, 60000ms, 5s),
"pending choice current control");
const auto input = encode_cellular_mutation_input(
{account, 9900, f.head.head.end_tick + 1, {100, 1, 1, 1}});
take(reserve_cellular_mutation(*f.db, f.key, f.lease, account, control.generation, 9900,
input, 5s),
"reserve pre-freeze player choice");
const auto pending = prepare(request);
take(f.db->query("ROLLBACK", {}, 5s), "rollback pre-freeze player choice");
require(!pending &&
std::string(pending.error().store.cause.sqlstate.data(), 5) == "EVL18",
"preparation stranded a player choice");
} else {
take(f.db->query("BEGIN", {}, 5s), "prepare with admitted work transaction");
take(admit_population_interval(*f.db, f.key, f.lease, f.head.head,
f.head.head.end_tick + 20, 5s),
"admitted work before freeze");
const auto pending = prepare(request);
take(f.db->query("ROLLBACK", {}, 5s), "rollback pending-work preparation fixture");
require(!pending &&
std::string(pending.error().store.cause.sqlstate.data(), 5) == "EVL18",
"preparation skipped durable ecological work");
}
auto frozen = take(prepare(request), "durable source freeze");
const CellularMigrationPrepared prepared{
f.key, take(encode_migration_trace(frozen.trace), "frozen canonical trace")};
auto malformed = prepared.trace;
malformed.back() ^= 1;
const std::array<std::string, 3> malformed_args{hexadecimal(f.lease.holder, false),
std::to_string(f.lease.epoch),
hexadecimal(malformed)};
const auto malformed_sql = f.db->query("SELECT * FROM "
"evolution.prepare_cellular_migration_freeze(30201,"
"1,$1::uuid,$2::evolution.u64,$3::bytea)",
malformed_args, 5s);
require(!malformed_sql && std::string(malformed_sql.error().sqlstate.data(), 5) == "22023",
"SQL accepted an invalid trace checksum before duplicate lookup");
require(
!frozen.aborted && frozen.trace.request == request &&
take(take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s),
"replay frozen capsule"),
"missing durable freeze")
.trace == frozen.trace &&
take(load_region_checkpoint(*f.db, f.key, 5s), "frozen source head").head ==
source_before.head &&
take(load_region_checkpoint(*f.db, destination_key, 5s), "unowned destination head")
.head == destination_before.head,
"preparation changed either owner or lost its full replay capsule");
require(
!take(retry_cellular_migration_freeze(*f.db, f.lease, prepared, 5s), "duplicate freeze")
.aborted,
"duplicate freeze lost its state");
auto conflict_trace = frozen.trace;
++conflict_trace.transferred.energy.value;
auto conflict = retry_cellular_migration_freeze(
*f.db, f.lease, {f.key, take(encode_migration_trace(conflict_trace), "conflict trace")},
5s);
require(!conflict && conflict.error().store.kind == LeaseStoreError::commit_conflict &&
!prepare(request_for()),
"freeze accepted changed bytes or replaced pending request");
if (representation == 0) {
const auto h = f.header(1);
const auto inputs = take(unattended_population_inputs(*f.raw, h), "frozen raw inputs");
auto advanced = take(replay_cellular_batch(*f.raw, inputs), "frozen detached future");
auto denied = publish_cellular_batch(*f.db, *f.objects, f.lease, f.head, inputs,
*advanced, f.id(), f.build, 5s);
require(!denied &&
std::string(denied.error().store.cause.sqlstate.data(), 5) == "EVL18" &&
!migrate_cellular_to_population(*f.db, *f.objects, f.lease, f.head, f.id(),
f.build, {}, 5s),
"raw source progressed or changed profile while frozen");
Uuid account{};
account.back() = 1;
const auto input = encode_cellular_mutation_input(
{account, 9901, f.head.head.end_tick + 1, {100, 1, 1, 1}});
take(f.db->query("BEGIN", {}, 5s), "current control while frozen fixture");
const auto control =
take(claim_account_control(*f.db, f.key, f.lease, account, 2, 9901, 60000ms, 5s),
"current frozen-source account control");
const auto choice = reserve_cellular_mutation(*f.db, f.key, f.lease, account,
control.generation, 9901, input, 5s);
take(f.db->query("ROLLBACK", {}, 5s), "rollback frozen control fixture");
require(!choice, "frozen raw owner admitted a new mutation reservation");
if (std::string(choice.error().cause.sqlstate.data(), 5) != "EVL18")
throw std::runtime_error("frozen mutation guard SQLSTATE=" +
std::string(choice.error().cause.sqlstate.data(), 5));
} else {
require(!admit_population_interval(*f.db, f.key, f.lease, f.head.head,
f.head.head.end_tick + 20, 5s) &&
!materialize_population_profile(*f.db, *f.objects, f.lease, f.head, f.id(),
f.build, 5s),
"population source admitted work or changed profile while frozen");
}
const auto capsule_manifest =
take(decode_manifest(destination_head.manifest), "capsule manifest");
const auto capsule_path =
f.objects->directory() / hexadecimal(capsule_manifest.payload_sha256, false);
const auto backup = capsule_path.string() + ".freeze-test";
std::filesystem::rename(capsule_path, backup);
const auto missing = load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s);
std::filesystem::rename(backup, capsule_path);
require(!missing && missing.error().kind == RecoveryError::tail,
"freeze recovery fell back to destination live state without the retained capsule");
char first{};
{
std::fstream file(capsule_path, std::ios::binary | std::ios::in | std::ios::out);
file.get(first);
file.seekp(0);
file.put(static_cast<char>(static_cast<unsigned char>(first) ^ 1U));
file.flush();
require(file.good(), "corrupt same-length capsule fixture");
}
const auto corrupt = load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s);
{
std::fstream file(capsule_path, std::ios::binary | std::ios::in | std::ios::out);
file.put(first);
file.flush();
require(file.good(), "restore same-length capsule fixture");
}
require(!corrupt && corrupt.error().kind == RecoveryError::tail,
"corrupt capsule bypassed strong object binding");
const auto old_lease = f.lease;
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release frozen source owner");
f.lease = take(acquire_region_lease(*f.db, f.key, f.build, 60000ms, 5s),
"new frozen source owner");
require(take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s),
"recover freeze under new epoch")
.has_value() &&
!abort_cellular_migration_freeze(*f.db, f.key, old_lease, request.request, 5s),
"new epoch released a freeze or old epoch aborted it");
require(
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, request.request, 5s),
"explicit source abort")
.aborted &&
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, request.request, 5s),
"duplicate source abort")
.aborted &&
take(retry_cellular_migration_freeze(*f.db, f.lease, prepared, 5s),
"retry aborted preparation")
.aborted &&
!take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s),
"aborted pointer load")
.has_value(),
"abort was not durable or preparation retry reopened it");
require(take(load_region_checkpoint(*f.db, f.key, 5s), "aborted source head").head ==
source_before.head &&
f.raw->snapshot() == source_bytes,
"abort changed biological stock/time/history");
auto false_trace = frozen.trace;
false_trace.request.request = f.id();
false_trace.request.source.commit = f.id();
false_trace.request.destination.commit = f.id();
++false_trace.transferred.energy.value;
take(retry_cellular_migration_freeze(*f.db, f.lease,
{f.key, take(encode_migration_trace(false_trace),
"canonical false inventory trace")},
5s),
"metadata-only false preparation");
const auto false_replay =
load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s);
require(!false_replay && false_replay.error().kind == RecoveryError::history,
"freeze recovery trusted declared material without complete biological replay");
take(
abort_cellular_migration_freeze(*f.db, f.key, f.lease, false_trace.request.request, 5s),
"explicitly abort false preparation without moving stock");
// A genuine unknown reply retains exactly the chosen cut and group.
auto locker = connection(true);
take(locker->query("BEGIN", {}, 5s), "freeze ambiguity lock");
take(locker->query("SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND "
"region_id=1 FOR UPDATE",
{}, 5s),
"hold freeze reply");
auto timed = connection();
auto unknown =
prepare_cellular_migration_freeze(*timed, *f.objects, *destination_objects, f.lease,
f.head, destination_head, request_for(), 300ms);
require(!unknown && unknown.error().kind == PublicationError::unknown &&
!unknown.error().prepared.trace.empty(),
"freeze ambiguity lost its immutable request");
take(locker->query("ROLLBACK", {}, 5s), "release freeze ambiguity");
frozen = take(retry_cellular_migration_freeze(*f.db, f.lease, unknown.error().prepared, 5s),
"reconcile unknown freeze");
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, frozen.trace.request.request,
5s),
"abort reconciled freeze");
// Both sides of preparation's transaction boundary, with real process death.
for (unsigned point = 0; point < 2; ++point) {
const auto killed_request = request_for();
const auto child = ::fork();
require(child >= 0, "freeze publisher fork");
if (child == 0) {
try {
auto db = connection();
PublishHooks hooks;
if (point == 0)
hooks.before_transaction = [] { ::kill(::getpid(), SIGKILL); };
if (!prepare_cellular_migration_freeze(*db, *f.objects, *destination_objects,
f.lease, f.head, destination_head,
killed_request, 5s, hooks))
::_exit(41);
::kill(::getpid(), SIGKILL);
} catch (...) {
::_exit(42);
}
::_exit(43);
}
int status{};
require(::waitpid(child, &status, 0) == child && WIFSIGNALED(status) &&
WTERMSIG(status) == SIGKILL &&
take(load_cellular_migration_freeze(*f.db, *f.objects, f.key, f.lease, 5s),
"recover killed preparation")
.has_value() == (point == 1),
"publisher death changed source ownership freeze");
if (point == 1)
take(abort_cellular_migration_freeze(*f.db, f.key, f.lease, killed_request.request,
5s),
"abort killed preparation");
}
}
// The owning host must retain the freeze through drain/recreation with entry off.
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release fixture to frozen host");
const auto make_host = [&] {
auto objects = take(FilesystemObjectStore::open(environment("PGHOST"), f.key,
max_cellular_population_bytes),
"freeze host objects");
CellularHostConfig config{f.key, 1, {}, f.build, {f.build}};
config.lease.commit_timeout = 300ms;
return take(
CellularRegionHost::create(
config,
[] { return std::expected<std::unique_ptr<Postgres>, PgFailure>{connection()}; },
std::move(objects)),
"freeze host");
};
auto host = make_host();
require(host->step().ready, "source host before freeze");
const auto before = host->state()->snapshot();
const auto host_request = request_for();
take(host->prepare_migration_freeze(destination_head, *destination_objects, host_request),
"host source freeze");
require(!host->status().ready && host->status().migration_frozen &&
!host->advance_unattended(64) && host->state()->snapshot() == before,
"frozen host advanced or originated current output");
host->drain();
host = make_host();
require(!host->step().ready && host->status().migration_frozen &&
host->state()->snapshot() == before && !host->authority().has_value(),
"replacement host implicitly aborted or ignored its freeze");
take(host->abort_migration_freeze(host_request.request), "replacement host explicit abort");
require(host->step().ready && !host->status().migration_frozen &&
host->state()->snapshot() == before,
"host failed to resume the identical source after confirmed abort");
auto locker = connection(true);
take(locker->query("BEGIN", {}, 5s), "host unknown freeze lock");
take(
locker->query(
"SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND region_id=1 FOR UPDATE",
{}, 5s),
"hold host freeze reply");
const auto uncertain_request = request_for();
const auto uncertain =
host->prepare_migration_freeze(destination_head, *destination_objects, uncertain_request);
require(!uncertain && !host->status().ready, "host originated during uncertain freeze");
take(locker->query("ROLLBACK", {}, 5s), "release host freeze reply");
for (unsigned attempt = 0; attempt < 200 && !host->status().migration_frozen; ++attempt) {
host->step();
std::this_thread::sleep_for(25ms);
}
require(host->status().migration_frozen && !host->status().ready &&
host->state()->snapshot() == before,
"host chose different biology after unknown preparation");
take(locker->query("BEGIN", {}, 5s), "host unknown abort lock");
take(
locker->query(
"SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND region_id=1 FOR UPDATE",
{}, 5s),
"hold host abort reply");
const auto unknown_abort = host->abort_migration_freeze(uncertain_request.request);
require(!unknown_abort && !host->status().ready, "unconfirmed abort resumed the source");
take(locker->query("ROLLBACK", {}, 5s), "release host abort reply");
for (unsigned attempt = 0; attempt < 200 && !host->step().ready; ++attempt)
std::this_thread::sleep_for(25ms);
require(host->status().ready && !host->status().migration_frozen &&
host->state()->snapshot() == before,
"host failed exact ambiguous-abort reconciliation");
host->drain();
f.lease = take(acquire_region_lease(*f.db, f.key, f.build, 60000ms, 5s),
"return freeze fixture authority");
f.head = take(
materialize_population_profile(*f.db, *f.objects, f.lease, f.head, f.id(), f.build, 5s),
"resume frozen population as raw owner");
f.raw = take(recover_cellular_checkpoint(*f.db, *f.objects, f.key, f.lease, builds, 5s),
"source after frozen profile materialization")
.state;
take(release_region_lease(*f.db, destination_key, destination_lease, 5s),
"release freeze destination");
}
void host_ownership_and_admission(Fixture& f) {
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release fixture host authority");
const auto make_host = [&](bool enabled) {
@@ -886,6 +1258,7 @@ void run_population_publication_tests() {
missing_and_corrupt_objects(fixture);
isolation_and_reverse(fixture);
boundary_upgrades(fixture);
migration_freezes(fixture);
host_ownership_and_admission(fixture);
std::puts("population publication, exact recovery, process loss and fencing passed");
}
+2 -1
View File
@@ -212,6 +212,7 @@ void upgrade_across_migration_and_barrier() {
" -v migration_sha_44=$(s 0044_exact_population_publication.sql)"
" -v migration_sha_45=$(s 0045_owned_population_intervals.sql)"
" -v migration_sha_46=$(s 0046_boundary_accounting_upgrade.sql)"
" -v migration_sha_47=$(s 0047_cellular_migration_freeze.sql)"
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
" > /dev/null 2>&1") != 0);
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
@@ -229,7 +230,7 @@ void upgrade_across_migration_and_barrier() {
"tools/migrate-postgres.sh\" > /dev/null") == 0);
CHECK(history(*owner_db) ==
"1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,"
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46");
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47");
CHECK(scalar(*owner_db, "SELECT (c.condeferrable AND c.condeferred AND "
"c.confrelid = 'evolution.region_commits'::regclass)::text "
"FROM pg_catalog.pg_constraint c WHERE c.conrelid = "
+5 -5
View File
@@ -129,7 +129,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
v2_sha="${v2_sha%% *}"
@@ -150,7 +150,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 46:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 47:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
@@ -204,7 +204,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
phase permission-audit
@@ -257,9 +257,9 @@ expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
phase checksum-drift
# Unknown future versions and checksum drift must not be repaired silently.
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=47 WHERE version=46'
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=48 WHERE version=47'
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=46 WHERE version=47"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=47 WHERE version=48"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14"
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13"
+3
View File
@@ -104,6 +104,8 @@ digest_45="$(sha256sum "$root/migrations/0045_owned_population_intervals.sql")"
digest_45="${digest_45%% *}"
digest_46="$(sha256sum "$root/migrations/0046_boundary_accounting_upgrade.sql")"
digest_46="${digest_46%% *}"
digest_47="$(sha256sum "$root/migrations/0047_cellular_migration_freeze.sql")"
digest_47="${digest_47%% *}"
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
@@ -143,5 +145,6 @@ psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
--set="migration_sha_44=$digest_44" \
--set="migration_sha_45=$digest_45" \
--set="migration_sha_46=$digest_46" \
--set="migration_sha_47=$digest_47" \
--set="runtime_role=$runtime_role" \
--file="$root/migrations/apply.sql"