diff --git a/docs/adr/0061-checkpoint-publication.md b/docs/adr/0061-checkpoint-publication.md index 7611ec9..a734e6a 100644 --- a/docs/adr/0061-checkpoint-publication.md +++ b/docs/adr/0061-checkpoint-publication.md @@ -54,8 +54,12 @@ computes commit digests, and the host checks that the receipt matches. are named by the hex SHA-256 of their content. - Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a link never replaces an existing name), and then fsyncs the directory. -- Existing identical content is reused. Differing content already under a key is - reported as corrupt and never overwritten. +- Existing identical content is reused only after its file and directory are + fsynced again, whether it came from an earlier attempt whose directory sync failed + or from a concurrent installer. A visible name alone is not durability (review + fix). Namespace directories sync their parent on every open, not only on + creation. Differing content already under a key is reported as corrupt and never + overwritten. - Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating, and verify the digest. - There is no delete or GC operation: unreferenced uploads are orphans. diff --git a/docs/adr/0063-batch-outcomes.md b/docs/adr/0063-batch-outcomes.md index f0a89c6..4d5bf2a 100644 --- a/docs/adr/0063-batch-outcomes.md +++ b/docs/adr/0063-batch-outcomes.md @@ -63,6 +63,13 @@ parameter; these profiles can produce at most 2. The decision identity is outbox rows only a first delivery timestamp (EVL09). - The runtime role's audited function set grows to ten. +**Upgrades** from v3 or v4 run `migrations/pre_v5_guard.sql` inside the same +transaction, before 0005. The guard refuses (EVL10), changing nothing, if any +published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold +terminal organisms without decisions, and v5 recovery would quarantine it. +Genesis-only regions pass, because tick 0 has no terminal organisms. A verified +legacy-history conversion is future work; there is none now. + **Recovery** loads decisions and requires lifecycle agreement: - one death per terminal actor, none for an active actor; - no death after the head tick, and organisms match the profile; diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 8f5df8d..0d93854 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -2265,3 +2265,34 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate 0.75-0.86 ms. - Not executed: Docker/Gitea CI, remote object store, process kill, a broker consumer, an injected failure between commit insert and statement end, and TSan. + +### M7.5 review fixes - 2026-09-28 + +- P1 object durability: + - `put()` now re-fsyncs the file and the directory before reusing an existing + object, on both the found-existing and the lost link-race paths. + - Namespace opens always sync the parent directory. + - A test-only `ObjectStoreFaults` hook injects directory-sync failures and + counts successful syncs. + - Tests: a namespace-creation sync failure, then a reopen still failing, then + success with exactly 2 syncs. An install sync failure leaves the object + visible but unacknowledged, three retries while syncs fail all return io with + no extra syncs, then success with +1 sync; every reuse adds a sync. + - Publication with 1, 1 and 2 injected failures returns `upload`/io and sends + no manifest (head stays 0), then publishes once syncs succeed. + - Reverting the reuse sync or the always-sync namespace behavior makes these + tests fail. +- P1 pre-v5 history: + - `pre_v5_guard.sql` (EVL10) runs before 0005 in every upgrade path. 0005 and + its checksum are unchanged. + - CI builds a real v4 database: v1 genesis, a lease, and a sequence-1 + publication through the v4 function. The v5 upgrade is refused with the + expected message; history stays 1..4, both commits remain, and no decision + table is created. Replacing the guard with a no-op makes that check fail. + - A committed death published without any decision (the state an accepted + legacy upgrade would produce) recovers as `history`, quarantined and never + served. +- Reran GCC14 Release (3 runs), clang-tidy analysis (1) and ASan/UBSan (2) + PostgreSQL suites, all passing, and the repository format check. The changes + are Linux-only persistence code, SQL and tests; the portable targets and + Windows are unchanged. diff --git a/docs/infrastructure/POSTGRESQL.md b/docs/infrastructure/POSTGRESQL.md index 04a5e79..1a5474e 100644 --- a/docs/infrastructure/POSTGRESQL.md +++ b/docs/infrastructure/POSTGRESQL.md @@ -120,7 +120,9 @@ Triggers make commits and decisions immutable (EVL09); outbox rows may only gain a first delivery time. Owners can disable triggers, so treat that as an audited break-glass action. EVL08 means an outcome was already final. After an ambiguous publication, resolve the exact prepared bytes before doing anything -else. +else. Upgrading a v3/v4 database that has published checkpoints (not just +genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions, +or wait for an explicit legacy-history conversion. There is no operator CLI for provisioning yet, and no real world has been provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads diff --git a/include/evo/persistence/object_store.hpp b/include/evo/persistence/object_store.hpp index ea0d321..b3bc98b 100644 --- a/include/evo/persistence/object_store.hpp +++ b/include/evo/persistence/object_store.hpp @@ -20,6 +20,12 @@ enum class ObjectError : std::uint8_t { bounds, // Larger than the configured maximum; rejected before allocation. io, // Write/sync/link failure; the object is absent or unverified. }; +// Test-only fault injection and accounting for directory fsyncs. When passed to +// open() it must outlive the store; production callers pass nothing. +struct ObjectStoreFaults { + std::uint32_t fail_directory_syncs{}; // The next N directory syncs report failure. + std::uint64_t directory_syncs{}; // Directory syncs that succeeded. +}; struct VerifiedObject { Sha256Digest sha256{}; std::uint64_t bytes{}; @@ -28,9 +34,11 @@ struct VerifiedObject { class FilesystemObjectStore { public: // trusted_root must be an existing real directory (not a symlink). Namespace - // directories w/r are created 0700 and their parents synced. + // directories w/r are created 0700; their parents are synced on + // every open, so a creation whose sync failed earlier is made durable now. [[nodiscard]] static std::expected, ObjectError> - open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes); + open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes, + ObjectStoreFaults* faults = nullptr); FilesystemObjectStore(const FilesystemObjectStore&) = delete; FilesystemObjectStore& operator=(const FilesystemObjectStore&) = delete; FilesystemObjectStore(FilesystemObjectStore&&) = delete; @@ -38,7 +46,9 @@ class FilesystemObjectStore { ~FilesystemObjectStore(); // Writes a private temporary file, fsyncs it, links it to its key without // replacing anything, then fsyncs the directory. Identical bytes already stored - // are verified and reused; a differing existing object is corrupt, never replaced. + // (an earlier attempt whose directory sync failed, or a concurrent installer) + // are verified and fsynced, file and directory, before success is reported. A + // differing existing object is corrupt, never replaced. [[nodiscard]] std::expected put(std::span bytes) const; // Reads only an exact-length regular file and verifies its SHA-256. @@ -49,9 +59,13 @@ class FilesystemObjectStore { } private: - FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum); + FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum, + ObjectStoreFaults* faults); + [[nodiscard]] std::expected + durable_existing(const VerifiedObject& object) const; std::filesystem::path directory_; int descriptor_; std::size_t maximum_; + ObjectStoreFaults* faults_; }; } // namespace evo::persistence diff --git a/migrations/apply.sql b/migrations/apply.sql index 0712639..4b7eade 100644 --- a/migrations/apply.sql +++ b/migrations/apply.sql @@ -27,18 +27,22 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') || ',5:' || :'migration_sha_5' AS at_v5 \gset \if :at_v5 \elif :at_v4 + \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \elif :at_v3 \ir 0004_lineage_profile.sql + \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \elif :at_v2 \ir 0003_region_checkpoints.sql \ir 0004_lineage_profile.sql + \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \elif :at_v1 \ir 0002_region_fencing.sql \ir 0003_region_checkpoints.sql \ir 0004_lineage_profile.sql + \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \else \echo 'Unknown or changed migration history' @@ -49,6 +53,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0002_region_fencing.sql \ir 0003_region_checkpoints.sql \ir 0004_lineage_profile.sql + \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql \endif -- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes diff --git a/migrations/pre_v5_guard.sql b/migrations/pre_v5_guard.sql new file mode 100644 index 0000000..d4bf8d1 --- /dev/null +++ b/migrations/pre_v5_guard.sql @@ -0,0 +1,17 @@ +-- Runs inside apply.sql's transaction immediately before 0005 (ADR-0063). Not a +-- migration and not recorded in history; it only refuses an unsafe upgrade. +-- +-- Pre-v5 published checkpoints (sequence > 0) may hold organisms that died with no +-- durable decision row, because decisions did not exist before v5. The database +-- cannot inspect payloads to prove otherwise, and v5 recovery requires one +-- decision per terminal actor, so accepting them would make those regions +-- unrecoverable (RecoveryError::history). Genesis-only regions are safe: a tick-0 +-- state has no terminal organisms. Until an explicit, verified legacy-history +-- conversion exists, refuse and change nothing. +DO $$ +BEGIN + IF EXISTS (SELECT 1 FROM evolution.region_commits WHERE commit_sequence > 0) THEN + RAISE EXCEPTION 'pre-v5 published checkpoints lack durable decisions; convert or re-provision before v5' + USING ERRCODE = 'EVL10'; + END IF; +END $$; diff --git a/src/persistence/object_store.cpp b/src/persistence/object_store.cpp index c645dcb..7b7449f 100644 --- a/src/persistence/object_store.cpp +++ b/src/persistence/object_store.cpp @@ -48,15 +48,26 @@ std::string key_name(const Sha256Digest& d) { } return out; } -// Opens or creates a private subdirectory without following symlinks. -int subdirectory(int parent, const std::string& name) { - bool created = false; - if (::mkdirat(parent, name.c_str(), 0700) == 0) - created = true; - else if (errno != EEXIST) +// Every directory fsync goes through here so tests can inject failures. +bool sync_directory(int fd, ObjectStoreFaults* faults) { + if (faults && faults->fail_directory_syncs > 0) { + --faults->fail_directory_syncs; + return false; + } + if (::fsync(fd) != 0) + return false; + if (faults) + ++faults->directory_syncs; + return true; +} +// Opens or creates a private subdirectory without following symlinks. The parent +// is synced even when the entry already existed: an earlier creation may have +// failed its sync, and an unsynced entry must not be trusted as durable. +int subdirectory(int parent, const std::string& name, ObjectStoreFaults* faults) { + if (::mkdirat(parent, name.c_str(), 0700) != 0 && errno != EEXIST) return -1; const int fd = ::openat(parent, name.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC); - if (fd >= 0 && created && ::fsync(parent) != 0) { + if (fd >= 0 && !sync_directory(parent, faults)) { static_cast(::close(fd)); return -1; } @@ -74,8 +85,9 @@ bool write_all(int fd, std::span bytes) { return true; } // Exact-length regular file whose content has the expected digest. -std::expected, ObjectError> read_verified(int directory, - const VerifiedObject& object) { +// With sync_file, the verified file is also fsynced before it is trusted as durable. +std::expected, ObjectError> +read_verified(int directory, const VerifiedObject& object, bool sync_file = false) { const auto name = key_name(object.sha256); const Descriptor fd{::openat(directory, name.c_str(), O_RDONLY | O_NOFOLLOW | O_CLOEXEC)}; if (fd.get() < 0) @@ -109,13 +121,16 @@ std::expected, ObjectError> read_verified(int director return std::unexpected{ObjectError::io}; if (*digest != object.sha256) return std::unexpected{ObjectError::corrupt}; + if (sync_file && ::fsync(fd.get()) != 0) + return std::unexpected{ObjectError::io}; return bytes; } } // namespace FilesystemObjectStore::FilesystemObjectStore(std::filesystem::path directory, int descriptor, - std::size_t maximum) - : directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum) {} + std::size_t maximum, ObjectStoreFaults* faults) + : directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum), + faults_(faults) {} FilesystemObjectStore::~FilesystemObjectStore() { static_cast(::close(descriptor_)); @@ -123,7 +138,7 @@ FilesystemObjectStore::~FilesystemObjectStore() { std::expected, ObjectError> FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey key, - std::size_t max_object_bytes) { + std::size_t max_object_bytes, ObjectStoreFaults* faults) { if (key.world == 0 || key.region == 0 || max_object_bytes == 0 || max_object_bytes > maximum_object_limit || trusted_root.empty()) return std::unexpected{ObjectError::invalid}; @@ -133,14 +148,14 @@ FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey return std::unexpected{ObjectError::invalid}; const auto world = "w" + std::to_string(key.world); const auto region = "r" + std::to_string(key.region); - const Descriptor world_fd{subdirectory(root.get(), world)}; + const Descriptor world_fd{subdirectory(root.get(), world, faults)}; if (world_fd.get() < 0) return std::unexpected{ObjectError::io}; - Descriptor region_fd{subdirectory(world_fd.get(), region)}; + Descriptor region_fd{subdirectory(world_fd.get(), region, faults)}; if (region_fd.get() < 0) return std::unexpected{ObjectError::io}; return std::unique_ptr(new FilesystemObjectStore( - trusted_root / world / region, region_fd.release(), max_object_bytes)); + trusted_root / world / region, region_fd.release(), max_object_bytes, faults)); } std::expected @@ -153,14 +168,8 @@ FilesystemObjectStore::put(std::span bytes) const { if (!digest) return std::unexpected{ObjectError::io}; const VerifiedObject object{*digest, bytes.size()}; - const auto existing = [&]() -> std::expected { - auto stored = read_verified(descriptor_, object); - if (!stored) - return std::unexpected{stored.error()}; - return object; - }; - // Immutable: identical content is reused; anything else under the key is corrupt. - if (auto found = existing(); found || found.error() != ObjectError::missing) + // Immutable: identical content is reused (durably); anything else is corrupt. + if (auto found = durable_existing(object); found || found.error() != ObjectError::missing) return found; const auto name = key_name(*digest); @@ -180,11 +189,23 @@ FilesystemObjectStore::put(std::span bytes) const { const bool raced = errno == EEXIST; discard(); if (raced) - return existing(); + return durable_existing(object); return std::unexpected{ObjectError::io}; } discard(); - if (::fsync(descriptor_) != 0) + if (!sync_directory(descriptor_, faults_)) + return std::unexpected{ObjectError::io}; + return object; +} + +// A visible name is not proof of durability: its installer's directory sync may +// have failed. Re-sync the verified file and the directory before reusing it. +std::expected +FilesystemObjectStore::durable_existing(const VerifiedObject& object) const { + auto stored = read_verified(descriptor_, object, true); + if (!stored) + return std::unexpected{stored.error()}; + if (!sync_directory(descriptor_, faults_)) return std::unexpected{ObjectError::io}; return object; } diff --git a/tests/region_checkpoint_store_tests.cpp b/tests/region_checkpoint_store_tests.cpp index 1ced712..e5b2ea2 100644 --- a/tests/region_checkpoint_store_tests.cpp +++ b/tests/region_checkpoint_store_tests.cpp @@ -143,6 +143,35 @@ void object_store_is_immutable_and_verified() { } } +// Review P1: a name made visible by an install whose directory sync failed is not +// durable. Neither a retry that finds it nor a namespace reopen may report success +// until a directory sync actually succeeds. +void directory_sync_failures_are_never_acknowledged() { + const auto root = scratch("syncs"); + ObjectStoreFaults faults; + faults.fail_directory_syncs = 1; // Creating w1 under the root. + CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io); + CHECK(fs::exists(root / "w1")); // Created, but its entry is unconfirmed. + faults.fail_directory_syncs = 1; // Reopen finds it and must sync again. + CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io); + auto store = FilesystemObjectStore::open(root, {1, 5}, 1024, &faults); + CHECK(store && faults.directory_syncs == 2); // root->w1 and w1->r5. + + const auto payload = bytes_of("durable only after a directory sync succeeds"); + const VerifiedObject object{*sha256(payload), payload.size()}; + faults.fail_directory_syncs = 1; + CHECK((*store)->put(payload).error() == ObjectError::io); + CHECK((*store)->get(object)); // Visible, yet never acknowledged. + faults.fail_directory_syncs = 3; // Retries while the directory still cannot sync. + for (int attempt = 0; attempt < 3; ++attempt) + CHECK((*store)->put(payload).error() == ObjectError::io); + CHECK(faults.directory_syncs == 2); + auto retried = (*store)->put(payload); + CHECK(retried && *retried == object && faults.directory_syncs == 3); + // Every reuse re-establishes durability rather than trusting the visible name. + CHECK((*store)->put(payload) && faults.directory_syncs == 4); +} + struct Region { RegionKey key{7, 3}; std::unique_ptr owner, runtime; @@ -675,13 +704,76 @@ void durable_outcomes_and_ambiguity() { CHECK(!undo && std::string(undo.error().sqlstate.data()) == "EVL09"); // Stays quarantined. CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s)); } +// Publication cannot acknowledge a checkpoint whose objects are not yet durable. +void publication_waits_for_directory_durability() { + auto r = provisioned("sync-publish", 9, 0xA8); + ObjectStoreFaults faults; + auto faulty = FilesystemObjectStore::open(r.objects->directory().parent_path().parent_path(), + r.key, max_environmental_payload_bytes, &faults); + CHECK(faulty); + auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s); + CHECK(lease); + auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s); + CHECK(genesis); + auto state = evo::test::region_community(); + EnvBatch batch{*state}; + run_batch(batch, 8); + for (const std::uint32_t failures : {1U, 1U, 2U}) { + faults.fail_directory_syncs = failures; + auto attempt = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch, + filled<16>(0xA9), build, budget, 5s); + CHECK(!attempt && attempt.error().kind == PublicationError::upload && + attempt.error().object == ObjectError::io); + CHECK(load_region_head(*r.runtime, r.key, 5s)->sequence == 0); // Nothing sent. + } + faults.fail_directory_syncs = 0; + auto published = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch, + filled<16>(0xA9), build, budget, 5s); + CHECK(published && published->committed.head.sequence == 1); + CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s)); +} + +// Review P1: a committed death with no durable decision (what an accepted pre-v5 +// upgrade would leave, or a host that skipped derivation) is never served as a +// recovered region. The database cannot see payload deaths; recovery must. +void undecided_death_is_quarantined() { + auto r = provisioned("undecided", 10, 0xB6); + auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s); + CHECK(lease); + auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s); + CHECK(genesis); + auto state = evo::test::region_community(); + EnvBatch batch{*state}; + run_batch(batch, 64, true); // The consumer dies at tick 20. + CHECK(batch.end_tick() == 20 && state->progress().consumer_terminal == 1); + const auto previous = *decode_manifest(genesis->manifest); + auto payload = state->snapshot(); + auto trace = + encode_trace({1, r.key, previous.commit, genesis->head.digest, 0, 20}, batch.steps()); + CHECK(payload && trace && r.objects->put(*payload) && r.objects->put(*trace)); + auto manifest = successor_manifest(previous, genesis->head.digest, filled<16>(0xB7), *lease, 20, + build, payload->size(), *sha256(*payload), + {trace->size(), *sha256(*trace), 0, {}}); + CHECK(manifest); + auto bytes = encode_manifest(*manifest); + CHECK(bytes && publish_region_checkpoint(*r.runtime, *bytes, {}, 5s)); + const std::array retained{build}; + auto recovered = + recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s); + CHECK(!recovered && recovered.error().kind == RecoveryError::history); + CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s)); +} + } // namespace void run_region_checkpoint_store_tests() { sha256_vectors_and_database_agreement(); object_store_is_immutable_and_verified(); + directory_sync_failures_are_never_acknowledged(); publish_recover_and_continue(); failures_leave_the_previous_head(); lineage_publish_and_recover(); durable_outcomes_and_ambiguity(); + publication_waits_for_directory_durability(); + undecided_death_is_quarantined(); } diff --git a/tools/ci/postgres-check.sh b/tools/ci/postgres-check.sh index b099a18..de93066 100644 --- a/tools/ci/postgres-check.sh +++ b/tools/ci/postgres-check.sh @@ -66,6 +66,40 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \ PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh" test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 5:1,2,3,4,5 test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1 +# Review P1: a v4 database holding a published (possibly terminal) checkpoint has +# no durable decisions for it. The v5 upgrade must refuse before changing anything. +v4_sha="$(sha256sum "$root/migrations/0004_lineage_profile.sql")" +v4_sha="${v4_sha%% *}" +createdb v4_published +PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" \ + -v "migration_sha_2=$v2_sha" -v "migration_sha_3=$v3_sha" -v "migration_sha_4=$v4_sha" \ + -v runtime_role=evo_runtime_test \ + -f "$root/migrations/0001_region_bootstrap.sql" -f "$root/migrations/0002_region_fencing.sql" \ + -f "$root/migrations/0003_region_checkpoints.sql" -f "$root/migrations/0004_lineage_profile.sql" > /dev/null +PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 > /dev/null <<'SQL' +INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex')); +SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex')); +SELECT lease_holder, lease_epoch FROM evolution.acquire_region_lease(1, 1, 20000) \gset +-- A real v4 publication of ticks 0..20 through the v4 fenced function. +SELECT p.* FROM evolution.region_commits c, LATERAL evolution.publish_region_checkpoint( + overlay(overlay(overlay(overlay(overlay(overlay(overlay(c.manifest + placing decode(repeat('22', 16), 'hex') from 41) + placing decode('0100000000000000', 'hex') from 57) + placing decode(repeat('11', 16), 'hex') from 65) + placing c.commit_digest from 81) + placing decode('1400000000000000', 'hex') from 121) + placing decode(replace(:'lease_holder', '-', ''), 'hex') from 129) + placing decode('0100000000000000', 'hex') from 145)) p + WHERE c.commit_sequence = 0; +SQL +test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 2 +if PGDATABASE=v4_published bash "$root/tools/migrate-postgres.sh" 2> "$test_dir/v4_published.err"; then + echo 'Expected pre-v5 published checkpoints to block v5' >&2; exit 1 +fi +grep -q 'pre-v5 published checkpoints lack durable decisions' "$test_dir/v4_published.err" +test "$(PGDATABASE=v4_published psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4 +test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) || ':' || max(end_tick) FROM evolution.region_commits")" = 2:20 +test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'region_decisions'")" = 0 test "$(PGDATABASE=v2_heads psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2 # Inherited privileges bypass fencing, so every migrate run audits effective rights. psql -X -v ON_ERROR_STOP=1 -c 'CREATE ROLE evo_bad_runtime NOLOGIN IN ROLE pg_write_all_data'