Fix object reuse durability and refuse unsafe pre-v5 upgrades

Review P1 (object durability): a name installed by a put whose directory
fsync failed was later reused as success without syncing again, so a
publication could acknowledge an object that might vanish on power
loss. Reuse (existing and lost-link-race paths) now fsyncs the verified
file and the directory before succeeding, and namespace opens always
sync their parent. A test-only ObjectStoreFaults hook injects directory
sync failures; tests cover failed sync then retries, namespace creation
syncs, and publication refusing to proceed until durability holds.

Review P1 (pre-v5 history): v5 recovery requires one durable decision
per terminal actor, but v3/v4 published checkpoints carry none, so an
accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now
runs before 0005 in every upgrade path and refuses (EVL10) when any
published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is
unchanged. CI publishes a real v4 checkpoint and proves the upgrade is
refused with nothing changed; a committed death without a decision is
shown to recover as quarantined history.

Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2;
reverting each fix is caught by its test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 19:20:35 -04:00
co-authored by Claude Opus 5.5
parent dd7e8028b0
commit ab4bf5355f
10 changed files with 259 additions and 32 deletions
+6 -2
View File
@@ -54,8 +54,12 @@ computes commit digests, and the host checks that the receipt matches.
are named by the hex SHA-256 of their content.
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
link never replaces an existing name), and then fsyncs the directory.
- Existing identical content is reused. Differing content already under a key is
reported as corrupt and never overwritten.
- Existing identical content is reused only after its file and directory are
fsynced again, whether it came from an earlier attempt whose directory sync failed
or from a concurrent installer. A visible name alone is not durability (review
fix). Namespace directories sync their parent on every open, not only on
creation. Differing content already under a key is reported as corrupt and never
overwritten.
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
and verify the digest.
- There is no delete or GC operation: unreferenced uploads are orphans.
+7
View File
@@ -63,6 +63,13 @@ parameter; these profiles can produce at most 2. The decision identity is
outbox rows only a first delivery timestamp (EVL09).
- The runtime role's audited function set grows to ten.
**Upgrades** from v3 or v4 run `migrations/pre_v5_guard.sql` inside the same
transaction, before 0005. The guard refuses (EVL10), changing nothing, if any
published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold
terminal organisms without decisions, and v5 recovery would quarantine it.
Genesis-only regions pass, because tick 0 has no terminal organisms. A verified
legacy-history conversion is future work; there is none now.
**Recovery** loads decisions and requires lifecycle agreement:
- one death per terminal actor, none for an active actor;
- no death after the head tick, and organisms match the profile;
+31
View File
@@ -2265,3 +2265,34 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
0.75-0.86 ms.
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
consumer, an injected failure between commit insert and statement end, and TSan.
### M7.5 review fixes - 2026-09-28
- P1 object durability:
- `put()` now re-fsyncs the file and the directory before reusing an existing
object, on both the found-existing and the lost link-race paths.
- Namespace opens always sync the parent directory.
- A test-only `ObjectStoreFaults` hook injects directory-sync failures and
counts successful syncs.
- Tests: a namespace-creation sync failure, then a reopen still failing, then
success with exactly 2 syncs. An install sync failure leaves the object
visible but unacknowledged, three retries while syncs fail all return io with
no extra syncs, then success with +1 sync; every reuse adds a sync.
- Publication with 1, 1 and 2 injected failures returns `upload`/io and sends
no manifest (head stays 0), then publishes once syncs succeed.
- Reverting the reuse sync or the always-sync namespace behavior makes these
tests fail.
- P1 pre-v5 history:
- `pre_v5_guard.sql` (EVL10) runs before 0005 in every upgrade path. 0005 and
its checksum are unchanged.
- CI builds a real v4 database: v1 genesis, a lease, and a sequence-1
publication through the v4 function. The v5 upgrade is refused with the
expected message; history stays 1..4, both commits remain, and no decision
table is created. Replacing the guard with a no-op makes that check fail.
- A committed death published without any decision (the state an accepted
legacy upgrade would produce) recovers as `history`, quarantined and never
served.
- Reran GCC14 Release (3 runs), clang-tidy analysis (1) and ASan/UBSan (2)
PostgreSQL suites, all passing, and the repository format check. The changes
are Linux-only persistence code, SQL and tests; the portable targets and
Windows are unchanged.
+3 -1
View File
@@ -120,7 +120,9 @@ Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
a first delivery time. Owners can disable triggers, so treat that as an audited
break-glass action. EVL08 means an outcome was already final. After an
ambiguous publication, resolve the exact prepared bytes before doing anything
else.
else. Upgrading a v3/v4 database that has published checkpoints (not just
genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions,
or wait for an explicit legacy-history conversion.
There is no operator CLI for provisioning yet, and no real world has been
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
+18 -4
View File
@@ -20,6 +20,12 @@ enum class ObjectError : std::uint8_t {
bounds, // Larger than the configured maximum; rejected before allocation.
io, // Write/sync/link failure; the object is absent or unverified.
};
// Test-only fault injection and accounting for directory fsyncs. When passed to
// open() it must outlive the store; production callers pass nothing.
struct ObjectStoreFaults {
std::uint32_t fail_directory_syncs{}; // The next N directory syncs report failure.
std::uint64_t directory_syncs{}; // Directory syncs that succeeded.
};
struct VerifiedObject {
Sha256Digest sha256{};
std::uint64_t bytes{};
@@ -28,9 +34,11 @@ struct VerifiedObject {
class FilesystemObjectStore {
public:
// trusted_root must be an existing real directory (not a symlink). Namespace
// directories w<world>/r<region> are created 0700 and their parents synced.
// directories w<world>/r<region> are created 0700; their parents are synced on
// every open, so a creation whose sync failed earlier is made durable now.
[[nodiscard]] static std::expected<std::unique_ptr<FilesystemObjectStore>, ObjectError>
open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes);
open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes,
ObjectStoreFaults* faults = nullptr);
FilesystemObjectStore(const FilesystemObjectStore&) = delete;
FilesystemObjectStore& operator=(const FilesystemObjectStore&) = delete;
FilesystemObjectStore(FilesystemObjectStore&&) = delete;
@@ -38,7 +46,9 @@ class FilesystemObjectStore {
~FilesystemObjectStore();
// Writes a private temporary file, fsyncs it, links it to its key without
// replacing anything, then fsyncs the directory. Identical bytes already stored
// are verified and reused; a differing existing object is corrupt, never replaced.
// (an earlier attempt whose directory sync failed, or a concurrent installer)
// are verified and fsynced, file and directory, before success is reported. A
// differing existing object is corrupt, never replaced.
[[nodiscard]] std::expected<VerifiedObject, ObjectError>
put(std::span<const std::uint8_t> bytes) const;
// Reads only an exact-length regular file and verifies its SHA-256.
@@ -49,9 +59,13 @@ class FilesystemObjectStore {
}
private:
FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum);
FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum,
ObjectStoreFaults* faults);
[[nodiscard]] std::expected<VerifiedObject, ObjectError>
durable_existing(const VerifiedObject& object) const;
std::filesystem::path directory_;
int descriptor_;
std::size_t maximum_;
ObjectStoreFaults* faults_;
};
} // namespace evo::persistence
+5
View File
@@ -27,18 +27,22 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|| ',5:' || :'migration_sha_5' AS at_v5 \gset
\if :at_v5
\elif :at_v4
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\elif :at_v3
\ir 0004_lineage_profile.sql
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\elif :at_v2
\ir 0003_region_checkpoints.sql
\ir 0004_lineage_profile.sql
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\elif :at_v1
\ir 0002_region_fencing.sql
\ir 0003_region_checkpoints.sql
\ir 0004_lineage_profile.sql
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\else
\echo 'Unknown or changed migration history'
@@ -49,6 +53,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
\ir 0002_region_fencing.sql
\ir 0003_region_checkpoints.sql
\ir 0004_lineage_profile.sql
\ir pre_v5_guard.sql
\ir 0005_batch_decisions.sql
\endif
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
+17
View File
@@ -0,0 +1,17 @@
-- Runs inside apply.sql's transaction immediately before 0005 (ADR-0063). Not a
-- migration and not recorded in history; it only refuses an unsafe upgrade.
--
-- Pre-v5 published checkpoints (sequence > 0) may hold organisms that died with no
-- durable decision row, because decisions did not exist before v5. The database
-- cannot inspect payloads to prove otherwise, and v5 recovery requires one
-- decision per terminal actor, so accepting them would make those regions
-- unrecoverable (RecoveryError::history). Genesis-only regions are safe: a tick-0
-- state has no terminal organisms. Until an explicit, verified legacy-history
-- conversion exists, refuse and change nothing.
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM evolution.region_commits WHERE commit_sequence > 0) THEN
RAISE EXCEPTION 'pre-v5 published checkpoints lack durable decisions; convert or re-provision before v5'
USING ERRCODE = 'EVL10';
END IF;
END $$;
+46 -25
View File
@@ -48,15 +48,26 @@ std::string key_name(const Sha256Digest& d) {
}
return out;
}
// Opens or creates a private subdirectory without following symlinks.
int subdirectory(int parent, const std::string& name) {
bool created = false;
if (::mkdirat(parent, name.c_str(), 0700) == 0)
created = true;
else if (errno != EEXIST)
// Every directory fsync goes through here so tests can inject failures.
bool sync_directory(int fd, ObjectStoreFaults* faults) {
if (faults && faults->fail_directory_syncs > 0) {
--faults->fail_directory_syncs;
return false;
}
if (::fsync(fd) != 0)
return false;
if (faults)
++faults->directory_syncs;
return true;
}
// Opens or creates a private subdirectory without following symlinks. The parent
// is synced even when the entry already existed: an earlier creation may have
// failed its sync, and an unsynced entry must not be trusted as durable.
int subdirectory(int parent, const std::string& name, ObjectStoreFaults* faults) {
if (::mkdirat(parent, name.c_str(), 0700) != 0 && errno != EEXIST)
return -1;
const int fd = ::openat(parent, name.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
if (fd >= 0 && created && ::fsync(parent) != 0) {
if (fd >= 0 && !sync_directory(parent, faults)) {
static_cast<void>(::close(fd));
return -1;
}
@@ -74,8 +85,9 @@ bool write_all(int fd, std::span<const std::uint8_t> bytes) {
return true;
}
// Exact-length regular file whose content has the expected digest.
std::expected<std::vector<std::uint8_t>, ObjectError> read_verified(int directory,
const VerifiedObject& object) {
// With sync_file, the verified file is also fsynced before it is trusted as durable.
std::expected<std::vector<std::uint8_t>, ObjectError>
read_verified(int directory, const VerifiedObject& object, bool sync_file = false) {
const auto name = key_name(object.sha256);
const Descriptor fd{::openat(directory, name.c_str(), O_RDONLY | O_NOFOLLOW | O_CLOEXEC)};
if (fd.get() < 0)
@@ -109,13 +121,16 @@ std::expected<std::vector<std::uint8_t>, ObjectError> read_verified(int director
return std::unexpected{ObjectError::io};
if (*digest != object.sha256)
return std::unexpected{ObjectError::corrupt};
if (sync_file && ::fsync(fd.get()) != 0)
return std::unexpected{ObjectError::io};
return bytes;
}
} // namespace
FilesystemObjectStore::FilesystemObjectStore(std::filesystem::path directory, int descriptor,
std::size_t maximum)
: directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum) {}
std::size_t maximum, ObjectStoreFaults* faults)
: directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum),
faults_(faults) {}
FilesystemObjectStore::~FilesystemObjectStore() {
static_cast<void>(::close(descriptor_));
@@ -123,7 +138,7 @@ FilesystemObjectStore::~FilesystemObjectStore() {
std::expected<std::unique_ptr<FilesystemObjectStore>, ObjectError>
FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey key,
std::size_t max_object_bytes) {
std::size_t max_object_bytes, ObjectStoreFaults* faults) {
if (key.world == 0 || key.region == 0 || max_object_bytes == 0 ||
max_object_bytes > maximum_object_limit || trusted_root.empty())
return std::unexpected{ObjectError::invalid};
@@ -133,14 +148,14 @@ FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey
return std::unexpected{ObjectError::invalid};
const auto world = "w" + std::to_string(key.world);
const auto region = "r" + std::to_string(key.region);
const Descriptor world_fd{subdirectory(root.get(), world)};
const Descriptor world_fd{subdirectory(root.get(), world, faults)};
if (world_fd.get() < 0)
return std::unexpected{ObjectError::io};
Descriptor region_fd{subdirectory(world_fd.get(), region)};
Descriptor region_fd{subdirectory(world_fd.get(), region, faults)};
if (region_fd.get() < 0)
return std::unexpected{ObjectError::io};
return std::unique_ptr<FilesystemObjectStore>(new FilesystemObjectStore(
trusted_root / world / region, region_fd.release(), max_object_bytes));
trusted_root / world / region, region_fd.release(), max_object_bytes, faults));
}
std::expected<VerifiedObject, ObjectError>
@@ -153,14 +168,8 @@ FilesystemObjectStore::put(std::span<const std::uint8_t> bytes) const {
if (!digest)
return std::unexpected{ObjectError::io};
const VerifiedObject object{*digest, bytes.size()};
const auto existing = [&]() -> std::expected<VerifiedObject, ObjectError> {
auto stored = read_verified(descriptor_, object);
if (!stored)
return std::unexpected{stored.error()};
return object;
};
// Immutable: identical content is reused; anything else under the key is corrupt.
if (auto found = existing(); found || found.error() != ObjectError::missing)
// Immutable: identical content is reused (durably); anything else is corrupt.
if (auto found = durable_existing(object); found || found.error() != ObjectError::missing)
return found;
const auto name = key_name(*digest);
@@ -180,11 +189,23 @@ FilesystemObjectStore::put(std::span<const std::uint8_t> bytes) const {
const bool raced = errno == EEXIST;
discard();
if (raced)
return existing();
return durable_existing(object);
return std::unexpected{ObjectError::io};
}
discard();
if (::fsync(descriptor_) != 0)
if (!sync_directory(descriptor_, faults_))
return std::unexpected{ObjectError::io};
return object;
}
// A visible name is not proof of durability: its installer's directory sync may
// have failed. Re-sync the verified file and the directory before reusing it.
std::expected<VerifiedObject, ObjectError>
FilesystemObjectStore::durable_existing(const VerifiedObject& object) const {
auto stored = read_verified(descriptor_, object, true);
if (!stored)
return std::unexpected{stored.error()};
if (!sync_directory(descriptor_, faults_))
return std::unexpected{ObjectError::io};
return object;
}
+92
View File
@@ -143,6 +143,35 @@ void object_store_is_immutable_and_verified() {
}
}
// Review P1: a name made visible by an install whose directory sync failed is not
// durable. Neither a retry that finds it nor a namespace reopen may report success
// until a directory sync actually succeeds.
void directory_sync_failures_are_never_acknowledged() {
const auto root = scratch("syncs");
ObjectStoreFaults faults;
faults.fail_directory_syncs = 1; // Creating w1 under the root.
CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io);
CHECK(fs::exists(root / "w1")); // Created, but its entry is unconfirmed.
faults.fail_directory_syncs = 1; // Reopen finds it and must sync again.
CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io);
auto store = FilesystemObjectStore::open(root, {1, 5}, 1024, &faults);
CHECK(store && faults.directory_syncs == 2); // root->w1 and w1->r5.
const auto payload = bytes_of("durable only after a directory sync succeeds");
const VerifiedObject object{*sha256(payload), payload.size()};
faults.fail_directory_syncs = 1;
CHECK((*store)->put(payload).error() == ObjectError::io);
CHECK((*store)->get(object)); // Visible, yet never acknowledged.
faults.fail_directory_syncs = 3; // Retries while the directory still cannot sync.
for (int attempt = 0; attempt < 3; ++attempt)
CHECK((*store)->put(payload).error() == ObjectError::io);
CHECK(faults.directory_syncs == 2);
auto retried = (*store)->put(payload);
CHECK(retried && *retried == object && faults.directory_syncs == 3);
// Every reuse re-establishes durability rather than trusting the visible name.
CHECK((*store)->put(payload) && faults.directory_syncs == 4);
}
struct Region {
RegionKey key{7, 3};
std::unique_ptr<Postgres> owner, runtime;
@@ -675,13 +704,76 @@ void durable_outcomes_and_ambiguity() {
CHECK(!undo && std::string(undo.error().sqlstate.data()) == "EVL09"); // Stays quarantined.
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
}
// Publication cannot acknowledge a checkpoint whose objects are not yet durable.
void publication_waits_for_directory_durability() {
auto r = provisioned("sync-publish", 9, 0xA8);
ObjectStoreFaults faults;
auto faulty = FilesystemObjectStore::open(r.objects->directory().parent_path().parent_path(),
r.key, max_environmental_payload_bytes, &faults);
CHECK(faulty);
auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s);
CHECK(lease);
auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s);
CHECK(genesis);
auto state = evo::test::region_community();
EnvBatch batch{*state};
run_batch(batch, 8);
for (const std::uint32_t failures : {1U, 1U, 2U}) {
faults.fail_directory_syncs = failures;
auto attempt = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch,
filled<16>(0xA9), build, budget, 5s);
CHECK(!attempt && attempt.error().kind == PublicationError::upload &&
attempt.error().object == ObjectError::io);
CHECK(load_region_head(*r.runtime, r.key, 5s)->sequence == 0); // Nothing sent.
}
faults.fail_directory_syncs = 0;
auto published = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch,
filled<16>(0xA9), build, budget, 5s);
CHECK(published && published->committed.head.sequence == 1);
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
}
// Review P1: a committed death with no durable decision (what an accepted pre-v5
// upgrade would leave, or a host that skipped derivation) is never served as a
// recovered region. The database cannot see payload deaths; recovery must.
void undecided_death_is_quarantined() {
auto r = provisioned("undecided", 10, 0xB6);
auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s);
CHECK(lease);
auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s);
CHECK(genesis);
auto state = evo::test::region_community();
EnvBatch batch{*state};
run_batch(batch, 64, true); // The consumer dies at tick 20.
CHECK(batch.end_tick() == 20 && state->progress().consumer_terminal == 1);
const auto previous = *decode_manifest(genesis->manifest);
auto payload = state->snapshot();
auto trace =
encode_trace({1, r.key, previous.commit, genesis->head.digest, 0, 20}, batch.steps());
CHECK(payload && trace && r.objects->put(*payload) && r.objects->put(*trace));
auto manifest = successor_manifest(previous, genesis->head.digest, filled<16>(0xB7), *lease, 20,
build, payload->size(), *sha256(*payload),
{trace->size(), *sha256(*trace), 0, {}});
CHECK(manifest);
auto bytes = encode_manifest(*manifest);
CHECK(bytes && publish_region_checkpoint(*r.runtime, *bytes, {}, 5s));
const std::array<Sha256Digest, 1> retained{build};
auto recovered =
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
CHECK(!recovered && recovered.error().kind == RecoveryError::history);
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
}
} // namespace
void run_region_checkpoint_store_tests() {
sha256_vectors_and_database_agreement();
object_store_is_immutable_and_verified();
directory_sync_failures_are_never_acknowledged();
publish_recover_and_continue();
failures_leave_the_previous_head();
lineage_publish_and_recover();
durable_outcomes_and_ambiguity();
publication_waits_for_directory_durability();
undecided_death_is_quarantined();
}
+34
View File
@@ -66,6 +66,40 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 5:1,2,3,4,5
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
v4_sha="$(sha256sum "$root/migrations/0004_lineage_profile.sql")"
v4_sha="${v4_sha%% *}"
createdb v4_published
PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" \
-v "migration_sha_2=$v2_sha" -v "migration_sha_3=$v3_sha" -v "migration_sha_4=$v4_sha" \
-v runtime_role=evo_runtime_test \
-f "$root/migrations/0001_region_bootstrap.sql" -f "$root/migrations/0002_region_fencing.sql" \
-f "$root/migrations/0003_region_checkpoints.sql" -f "$root/migrations/0004_lineage_profile.sql" > /dev/null
PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 > /dev/null <<'SQL'
INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'));
SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'));
SELECT lease_holder, lease_epoch FROM evolution.acquire_region_lease(1, 1, 20000) \gset
-- A real v4 publication of ticks 0..20 through the v4 fenced function.
SELECT p.* FROM evolution.region_commits c, LATERAL evolution.publish_region_checkpoint(
overlay(overlay(overlay(overlay(overlay(overlay(overlay(c.manifest
placing decode(repeat('22', 16), 'hex') from 41)
placing decode('0100000000000000', 'hex') from 57)
placing decode(repeat('11', 16), 'hex') from 65)
placing c.commit_digest from 81)
placing decode('1400000000000000', 'hex') from 121)
placing decode(replace(:'lease_holder', '-', ''), 'hex') from 129)
placing decode('0100000000000000', 'hex') from 145)) p
WHERE c.commit_sequence = 0;
SQL
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 2
if PGDATABASE=v4_published bash "$root/tools/migrate-postgres.sh" 2> "$test_dir/v4_published.err"; then
echo 'Expected pre-v5 published checkpoints to block v5' >&2; exit 1
fi
grep -q 'pre-v5 published checkpoints lack durable decisions' "$test_dir/v4_published.err"
test "$(PGDATABASE=v4_published psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) || ':' || max(end_tick) FROM evolution.region_commits")" = 2:20
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'region_decisions'")" = 0
test "$(PGDATABASE=v2_heads psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
psql -X -v ON_ERROR_STOP=1 -c 'CREATE ROLE evo_bad_runtime NOLOGIN IN ROLE pg_write_all_data'