Fix object reuse durability and refuse unsafe pre-v5 upgrades
Review P1 (object durability): a name installed by a put whose directory fsync failed was later reused as success without syncing again, so a publication could acknowledge an object that might vanish on power loss. Reuse (existing and lost-link-race paths) now fsyncs the verified file and the directory before succeeding, and namespace opens always sync their parent. A test-only ObjectStoreFaults hook injects directory sync failures; tests cover failed sync then retries, namespace creation syncs, and publication refusing to proceed until durability holds. Review P1 (pre-v5 history): v5 recovery requires one durable decision per terminal actor, but v3/v4 published checkpoints carry none, so an accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now runs before 0005 in every upgrade path and refuses (EVL10) when any published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is unchanged. CI publishes a real v4 checkpoint and proves the upgrade is refused with nothing changed; a committed death without a decision is shown to recover as quarantined history. Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2; reverting each fix is caught by its test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -54,8 +54,12 @@ computes commit digests, and the host checks that the receipt matches.
|
||||
are named by the hex SHA-256 of their content.
|
||||
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
|
||||
link never replaces an existing name), and then fsyncs the directory.
|
||||
- Existing identical content is reused. Differing content already under a key is
|
||||
reported as corrupt and never overwritten.
|
||||
- Existing identical content is reused only after its file and directory are
|
||||
fsynced again, whether it came from an earlier attempt whose directory sync failed
|
||||
or from a concurrent installer. A visible name alone is not durability (review
|
||||
fix). Namespace directories sync their parent on every open, not only on
|
||||
creation. Differing content already under a key is reported as corrupt and never
|
||||
overwritten.
|
||||
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
|
||||
and verify the digest.
|
||||
- There is no delete or GC operation: unreferenced uploads are orphans.
|
||||
|
||||
@@ -63,6 +63,13 @@ parameter; these profiles can produce at most 2. The decision identity is
|
||||
outbox rows only a first delivery timestamp (EVL09).
|
||||
- The runtime role's audited function set grows to ten.
|
||||
|
||||
**Upgrades** from v3 or v4 run `migrations/pre_v5_guard.sql` inside the same
|
||||
transaction, before 0005. The guard refuses (EVL10), changing nothing, if any
|
||||
published pre-v5 checkpoint exists (sequence > 0). Such a checkpoint may hold
|
||||
terminal organisms without decisions, and v5 recovery would quarantine it.
|
||||
Genesis-only regions pass, because tick 0 has no terminal organisms. A verified
|
||||
legacy-history conversion is future work; there is none now.
|
||||
|
||||
**Recovery** loads decisions and requires lifecycle agreement:
|
||||
- one death per terminal actor, none for an active actor;
|
||||
- no death after the head tick, and organisms match the profile;
|
||||
|
||||
@@ -2265,3 +2265,34 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
0.75-0.86 ms.
|
||||
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
|
||||
consumer, an injected failure between commit insert and statement end, and TSan.
|
||||
|
||||
### M7.5 review fixes - 2026-09-28
|
||||
|
||||
- P1 object durability:
|
||||
- `put()` now re-fsyncs the file and the directory before reusing an existing
|
||||
object, on both the found-existing and the lost link-race paths.
|
||||
- Namespace opens always sync the parent directory.
|
||||
- A test-only `ObjectStoreFaults` hook injects directory-sync failures and
|
||||
counts successful syncs.
|
||||
- Tests: a namespace-creation sync failure, then a reopen still failing, then
|
||||
success with exactly 2 syncs. An install sync failure leaves the object
|
||||
visible but unacknowledged, three retries while syncs fail all return io with
|
||||
no extra syncs, then success with +1 sync; every reuse adds a sync.
|
||||
- Publication with 1, 1 and 2 injected failures returns `upload`/io and sends
|
||||
no manifest (head stays 0), then publishes once syncs succeed.
|
||||
- Reverting the reuse sync or the always-sync namespace behavior makes these
|
||||
tests fail.
|
||||
- P1 pre-v5 history:
|
||||
- `pre_v5_guard.sql` (EVL10) runs before 0005 in every upgrade path. 0005 and
|
||||
its checksum are unchanged.
|
||||
- CI builds a real v4 database: v1 genesis, a lease, and a sequence-1
|
||||
publication through the v4 function. The v5 upgrade is refused with the
|
||||
expected message; history stays 1..4, both commits remain, and no decision
|
||||
table is created. Replacing the guard with a no-op makes that check fail.
|
||||
- A committed death published without any decision (the state an accepted
|
||||
legacy upgrade would produce) recovers as `history`, quarantined and never
|
||||
served.
|
||||
- Reran GCC14 Release (3 runs), clang-tidy analysis (1) and ASan/UBSan (2)
|
||||
PostgreSQL suites, all passing, and the repository format check. The changes
|
||||
are Linux-only persistence code, SQL and tests; the portable targets and
|
||||
Windows are unchanged.
|
||||
|
||||
@@ -120,7 +120,9 @@ Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
|
||||
a first delivery time. Owners can disable triggers, so treat that as an audited
|
||||
break-glass action. EVL08 means an outcome was already final. After an
|
||||
ambiguous publication, resolve the exact prepared bytes before doing anything
|
||||
else.
|
||||
else. Upgrading a v3/v4 database that has published checkpoints (not just
|
||||
genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions,
|
||||
or wait for an explicit legacy-history conversion.
|
||||
|
||||
There is no operator CLI for provisioning yet, and no real world has been
|
||||
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
|
||||
@@ -20,6 +20,12 @@ enum class ObjectError : std::uint8_t {
|
||||
bounds, // Larger than the configured maximum; rejected before allocation.
|
||||
io, // Write/sync/link failure; the object is absent or unverified.
|
||||
};
|
||||
// Test-only fault injection and accounting for directory fsyncs. When passed to
|
||||
// open() it must outlive the store; production callers pass nothing.
|
||||
struct ObjectStoreFaults {
|
||||
std::uint32_t fail_directory_syncs{}; // The next N directory syncs report failure.
|
||||
std::uint64_t directory_syncs{}; // Directory syncs that succeeded.
|
||||
};
|
||||
struct VerifiedObject {
|
||||
Sha256Digest sha256{};
|
||||
std::uint64_t bytes{};
|
||||
@@ -28,9 +34,11 @@ struct VerifiedObject {
|
||||
class FilesystemObjectStore {
|
||||
public:
|
||||
// trusted_root must be an existing real directory (not a symlink). Namespace
|
||||
// directories w<world>/r<region> are created 0700 and their parents synced.
|
||||
// directories w<world>/r<region> are created 0700; their parents are synced on
|
||||
// every open, so a creation whose sync failed earlier is made durable now.
|
||||
[[nodiscard]] static std::expected<std::unique_ptr<FilesystemObjectStore>, ObjectError>
|
||||
open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes);
|
||||
open(const std::filesystem::path& trusted_root, RegionKey key, std::size_t max_object_bytes,
|
||||
ObjectStoreFaults* faults = nullptr);
|
||||
FilesystemObjectStore(const FilesystemObjectStore&) = delete;
|
||||
FilesystemObjectStore& operator=(const FilesystemObjectStore&) = delete;
|
||||
FilesystemObjectStore(FilesystemObjectStore&&) = delete;
|
||||
@@ -38,7 +46,9 @@ class FilesystemObjectStore {
|
||||
~FilesystemObjectStore();
|
||||
// Writes a private temporary file, fsyncs it, links it to its key without
|
||||
// replacing anything, then fsyncs the directory. Identical bytes already stored
|
||||
// are verified and reused; a differing existing object is corrupt, never replaced.
|
||||
// (an earlier attempt whose directory sync failed, or a concurrent installer)
|
||||
// are verified and fsynced, file and directory, before success is reported. A
|
||||
// differing existing object is corrupt, never replaced.
|
||||
[[nodiscard]] std::expected<VerifiedObject, ObjectError>
|
||||
put(std::span<const std::uint8_t> bytes) const;
|
||||
// Reads only an exact-length regular file and verifies its SHA-256.
|
||||
@@ -49,9 +59,13 @@ class FilesystemObjectStore {
|
||||
}
|
||||
|
||||
private:
|
||||
FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum);
|
||||
FilesystemObjectStore(std::filesystem::path directory, int descriptor, std::size_t maximum,
|
||||
ObjectStoreFaults* faults);
|
||||
[[nodiscard]] std::expected<VerifiedObject, ObjectError>
|
||||
durable_existing(const VerifiedObject& object) const;
|
||||
std::filesystem::path directory_;
|
||||
int descriptor_;
|
||||
std::size_t maximum_;
|
||||
ObjectStoreFaults* faults_;
|
||||
};
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -27,18 +27,22 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
|| ',5:' || :'migration_sha_5' AS at_v5 \gset
|
||||
\if :at_v5
|
||||
\elif :at_v4
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v3
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v2
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v1
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\else
|
||||
\echo 'Unknown or changed migration history'
|
||||
@@ -49,6 +53,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
-- Runs inside apply.sql's transaction immediately before 0005 (ADR-0063). Not a
|
||||
-- migration and not recorded in history; it only refuses an unsafe upgrade.
|
||||
--
|
||||
-- Pre-v5 published checkpoints (sequence > 0) may hold organisms that died with no
|
||||
-- durable decision row, because decisions did not exist before v5. The database
|
||||
-- cannot inspect payloads to prove otherwise, and v5 recovery requires one
|
||||
-- decision per terminal actor, so accepting them would make those regions
|
||||
-- unrecoverable (RecoveryError::history). Genesis-only regions are safe: a tick-0
|
||||
-- state has no terminal organisms. Until an explicit, verified legacy-history
|
||||
-- conversion exists, refuse and change nothing.
|
||||
DO $$
|
||||
BEGIN
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_commits WHERE commit_sequence > 0) THEN
|
||||
RAISE EXCEPTION 'pre-v5 published checkpoints lack durable decisions; convert or re-provision before v5'
|
||||
USING ERRCODE = 'EVL10';
|
||||
END IF;
|
||||
END $$;
|
||||
@@ -48,15 +48,26 @@ std::string key_name(const Sha256Digest& d) {
|
||||
}
|
||||
return out;
|
||||
}
|
||||
// Opens or creates a private subdirectory without following symlinks.
|
||||
int subdirectory(int parent, const std::string& name) {
|
||||
bool created = false;
|
||||
if (::mkdirat(parent, name.c_str(), 0700) == 0)
|
||||
created = true;
|
||||
else if (errno != EEXIST)
|
||||
// Every directory fsync goes through here so tests can inject failures.
|
||||
bool sync_directory(int fd, ObjectStoreFaults* faults) {
|
||||
if (faults && faults->fail_directory_syncs > 0) {
|
||||
--faults->fail_directory_syncs;
|
||||
return false;
|
||||
}
|
||||
if (::fsync(fd) != 0)
|
||||
return false;
|
||||
if (faults)
|
||||
++faults->directory_syncs;
|
||||
return true;
|
||||
}
|
||||
// Opens or creates a private subdirectory without following symlinks. The parent
|
||||
// is synced even when the entry already existed: an earlier creation may have
|
||||
// failed its sync, and an unsynced entry must not be trusted as durable.
|
||||
int subdirectory(int parent, const std::string& name, ObjectStoreFaults* faults) {
|
||||
if (::mkdirat(parent, name.c_str(), 0700) != 0 && errno != EEXIST)
|
||||
return -1;
|
||||
const int fd = ::openat(parent, name.c_str(), O_RDONLY | O_DIRECTORY | O_NOFOLLOW | O_CLOEXEC);
|
||||
if (fd >= 0 && created && ::fsync(parent) != 0) {
|
||||
if (fd >= 0 && !sync_directory(parent, faults)) {
|
||||
static_cast<void>(::close(fd));
|
||||
return -1;
|
||||
}
|
||||
@@ -74,8 +85,9 @@ bool write_all(int fd, std::span<const std::uint8_t> bytes) {
|
||||
return true;
|
||||
}
|
||||
// Exact-length regular file whose content has the expected digest.
|
||||
std::expected<std::vector<std::uint8_t>, ObjectError> read_verified(int directory,
|
||||
const VerifiedObject& object) {
|
||||
// With sync_file, the verified file is also fsynced before it is trusted as durable.
|
||||
std::expected<std::vector<std::uint8_t>, ObjectError>
|
||||
read_verified(int directory, const VerifiedObject& object, bool sync_file = false) {
|
||||
const auto name = key_name(object.sha256);
|
||||
const Descriptor fd{::openat(directory, name.c_str(), O_RDONLY | O_NOFOLLOW | O_CLOEXEC)};
|
||||
if (fd.get() < 0)
|
||||
@@ -109,13 +121,16 @@ std::expected<std::vector<std::uint8_t>, ObjectError> read_verified(int director
|
||||
return std::unexpected{ObjectError::io};
|
||||
if (*digest != object.sha256)
|
||||
return std::unexpected{ObjectError::corrupt};
|
||||
if (sync_file && ::fsync(fd.get()) != 0)
|
||||
return std::unexpected{ObjectError::io};
|
||||
return bytes;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
FilesystemObjectStore::FilesystemObjectStore(std::filesystem::path directory, int descriptor,
|
||||
std::size_t maximum)
|
||||
: directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum) {}
|
||||
std::size_t maximum, ObjectStoreFaults* faults)
|
||||
: directory_(std::move(directory)), descriptor_(descriptor), maximum_(maximum),
|
||||
faults_(faults) {}
|
||||
|
||||
FilesystemObjectStore::~FilesystemObjectStore() {
|
||||
static_cast<void>(::close(descriptor_));
|
||||
@@ -123,7 +138,7 @@ FilesystemObjectStore::~FilesystemObjectStore() {
|
||||
|
||||
std::expected<std::unique_ptr<FilesystemObjectStore>, ObjectError>
|
||||
FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey key,
|
||||
std::size_t max_object_bytes) {
|
||||
std::size_t max_object_bytes, ObjectStoreFaults* faults) {
|
||||
if (key.world == 0 || key.region == 0 || max_object_bytes == 0 ||
|
||||
max_object_bytes > maximum_object_limit || trusted_root.empty())
|
||||
return std::unexpected{ObjectError::invalid};
|
||||
@@ -133,14 +148,14 @@ FilesystemObjectStore::open(const std::filesystem::path& trusted_root, RegionKey
|
||||
return std::unexpected{ObjectError::invalid};
|
||||
const auto world = "w" + std::to_string(key.world);
|
||||
const auto region = "r" + std::to_string(key.region);
|
||||
const Descriptor world_fd{subdirectory(root.get(), world)};
|
||||
const Descriptor world_fd{subdirectory(root.get(), world, faults)};
|
||||
if (world_fd.get() < 0)
|
||||
return std::unexpected{ObjectError::io};
|
||||
Descriptor region_fd{subdirectory(world_fd.get(), region)};
|
||||
Descriptor region_fd{subdirectory(world_fd.get(), region, faults)};
|
||||
if (region_fd.get() < 0)
|
||||
return std::unexpected{ObjectError::io};
|
||||
return std::unique_ptr<FilesystemObjectStore>(new FilesystemObjectStore(
|
||||
trusted_root / world / region, region_fd.release(), max_object_bytes));
|
||||
trusted_root / world / region, region_fd.release(), max_object_bytes, faults));
|
||||
}
|
||||
|
||||
std::expected<VerifiedObject, ObjectError>
|
||||
@@ -153,14 +168,8 @@ FilesystemObjectStore::put(std::span<const std::uint8_t> bytes) const {
|
||||
if (!digest)
|
||||
return std::unexpected{ObjectError::io};
|
||||
const VerifiedObject object{*digest, bytes.size()};
|
||||
const auto existing = [&]() -> std::expected<VerifiedObject, ObjectError> {
|
||||
auto stored = read_verified(descriptor_, object);
|
||||
if (!stored)
|
||||
return std::unexpected{stored.error()};
|
||||
return object;
|
||||
};
|
||||
// Immutable: identical content is reused; anything else under the key is corrupt.
|
||||
if (auto found = existing(); found || found.error() != ObjectError::missing)
|
||||
// Immutable: identical content is reused (durably); anything else is corrupt.
|
||||
if (auto found = durable_existing(object); found || found.error() != ObjectError::missing)
|
||||
return found;
|
||||
|
||||
const auto name = key_name(*digest);
|
||||
@@ -180,11 +189,23 @@ FilesystemObjectStore::put(std::span<const std::uint8_t> bytes) const {
|
||||
const bool raced = errno == EEXIST;
|
||||
discard();
|
||||
if (raced)
|
||||
return existing();
|
||||
return durable_existing(object);
|
||||
return std::unexpected{ObjectError::io};
|
||||
}
|
||||
discard();
|
||||
if (::fsync(descriptor_) != 0)
|
||||
if (!sync_directory(descriptor_, faults_))
|
||||
return std::unexpected{ObjectError::io};
|
||||
return object;
|
||||
}
|
||||
|
||||
// A visible name is not proof of durability: its installer's directory sync may
|
||||
// have failed. Re-sync the verified file and the directory before reusing it.
|
||||
std::expected<VerifiedObject, ObjectError>
|
||||
FilesystemObjectStore::durable_existing(const VerifiedObject& object) const {
|
||||
auto stored = read_verified(descriptor_, object, true);
|
||||
if (!stored)
|
||||
return std::unexpected{stored.error()};
|
||||
if (!sync_directory(descriptor_, faults_))
|
||||
return std::unexpected{ObjectError::io};
|
||||
return object;
|
||||
}
|
||||
|
||||
@@ -143,6 +143,35 @@ void object_store_is_immutable_and_verified() {
|
||||
}
|
||||
}
|
||||
|
||||
// Review P1: a name made visible by an install whose directory sync failed is not
|
||||
// durable. Neither a retry that finds it nor a namespace reopen may report success
|
||||
// until a directory sync actually succeeds.
|
||||
void directory_sync_failures_are_never_acknowledged() {
|
||||
const auto root = scratch("syncs");
|
||||
ObjectStoreFaults faults;
|
||||
faults.fail_directory_syncs = 1; // Creating w1 under the root.
|
||||
CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io);
|
||||
CHECK(fs::exists(root / "w1")); // Created, but its entry is unconfirmed.
|
||||
faults.fail_directory_syncs = 1; // Reopen finds it and must sync again.
|
||||
CHECK(FilesystemObjectStore::open(root, {1, 5}, 1024, &faults).error() == ObjectError::io);
|
||||
auto store = FilesystemObjectStore::open(root, {1, 5}, 1024, &faults);
|
||||
CHECK(store && faults.directory_syncs == 2); // root->w1 and w1->r5.
|
||||
|
||||
const auto payload = bytes_of("durable only after a directory sync succeeds");
|
||||
const VerifiedObject object{*sha256(payload), payload.size()};
|
||||
faults.fail_directory_syncs = 1;
|
||||
CHECK((*store)->put(payload).error() == ObjectError::io);
|
||||
CHECK((*store)->get(object)); // Visible, yet never acknowledged.
|
||||
faults.fail_directory_syncs = 3; // Retries while the directory still cannot sync.
|
||||
for (int attempt = 0; attempt < 3; ++attempt)
|
||||
CHECK((*store)->put(payload).error() == ObjectError::io);
|
||||
CHECK(faults.directory_syncs == 2);
|
||||
auto retried = (*store)->put(payload);
|
||||
CHECK(retried && *retried == object && faults.directory_syncs == 3);
|
||||
// Every reuse re-establishes durability rather than trusting the visible name.
|
||||
CHECK((*store)->put(payload) && faults.directory_syncs == 4);
|
||||
}
|
||||
|
||||
struct Region {
|
||||
RegionKey key{7, 3};
|
||||
std::unique_ptr<Postgres> owner, runtime;
|
||||
@@ -675,13 +704,76 @@ void durable_outcomes_and_ambiguity() {
|
||||
CHECK(!undo && std::string(undo.error().sqlstate.data()) == "EVL09"); // Stays quarantined.
|
||||
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
|
||||
}
|
||||
// Publication cannot acknowledge a checkpoint whose objects are not yet durable.
|
||||
void publication_waits_for_directory_durability() {
|
||||
auto r = provisioned("sync-publish", 9, 0xA8);
|
||||
ObjectStoreFaults faults;
|
||||
auto faulty = FilesystemObjectStore::open(r.objects->directory().parent_path().parent_path(),
|
||||
r.key, max_environmental_payload_bytes, &faults);
|
||||
CHECK(faulty);
|
||||
auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s);
|
||||
CHECK(lease);
|
||||
auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s);
|
||||
CHECK(genesis);
|
||||
auto state = evo::test::region_community();
|
||||
EnvBatch batch{*state};
|
||||
run_batch(batch, 8);
|
||||
for (const std::uint32_t failures : {1U, 1U, 2U}) {
|
||||
faults.fail_directory_syncs = failures;
|
||||
auto attempt = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch,
|
||||
filled<16>(0xA9), build, budget, 5s);
|
||||
CHECK(!attempt && attempt.error().kind == PublicationError::upload &&
|
||||
attempt.error().object == ObjectError::io);
|
||||
CHECK(load_region_head(*r.runtime, r.key, 5s)->sequence == 0); // Nothing sent.
|
||||
}
|
||||
faults.fail_directory_syncs = 0;
|
||||
auto published = publish_environmental_batch(*r.runtime, **faulty, *lease, *genesis, batch,
|
||||
filled<16>(0xA9), build, budget, 5s);
|
||||
CHECK(published && published->committed.head.sequence == 1);
|
||||
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
|
||||
}
|
||||
|
||||
// Review P1: a committed death with no durable decision (what an accepted pre-v5
|
||||
// upgrade would leave, or a host that skipped derivation) is never served as a
|
||||
// recovered region. The database cannot see payload deaths; recovery must.
|
||||
void undecided_death_is_quarantined() {
|
||||
auto r = provisioned("undecided", 10, 0xB6);
|
||||
auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s);
|
||||
CHECK(lease);
|
||||
auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s);
|
||||
CHECK(genesis);
|
||||
auto state = evo::test::region_community();
|
||||
EnvBatch batch{*state};
|
||||
run_batch(batch, 64, true); // The consumer dies at tick 20.
|
||||
CHECK(batch.end_tick() == 20 && state->progress().consumer_terminal == 1);
|
||||
const auto previous = *decode_manifest(genesis->manifest);
|
||||
auto payload = state->snapshot();
|
||||
auto trace =
|
||||
encode_trace({1, r.key, previous.commit, genesis->head.digest, 0, 20}, batch.steps());
|
||||
CHECK(payload && trace && r.objects->put(*payload) && r.objects->put(*trace));
|
||||
auto manifest = successor_manifest(previous, genesis->head.digest, filled<16>(0xB7), *lease, 20,
|
||||
build, payload->size(), *sha256(*payload),
|
||||
{trace->size(), *sha256(*trace), 0, {}});
|
||||
CHECK(manifest);
|
||||
auto bytes = encode_manifest(*manifest);
|
||||
CHECK(bytes && publish_region_checkpoint(*r.runtime, *bytes, {}, 5s));
|
||||
const std::array<Sha256Digest, 1> retained{build};
|
||||
auto recovered =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(!recovered && recovered.error().kind == RecoveryError::history);
|
||||
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void run_region_checkpoint_store_tests() {
|
||||
sha256_vectors_and_database_agreement();
|
||||
object_store_is_immutable_and_verified();
|
||||
directory_sync_failures_are_never_acknowledged();
|
||||
publish_recover_and_continue();
|
||||
failures_leave_the_previous_head();
|
||||
lineage_publish_and_recover();
|
||||
durable_outcomes_and_ambiguity();
|
||||
publication_waits_for_directory_durability();
|
||||
undecided_death_is_quarantined();
|
||||
}
|
||||
|
||||
@@ -66,6 +66,40 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 5:1,2,3,4,5
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
|
||||
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
|
||||
v4_sha="$(sha256sum "$root/migrations/0004_lineage_profile.sql")"
|
||||
v4_sha="${v4_sha%% *}"
|
||||
createdb v4_published
|
||||
PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" \
|
||||
-v "migration_sha_2=$v2_sha" -v "migration_sha_3=$v3_sha" -v "migration_sha_4=$v4_sha" \
|
||||
-v runtime_role=evo_runtime_test \
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" -f "$root/migrations/0002_region_fencing.sql" \
|
||||
-f "$root/migrations/0003_region_checkpoints.sql" -f "$root/migrations/0004_lineage_profile.sql" > /dev/null
|
||||
PGDATABASE=v4_published psql -X -v ON_ERROR_STOP=1 > /dev/null <<'SQL'
|
||||
INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'));
|
||||
SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'));
|
||||
SELECT lease_holder, lease_epoch FROM evolution.acquire_region_lease(1, 1, 20000) \gset
|
||||
-- A real v4 publication of ticks 0..20 through the v4 fenced function.
|
||||
SELECT p.* FROM evolution.region_commits c, LATERAL evolution.publish_region_checkpoint(
|
||||
overlay(overlay(overlay(overlay(overlay(overlay(overlay(c.manifest
|
||||
placing decode(repeat('22', 16), 'hex') from 41)
|
||||
placing decode('0100000000000000', 'hex') from 57)
|
||||
placing decode(repeat('11', 16), 'hex') from 65)
|
||||
placing c.commit_digest from 81)
|
||||
placing decode('1400000000000000', 'hex') from 121)
|
||||
placing decode(replace(:'lease_holder', '-', ''), 'hex') from 129)
|
||||
placing decode('0100000000000000', 'hex') from 145)) p
|
||||
WHERE c.commit_sequence = 0;
|
||||
SQL
|
||||
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 2
|
||||
if PGDATABASE=v4_published bash "$root/tools/migrate-postgres.sh" 2> "$test_dir/v4_published.err"; then
|
||||
echo 'Expected pre-v5 published checkpoints to block v5' >&2; exit 1
|
||||
fi
|
||||
grep -q 'pre-v5 published checkpoints lack durable decisions' "$test_dir/v4_published.err"
|
||||
test "$(PGDATABASE=v4_published psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4
|
||||
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) || ':' || max(end_tick) FROM evolution.region_commits")" = 2:20
|
||||
test "$(PGDATABASE=v4_published psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'region_decisions'")" = 0
|
||||
test "$(PGDATABASE=v2_heads psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2
|
||||
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'CREATE ROLE evo_bad_runtime NOLOGIN IN ROLE pg_write_all_data'
|
||||
|
||||
Reference in New Issue
Block a user