Add primitive object publication and recovery adapter (M11.2c-2d)
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (client) (push) Successful in 12m22s
Foundation / checks (network) (push) Successful in 13m52s
Foundation / checks (tsan) (push) Successful in 8m7s
Foundation / checks (checks) (push) Successful in 33m54s
Foundation / checks (postgres) (push) Successful in 47m38s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m39s
Foundation / server-image (push) Successful in 4m16s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m53s
Foundation / checks (client) (push) Successful in 12m22s
Foundation / checks (network) (push) Successful in 13m52s
Foundation / checks (tsan) (push) Successful in 8m7s
Foundation / checks (checks) (push) Successful in 33m54s
Foundation / checks (postgres) (push) Successful in 47m38s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m39s
Foundation / server-image (push) Successful in 4m16s
This commit is contained in:
+2
-1
@@ -224,7 +224,8 @@ if(EVO_BUILD_POSTGRES)
|
||||
# SHA-256 via system OpenSSL libcrypto (already required by libpq TLS; ADR-0061).
|
||||
find_package(OpenSSL 3 REQUIRED COMPONENTS Crypto)
|
||||
add_library(evo_region_store STATIC src/persistence/sha256.cpp src/persistence/object_store.cpp
|
||||
src/persistence/checkpoint_store.cpp src/persistence/region_clone.cpp)
|
||||
src/persistence/checkpoint_store.cpp src/persistence/primitive_checkpoint_store.cpp
|
||||
src/persistence/region_clone.cpp)
|
||||
add_library(evo::region_store ALIAS evo_region_store)
|
||||
target_link_libraries(evo_region_store PUBLIC evo::postgres evo::region_checkpoint
|
||||
PRIVATE OpenSSL::Crypto)
|
||||
|
||||
@@ -211,6 +211,18 @@ admission and retry behavior, not end-to-end recoverability or deployable gamepl
|
||||
Death, respawn, offline timers, biological tick integration and life transfer are
|
||||
outside 11.2 and remain governed by later gates.
|
||||
|
||||
## 11.2c-2d: Object-store publication and recovery adapter
|
||||
|
||||
The C++ persistence adapter writes the canonical primitive payload and batch trace
|
||||
to the immutable object store, verifies each read-back, and only then calls the
|
||||
schema v16 publisher. Before upload it restores the verified predecessor payload,
|
||||
replays the batch and requires byte-identical candidate state. Unknown outcomes
|
||||
retain the exact manifest and trace for an exact retry. Recovery verifies the
|
||||
committed head, retained build/profile, payload closure, head trace replay and
|
||||
reconciliation between restored lives and immutable SQL bindings; it fails closed
|
||||
on absent/corrupt evidence and never substitutes an older head. This adapter does
|
||||
not add a serving host, request protocol or session routing for primitive lives.
|
||||
|
||||
## 11.2c-1: Durable identity reservations
|
||||
|
||||
Schema v15 adds [life reservations](../../schemas/life-reservations-v1.md):
|
||||
|
||||
@@ -82,8 +82,11 @@ Local checks do not close ADR-0097's reference-worker qualification.
|
||||
converge, conflicting evidence and stale control refuse, and runtime can read
|
||||
immutable life bindings. Full PostgreSQL migrations, process, recovery,
|
||||
backup/restore, host-loss, upgrade and privilege checks pass.
|
||||
- Remaining 11.2c/d: C++ durable publication/object-store adapter, primitive-life
|
||||
region recovery/host and explicit session protocol integration, plus crash-boundary
|
||||
qualification. Schema admission alone does not make profile 3 hostable.
|
||||
- 11.2c-2d: C++ object-store publication, verified predecessor replay, exact-byte
|
||||
retry and profile-3 recovery adapter with SQL life-binding reconciliation. Full
|
||||
PostgreSQL/process/recovery and upgrade qualification passes.
|
||||
- Remaining 11.2d: primitive-capable region host and explicit new-life session
|
||||
protocol, crash-boundary qualification and readiness integration. The generic
|
||||
two-profile host still cannot serve profile 3.
|
||||
|
||||
No deployed world has been provisioned or changed by this work.
|
||||
|
||||
@@ -5620,3 +5620,19 @@ admission and a runtime binding read, but not the C++ object-store publication
|
||||
adapter or primitive-capable server recovery/session path. Do not interpret its
|
||||
green database gate as end-to-end life recovery. Full C++ unit/network/QUIC lanes
|
||||
were not rerun because this segment changes no replicated network type.
|
||||
|
||||
## 2026-10-10 — M11.2c-2d persistence adapter
|
||||
|
||||
Added verified immutable-object publication for profile 3, exact prepared manifest
|
||||
and trace retry, portable batch replay from the committed predecessor, and recovery
|
||||
that checks payload closure, head-trace replay and durable life-binding agreement.
|
||||
The focused fresh-cluster publication/recovery test passed. GCC 14 Release build,
|
||||
clang-format 19, clang-tidy 19 on the new adapter and test, and the complete
|
||||
`tools/ci/postgres-check.sh` passed, including all existing host/session/process,
|
||||
backup/restore, host-loss, upgrade and migration checks. The final full run is in
|
||||
`build/m112-c2d-full.log`.
|
||||
|
||||
No production world was provisioned or changed. The new adapter can persist and
|
||||
recover profile 3, but `RegionHost` and session routing remain limited to the
|
||||
existing environmental/lineage profiles; no primitive session can yet reach this
|
||||
adapter.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
#pragma once
|
||||
#include "evo/persistence/life_reservation_store.hpp"
|
||||
#include "evo/persistence/object_store.hpp"
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include "evo/persistence/region_checkpoint.hpp"
|
||||
@@ -79,6 +80,26 @@ struct RecoveredLineageCheckpoint {
|
||||
std::unique_ptr<processes::LineageRegion> state;
|
||||
std::vector<CommittedDecision> decisions;
|
||||
};
|
||||
struct PrimitivePreparedCommit {
|
||||
RegionKey key{};
|
||||
Uuid commit{};
|
||||
std::vector<std::uint8_t> manifest, trace;
|
||||
};
|
||||
struct PrimitivePublicationFailure {
|
||||
PublicationError kind{};
|
||||
LeaseStoreFailure store{};
|
||||
ObjectError object{};
|
||||
PrimitivePreparedCommit prepared;
|
||||
};
|
||||
struct PublishedPrimitiveBatch {
|
||||
CommittedCheckpoint committed;
|
||||
};
|
||||
struct RecoveredPrimitiveCheckpoint {
|
||||
CommittedCheckpoint committed;
|
||||
CheckpointManifest manifest;
|
||||
std::unique_ptr<server::PrimitiveLives> state;
|
||||
std::vector<PrimitiveLifeBinding> bindings;
|
||||
};
|
||||
|
||||
// Owner/operator connection only: uploads a tick-0 state and records genesis.
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, PublicationFailure> provision_environmental_region(
|
||||
@@ -128,6 +149,31 @@ recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, R
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
std::size_t restore_budget, std::chrono::milliseconds timeout);
|
||||
|
||||
// Primitive-life profile 3 stores a complete creation-only payload. Publication
|
||||
// replays the canonical birth trace from the verified predecessor before upload,
|
||||
// then uses schema v16's atomic reservation/control/checkpoint transaction.
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, PrimitivePublicationFailure>
|
||||
provision_primitive_region(Postgres& owner, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::uint64_t frame, const server::PrimitiveLives& opening, Uuid commit,
|
||||
const Sha256Digest& build, std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<PublishedPrimitiveBatch, PrimitivePublicationFailure>
|
||||
publish_primitive_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const server::PrimitiveBatch& batch,
|
||||
const server::PrimitiveLives& candidate, Uuid commit,
|
||||
const Sha256Digest& build, std::chrono::milliseconds timeout,
|
||||
const PublishHooks& hooks = {});
|
||||
// Retry after an ambiguous response with these exact immutable bytes. The SQL
|
||||
// function converges exact retries and rejects changed bytes or stale authority.
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, PrimitivePublicationFailure>
|
||||
retry_primitive_prepared(Postgres& db, const PrimitivePreparedCommit& prepared,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Read-only recovery adapter. The caller must acquire a fresh region lease before
|
||||
// using the recovered state or declaring the host ready.
|
||||
[[nodiscard]] std::expected<RecoveredPrimitiveCheckpoint, RecoveryFailure>
|
||||
recover_primitive_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
std::chrono::milliseconds timeout);
|
||||
|
||||
// After an `unknown` publication: resolve exactly the prepared bytes. A committed
|
||||
// result yields the final batch; absent may be retried with the same prepared bytes
|
||||
// only while the lease is still valid; conflict fails closed.
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <vector>
|
||||
|
||||
namespace evo::persistence {
|
||||
// Schema v15, ADR-0100. These operations allocate identity, never a biological
|
||||
@@ -32,6 +33,13 @@ struct LifeReservation {
|
||||
LifeRefusal refusal{};
|
||||
friend bool operator==(const LifeReservation&, const LifeReservation&) = default;
|
||||
};
|
||||
struct PrimitiveLifeBinding {
|
||||
Uuid account{};
|
||||
std::uint64_t request{}, identity{}, born{};
|
||||
Uuid commit{};
|
||||
std::uint64_t commit_sequence{};
|
||||
friend bool operator==(const PrimitiveLifeBinding&, const PrimitiveLifeBinding&) = default;
|
||||
};
|
||||
// Exact retry returns the original identity/owning region/refusal, even after
|
||||
// account control moves. The result grants no authority in the returned region.
|
||||
// request_id is in the new-life namespace, separate from M10 probe action IDs.
|
||||
@@ -45,4 +53,8 @@ reserve_primitive_life(Postgres& db, RegionKey key, const LeaseGrant& lease, con
|
||||
reject_primitive_life(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
|
||||
std::uint64_t generation, std::uint64_t request_id, LifeRefusal reason,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Bounded durable binding set for checkpoint reconciliation; session generations
|
||||
// are intentionally absent and must be reacquired after recovery.
|
||||
[[nodiscard]] std::expected<std::vector<PrimitiveLifeBinding>, LifeReservationFailure>
|
||||
load_primitive_lives(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -0,0 +1,347 @@
|
||||
#include "evo/persistence/checkpoint_store.hpp"
|
||||
#include "evo/persistence/sha256.hpp"
|
||||
#include <algorithm>
|
||||
#include <array>
|
||||
#include <charconv>
|
||||
#include <ranges>
|
||||
|
||||
namespace evo::persistence {
|
||||
namespace {
|
||||
using std::chrono::milliseconds;
|
||||
constexpr std::string_view digits = "0123456789abcdef";
|
||||
|
||||
std::string hex(std::span<const std::uint8_t> bytes) {
|
||||
std::string out = "\\x";
|
||||
out.reserve(2 + bytes.size() * 2);
|
||||
for (const auto b : bytes) {
|
||||
out.push_back(digits[b >> 4U]);
|
||||
out.push_back(digits[b & 0x0fU]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::optional<std::uint8_t> nibble(char c) {
|
||||
const auto i = digits.find(c);
|
||||
if (i == std::string_view::npos)
|
||||
return std::nullopt;
|
||||
return static_cast<std::uint8_t>(i);
|
||||
}
|
||||
template <std::size_t N> std::optional<std::array<std::uint8_t, N>> unhex(std::string_view s) {
|
||||
if (s.size() != N * 2)
|
||||
return std::nullopt;
|
||||
std::array<std::uint8_t, N> out{};
|
||||
for (std::size_t i = 0; i < N; ++i) {
|
||||
const auto hi = nibble(s[i * 2]);
|
||||
const auto lo = nibble(s[i * 2 + 1]);
|
||||
if (!hi || !lo)
|
||||
return std::nullopt;
|
||||
out[i] = static_cast<std::uint8_t>((*hi << 4U) | *lo);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::optional<Uuid> uuid(const std::optional<std::string>& value) {
|
||||
if (!value || value->size() != 36)
|
||||
return std::nullopt;
|
||||
std::string compact;
|
||||
compact.reserve(32);
|
||||
for (std::size_t i = 0; i < value->size(); ++i) {
|
||||
const bool dash = i == 8 || i == 13 || i == 18 || i == 23;
|
||||
if (dash != ((*value)[i] == '-'))
|
||||
return std::nullopt;
|
||||
if (!dash)
|
||||
compact.push_back((*value)[i]);
|
||||
}
|
||||
return unhex<16>(compact);
|
||||
}
|
||||
template <typename T> std::optional<T> integer(const std::optional<std::string>& s) {
|
||||
if (!s || s->empty() || s->size() > 20 || s->front() == '-' || s->front() == '+')
|
||||
return std::nullopt;
|
||||
T value{};
|
||||
const auto [end, ec] = std::from_chars(s->data(), s->data() + s->size(), value);
|
||||
if (ec != std::errc{} || end != s->data() + s->size())
|
||||
return std::nullopt;
|
||||
return value;
|
||||
}
|
||||
PrimitivePublicationFailure fail(PublicationError kind, ObjectError object = {},
|
||||
LeaseStoreFailure store = {},
|
||||
PrimitivePreparedCommit prepared = {}) {
|
||||
return {kind, store, object, std::move(prepared)};
|
||||
}
|
||||
std::expected<void, PrimitivePublicationFailure> put_verified(const FilesystemObjectStore& objects,
|
||||
std::span<const std::uint8_t> bytes,
|
||||
VerifiedObject expected) {
|
||||
auto stored = objects.put(bytes);
|
||||
if (!stored)
|
||||
return std::unexpected{fail(PublicationError::upload, stored.error())};
|
||||
auto readback = objects.get(*stored);
|
||||
if (!readback || *stored != expected || !std::ranges::equal(*readback, bytes))
|
||||
return std::unexpected{
|
||||
fail(PublicationError::verify, readback ? ObjectError::corrupt : readback.error())};
|
||||
return {};
|
||||
}
|
||||
std::optional<RegionHead> parse_head(const PgRows& rows) {
|
||||
if (rows.size() != 1 || rows[0].size() != 5)
|
||||
return std::nullopt;
|
||||
const auto frame = integer<std::uint64_t>(rows[0][0]);
|
||||
const auto sequence = integer<std::uint64_t>(rows[0][1]);
|
||||
const auto commit = uuid(rows[0][2]);
|
||||
const auto digest_text = rows[0][3].value_or(std::string{});
|
||||
const auto digest = unhex<32>(digest_text);
|
||||
const auto tick = integer<std::uint64_t>(rows[0][4]);
|
||||
if (!frame || !sequence || !commit || !digest || !tick)
|
||||
return std::nullopt;
|
||||
return RegionHead{*frame, *sequence, *commit, *digest, *tick};
|
||||
}
|
||||
std::expected<CommittedCheckpoint, PrimitivePublicationFailure>
|
||||
confirm(const RegionHead& head, const PrimitivePreparedCommit& prepared,
|
||||
const CheckpointManifest& manifest) {
|
||||
const auto digest = sha256(prepared.manifest);
|
||||
if (!digest || head.digest != *digest || head.sequence != manifest.sequence ||
|
||||
head.commit != manifest.commit || head.end_tick != manifest.end_tick ||
|
||||
head.frame != manifest.frame)
|
||||
return std::unexpected{fail(PublicationError::receipt)};
|
||||
return CommittedCheckpoint{head, prepared.manifest};
|
||||
}
|
||||
bool same_binding(const server::PrimitiveLife& life, const PrimitiveLifeBinding& binding) {
|
||||
return life.request.account == binding.account && life.request.action == binding.request &&
|
||||
life.identity.life == binding.identity && life.identity.born.value == binding.born;
|
||||
}
|
||||
std::expected<std::vector<PrimitiveLifeBinding>, RecoveryFailure>
|
||||
load_bindings(Postgres& db, RegionKey key, milliseconds timeout) {
|
||||
auto bindings = load_primitive_lives(db, key, timeout);
|
||||
if (!bindings)
|
||||
return std::unexpected{RecoveryFailure{
|
||||
RecoveryError::store,
|
||||
{bindings.error().kind == LifeReservationError::unknown ? LeaseStoreError::unknown
|
||||
: LeaseStoreError::store,
|
||||
bindings.error().cause}}};
|
||||
return std::move(*bindings);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<CommittedCheckpoint, PrimitivePublicationFailure>
|
||||
provision_primitive_region(Postgres& owner, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::uint64_t frame, const server::PrimitiveLives& opening, Uuid commit,
|
||||
const Sha256Digest& build, milliseconds timeout) {
|
||||
const auto& config = opening.config();
|
||||
const auto payload = opening.snapshot();
|
||||
if (config.world != key.world || config.region != key.region || config.frame != frame ||
|
||||
!opening.lives().empty())
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const auto digest = sha256(payload);
|
||||
if (!digest)
|
||||
return std::unexpected{fail(PublicationError::digest)};
|
||||
auto manifest = genesis_manifest(key, frame, commit, 50000, build, payload.size(), *digest,
|
||||
CheckpointProfile::primitive_lives_v1);
|
||||
auto encoded = manifest ? encode_manifest(*manifest)
|
||||
: std::expected<std::vector<std::uint8_t>, CheckpointError>{
|
||||
std::unexpected{CheckpointError::format}};
|
||||
if (!encoded || !restore_primitive_payload(payload, *manifest))
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
if (auto stored = put_verified(objects, payload, {*digest, payload.size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
const std::array<std::string, 1> p{hex(*encoded)};
|
||||
auto inserted =
|
||||
owner.query("SELECT evolution.provision_region_checkpoint($1::bytea)", p, timeout);
|
||||
if (!inserted) {
|
||||
const bool unknown = inserted.error().kind == PgError::unknown;
|
||||
return std::unexpected{
|
||||
fail(unknown ? PublicationError::unknown : PublicationError::rejected, {},
|
||||
{unknown ? LeaseStoreError::unknown : LeaseStoreError::store, inserted.error()})};
|
||||
}
|
||||
auto committed = load_region_checkpoint(owner, key, timeout);
|
||||
if (!committed)
|
||||
return std::unexpected{fail(PublicationError::unknown, {}, committed.error())};
|
||||
PrimitivePreparedCommit prepared{key, commit, *encoded, {}};
|
||||
auto confirmed = confirm(committed->head, prepared, *manifest);
|
||||
if (!confirmed)
|
||||
return std::unexpected{confirmed.error()};
|
||||
return *confirmed;
|
||||
}
|
||||
|
||||
std::expected<CommittedCheckpoint, PrimitivePublicationFailure>
|
||||
retry_primitive_prepared(Postgres& db, const PrimitivePreparedCommit& prepared,
|
||||
milliseconds timeout) {
|
||||
auto manifest = decode_manifest(prepared.manifest);
|
||||
auto trace = server::decode_primitive_batch(prepared.trace);
|
||||
const auto trace_digest = sha256(prepared.trace);
|
||||
if (!manifest || !trace || !trace_digest || manifest->key != prepared.key ||
|
||||
manifest->commit != prepared.commit ||
|
||||
manifest->profile != CheckpointProfile::primitive_lives_v1 || manifest->sequence == 0 ||
|
||||
manifest->batch.trace_bytes != prepared.trace.size() ||
|
||||
manifest->batch.trace_sha256 != *trace_digest ||
|
||||
trace->header.world != prepared.key.world || trace->header.region != prepared.key.region ||
|
||||
trace->header.frame != manifest->frame ||
|
||||
trace->header.predecessor != manifest->predecessor_commit ||
|
||||
trace->header.predecessor_digest != manifest->predecessor_digest ||
|
||||
trace->header.begin_tick != manifest->begin_tick ||
|
||||
trace->header.end_tick != manifest->end_tick)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const std::array<std::string, 2> p{hex(prepared.manifest), hex(prepared.trace)};
|
||||
auto rows = db.query(
|
||||
"SELECT * FROM evolution.publish_primitive_checkpoint($1::bytea, $2::bytea)", p, timeout);
|
||||
if (!rows) {
|
||||
const bool unknown = rows.error().kind == PgError::unknown;
|
||||
return std::unexpected{fail(
|
||||
unknown ? PublicationError::unknown : PublicationError::rejected, {},
|
||||
{unknown ? LeaseStoreError::unknown : LeaseStoreError::store, rows.error()}, prepared)};
|
||||
}
|
||||
const auto head = parse_head(*rows);
|
||||
if (!head)
|
||||
return std::unexpected{
|
||||
fail(PublicationError::unknown, {}, {LeaseStoreError::unknown, {}}, prepared)};
|
||||
return confirm(*head, prepared, *manifest);
|
||||
}
|
||||
|
||||
std::expected<PublishedPrimitiveBatch, PrimitivePublicationFailure>
|
||||
publish_primitive_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const server::PrimitiveBatch& batch,
|
||||
const server::PrimitiveLives& candidate, Uuid commit,
|
||||
const Sha256Digest& build, milliseconds timeout,
|
||||
const PublishHooks& hooks) {
|
||||
const auto previous = decode_manifest(predecessor.manifest);
|
||||
const auto predecessor_digest = sha256(predecessor.manifest);
|
||||
auto trace = server::encode_primitive_batch(batch);
|
||||
if (!previous || !predecessor_digest || !trace ||
|
||||
previous->profile != CheckpointProfile::primitive_lives_v1 ||
|
||||
predecessor.head.digest != *predecessor_digest ||
|
||||
previous->sequence != predecessor.head.sequence ||
|
||||
previous->commit != predecessor.head.commit ||
|
||||
previous->end_tick != predecessor.head.end_tick ||
|
||||
batch.header.world != previous->key.world || batch.header.region != previous->key.region ||
|
||||
batch.header.frame != previous->frame || batch.header.predecessor != previous->commit ||
|
||||
batch.header.predecessor_digest != *predecessor_digest ||
|
||||
batch.header.begin_tick != previous->end_tick)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
auto base_bytes = objects.get({previous->payload_sha256, previous->payload_bytes});
|
||||
if (!base_bytes)
|
||||
return std::unexpected{fail(PublicationError::predecessor, base_bytes.error())};
|
||||
auto base = restore_primitive_payload(*base_bytes, *previous);
|
||||
if (!base)
|
||||
return std::unexpected{fail(PublicationError::predecessor)};
|
||||
auto replay = server::replay_primitive_batch(**base, batch);
|
||||
const auto candidate_bytes = candidate.snapshot();
|
||||
if (!replay || (*replay)->snapshot() != candidate_bytes)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const auto digest = sha256(candidate_bytes);
|
||||
const auto trace_digest = sha256(*trace);
|
||||
if (!digest || !trace_digest)
|
||||
return std::unexpected{fail(PublicationError::digest)};
|
||||
auto manifest = successor_manifest(*previous, *predecessor_digest, commit, lease,
|
||||
batch.header.end_tick, build, candidate_bytes.size(),
|
||||
*digest, {trace->size(), *trace_digest, 0, {}});
|
||||
auto encoded = manifest ? encode_manifest(*manifest)
|
||||
: std::expected<std::vector<std::uint8_t>, CheckpointError>{
|
||||
std::unexpected{CheckpointError::format}};
|
||||
if (!encoded || !restore_primitive_payload(candidate_bytes, *manifest))
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
if (auto stored = put_verified(objects, candidate_bytes, {*digest, candidate_bytes.size()});
|
||||
!stored)
|
||||
return std::unexpected{stored.error()};
|
||||
if (auto stored = put_verified(objects, *trace, {*trace_digest, trace->size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
PrimitivePreparedCommit prepared{previous->key, commit, *encoded, *trace};
|
||||
if (hooks.before_transaction)
|
||||
hooks.before_transaction();
|
||||
auto committed = retry_primitive_prepared(db, prepared, timeout);
|
||||
if (!committed)
|
||||
return std::unexpected{committed.error()};
|
||||
return PublishedPrimitiveBatch{std::move(*committed)};
|
||||
}
|
||||
|
||||
std::expected<RecoveredPrimitiveCheckpoint, RecoveryFailure>
|
||||
recover_primitive_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds, milliseconds timeout) {
|
||||
auto committed = load_region_checkpoint(db, key, timeout);
|
||||
if (!committed)
|
||||
return std::unexpected{
|
||||
RecoveryFailure{committed.error().kind == LeaseStoreError::not_provisioned
|
||||
? RecoveryError::not_provisioned
|
||||
: RecoveryError::store,
|
||||
committed.error()}};
|
||||
const auto digest = sha256(committed->manifest);
|
||||
auto manifest = decode_manifest(committed->manifest);
|
||||
if (!digest || *digest != committed->head.digest || !manifest || manifest->key != key ||
|
||||
manifest->sequence != committed->head.sequence ||
|
||||
manifest->commit != committed->head.commit ||
|
||||
manifest->end_tick != committed->head.end_tick || manifest->frame != committed->head.frame)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::manifest, {}}};
|
||||
if (!supported(*manifest) || manifest->profile != CheckpointProfile::primitive_lives_v1 ||
|
||||
std::ranges::find(retained_builds, manifest->build) == retained_builds.end())
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::unsupported, {}}};
|
||||
auto payload = objects.get({manifest->payload_sha256, manifest->payload_bytes});
|
||||
if (!payload)
|
||||
return std::unexpected{RecoveryFailure{
|
||||
payload.error() == ObjectError::missing ? RecoveryError::missing_payload
|
||||
: payload.error() == ObjectError::corrupt ? RecoveryError::corrupt_payload
|
||||
: payload.error() == ObjectError::io ? RecoveryError::store
|
||||
: RecoveryError::corrupt_payload,
|
||||
{}}};
|
||||
auto state = restore_primitive_payload(*payload, *manifest);
|
||||
if (!state)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::payload, {}}};
|
||||
auto bindings = load_bindings(db, key, timeout);
|
||||
if (!bindings)
|
||||
return std::unexpected{bindings.error()};
|
||||
if ((*state)->lives().size() != bindings->size())
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::history, {}}};
|
||||
for (const auto& life : (*state)->lives()) {
|
||||
const auto binding = std::ranges::find_if(*bindings, [&](const auto& b) {
|
||||
return same_binding(life, b) && b.commit_sequence <= manifest->sequence;
|
||||
});
|
||||
if (binding == bindings->end())
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::history, {}}};
|
||||
}
|
||||
if (manifest->sequence == 0) {
|
||||
if (!bindings->empty() || !(*state)->lives().empty())
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::history, {}}};
|
||||
} else {
|
||||
auto trace_bytes = objects.get({manifest->batch.trace_sha256, manifest->batch.trace_bytes});
|
||||
auto trace = trace_bytes
|
||||
? server::decode_primitive_batch(*trace_bytes)
|
||||
: std::expected<server::PrimitiveBatch, server::PrimitiveBatchError>{
|
||||
std::unexpected{server::PrimitiveBatchError::format}};
|
||||
if (!trace_bytes || !trace)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
const auto& h = trace->header;
|
||||
if (h.world != key.world || h.region != key.region || h.frame != manifest->frame ||
|
||||
h.predecessor != manifest->predecessor_commit ||
|
||||
h.predecessor_digest != manifest->predecessor_digest ||
|
||||
h.begin_tick != manifest->begin_tick || h.end_tick != manifest->end_tick)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
auto predecessor = load_region_commit(db, key, manifest->sequence - 1, timeout);
|
||||
if (!predecessor)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, predecessor.error()}};
|
||||
const auto predecessor_digest = sha256(predecessor->manifest);
|
||||
auto predecessor_manifest = decode_manifest(predecessor->manifest);
|
||||
if (!predecessor_digest || *predecessor_digest != manifest->predecessor_digest ||
|
||||
predecessor->head.sequence != manifest->sequence - 1 ||
|
||||
predecessor->head.commit != manifest->predecessor_commit ||
|
||||
predecessor->head.digest != manifest->predecessor_digest || !predecessor_manifest ||
|
||||
predecessor_manifest->sequence != manifest->sequence - 1 ||
|
||||
predecessor_manifest->end_tick != manifest->begin_tick)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
auto predecessor_payload = objects.get(
|
||||
{predecessor_manifest->payload_sha256, predecessor_manifest->payload_bytes});
|
||||
if (!predecessor_payload)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
auto predecessor_state =
|
||||
restore_primitive_payload(*predecessor_payload, *predecessor_manifest);
|
||||
if (!predecessor_state)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
auto replay = server::replay_primitive_batch(**predecessor_state, *trace);
|
||||
if (!replay || (*replay)->snapshot() != *payload)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::tail, {}}};
|
||||
for (const auto& birth : trace->births) {
|
||||
const auto binding = std::ranges::find_if(*bindings, [&](const auto& b) {
|
||||
return b.account == birth.request.account && b.request == birth.request.action;
|
||||
});
|
||||
if (binding == bindings->end() || binding->identity != birth.identity ||
|
||||
binding->born != birth.born.value || binding->commit != manifest->commit ||
|
||||
binding->commit_sequence != manifest->sequence)
|
||||
return std::unexpected{RecoveryFailure{RecoveryError::history, {}}};
|
||||
}
|
||||
}
|
||||
return RecoveredPrimitiveCheckpoint{std::move(*committed), *manifest, std::move(*state),
|
||||
std::move(*bindings)};
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
@@ -661,6 +661,35 @@ reject_primitive_life(Postgres& db, RegionKey key, const LeaseGrant& lease, cons
|
||||
return static_cast<LifeRefusal>(*refusal);
|
||||
}
|
||||
|
||||
std::expected<std::vector<PrimitiveLifeBinding>, LifeReservationFailure>
|
||||
load_primitive_lives(Postgres& db, RegionKey key, milliseconds timeout) {
|
||||
const std::array<std::string, 2> p{std::to_string(key.world), std::to_string(key.region)};
|
||||
auto rows = db.query("SELECT * FROM evolution.load_primitive_lives($1::evolution.u64, "
|
||||
"$2::evolution.u64)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{life_failure(rows.error())};
|
||||
if (rows->size() > 256)
|
||||
return std::unexpected{LifeReservationFailure{LifeReservationError::store, {}}};
|
||||
std::vector<PrimitiveLifeBinding> out;
|
||||
out.reserve(rows->size());
|
||||
for (const auto& row : *rows) {
|
||||
if (row.size() != 6)
|
||||
return std::unexpected{LifeReservationFailure{LifeReservationError::store, {}}};
|
||||
const auto account = parse_uuid(row[0]);
|
||||
const auto request = parse_integer<std::uint64_t>(row[1]);
|
||||
const auto identity = parse_integer<std::uint64_t>(row[2]);
|
||||
const auto born = parse_integer<std::uint64_t>(row[3]);
|
||||
const auto commit = parse_uuid(row[4]);
|
||||
const auto sequence = parse_integer<std::uint64_t>(row[5]);
|
||||
if (!account || !request || *request == 0 || !identity || *identity == 0 || !born ||
|
||||
*born == 0 || !commit || !sequence)
|
||||
return std::unexpected{LifeReservationFailure{LifeReservationError::store, {}}};
|
||||
out.push_back({*account, *request, *identity, *born, *commit, *sequence});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Schema v12 probe records and handoff (ADR-0096). Shares this file's parsing rules.
|
||||
namespace {
|
||||
ProbeStoreFailure probe_failure(const PgFailure& f) {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
// Real database atomic-publication checks. Payloads/traces use the real portable
|
||||
// codecs; object upload/recovery belongs to the subsequent adapter subsection.
|
||||
// Real database publication checks using the portable payload and trace codecs.
|
||||
#include "evo/persistence/account_control_store.hpp"
|
||||
#include "evo/persistence/checkpoint_store.hpp"
|
||||
#include "evo/persistence/life_reservation_store.hpp"
|
||||
#include "evo/persistence/region_checkpoint.hpp"
|
||||
#include "evo/persistence/region_lease_store.hpp"
|
||||
@@ -55,6 +55,57 @@ Uuid test_commit(std::uint8_t suffix) {
|
||||
bool state_is(const auto& result, std::string_view state) {
|
||||
return !result && std::string_view(result.error().sqlstate.data()) == state;
|
||||
}
|
||||
void adapter_publication_and_recovery() {
|
||||
auto owner = connect_as(env("PGUSER"));
|
||||
auto db = connect_as(env("EVO_PG_RUNTIME_USER"));
|
||||
const RegionKey key{9754, 1};
|
||||
const auto build = id<32>(0xD7);
|
||||
CHECK(owner->query("INSERT INTO evolution.retained_builds(build_identity) VALUES "
|
||||
"(decode(repeat('d7',32),'hex')) ON CONFLICT DO NOTHING",
|
||||
{}, 5s));
|
||||
auto endowed = evo::biology::primitive_endowment();
|
||||
CHECK(endowed);
|
||||
auto funding = endowed->body;
|
||||
funding.energy.value = endowed->reserve.energy.value + endowed->construction.value;
|
||||
auto base = PrimitiveLives::create({9754, 1, 1, 1, funding, {{0, 0}}});
|
||||
CHECK(base);
|
||||
auto objects = FilesystemObjectStore::open(env("PGHOST"), key, max_primitive_lives_bytes);
|
||||
CHECK(objects);
|
||||
const auto genesis =
|
||||
provision_primitive_region(*owner, **objects, key, 1, **base, test_commit(10), build, 5s);
|
||||
CHECK(genesis);
|
||||
CHECK(owner->query("INSERT INTO evolution.life_allocators VALUES(9754,1)", {}, 5s));
|
||||
auto lease = acquire_region_lease(*db, key, build, 20000ms, 5s);
|
||||
CHECK(lease);
|
||||
const auto account = id<16>(0x72);
|
||||
const auto control = claim_account_control(*db, key, *lease, account, 1, 1, 20000ms, 5s);
|
||||
CHECK(control);
|
||||
const auto reserved =
|
||||
reserve_primitive_life(*db, key, *lease, account, control->generation, 1, 1, 5s);
|
||||
CHECK(reserved && reserved->identity == 1);
|
||||
const auto predecessor_digest = sha256(genesis->manifest);
|
||||
CHECK(predecessor_digest);
|
||||
PrimitiveBatch batch{
|
||||
{key.world, key.region, 1, genesis->head.commit, *predecessor_digest, 0, 1},
|
||||
{{{account, 1}, reserved->identity, control->generation, {1}}}};
|
||||
auto candidate = replay_primitive_batch(**base, batch);
|
||||
CHECK(candidate);
|
||||
const auto published = publish_primitive_batch(*db, **objects, *lease, *genesis, batch,
|
||||
**candidate, test_commit(11), build, 5s);
|
||||
CHECK(published && published->committed.head.sequence == 1);
|
||||
CHECK(release_region_lease(*db, key, *lease, 5s));
|
||||
const auto retry = publish_primitive_batch(*db, **objects, *lease, *genesis, batch, **candidate,
|
||||
test_commit(11), build, 5s);
|
||||
CHECK(retry && retry->committed.manifest == published->committed.manifest &&
|
||||
retry->committed.head == published->committed.head);
|
||||
const auto successor_lease = acquire_region_lease(*db, key, build, 20000ms, 5s);
|
||||
CHECK(successor_lease && successor_lease->epoch > lease->epoch);
|
||||
const std::array<Sha256Digest, 1> retained{build};
|
||||
auto recovered = recover_primitive_checkpoint(*db, **objects, key, retained, 5s);
|
||||
CHECK(recovered && recovered->bindings.size() == 1 &&
|
||||
recovered->state->snapshot() == (*candidate)->snapshot());
|
||||
CHECK(release_region_lease(*db, key, *successor_lease, 5s));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_primitive_publication_tests() {
|
||||
@@ -229,4 +280,5 @@ void run_primitive_publication_tests() {
|
||||
CHECK(release_region_lease(*db, key, *lease, 5s));
|
||||
// Exact committed retry survives loss of authority, but cannot originate work.
|
||||
CHECK(publish(*manifest, *trace));
|
||||
adapter_publication_and_recovery();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user