Add primitive profile region host
Foundation / checks (deploy) (push) Successful in 6s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (checks) (push) Failing after 3m43s
Foundation / checks (tsan) (push) Canceled after 0s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 3m2s
Foundation / checks (network) (push) Canceled after 4m39s
Foundation / checks (client) (push) Canceled after 6m48s
Foundation / checks (deploy) (push) Successful in 6s
Foundation / checks (ecs) (push) Successful in 1m57s
Foundation / checks (checks) (push) Failing after 3m43s
Foundation / checks (tsan) (push) Canceled after 0s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 3m2s
Foundation / checks (network) (push) Canceled after 4m39s
Foundation / checks (client) (push) Canceled after 6m48s
This commit is contained in:
+2
-1
@@ -231,7 +231,8 @@ if(EVO_BUILD_POSTGRES)
|
||||
PRIVATE OpenSSL::Crypto)
|
||||
evo_owned_target(evo_region_store)
|
||||
# M7.6 authoritative region host and its process (ADR-0064).
|
||||
add_library(evo_region_host STATIC src/server/region_host.cpp src/server/ops_http.cpp)
|
||||
add_library(evo_region_host STATIC src/server/region_host.cpp
|
||||
src/server/primitive_region_host.cpp src/server/ops_http.cpp)
|
||||
add_library(evo::region_host ALIAS evo_region_host)
|
||||
target_link_libraries(evo_region_host PUBLIC evo::region_store evo::ops)
|
||||
evo_owned_target(evo_region_host)
|
||||
|
||||
+5
-1
@@ -45,7 +45,11 @@ the maintainer chose directional rounding):
|
||||
closure and exact retry recovery. 11.2c-1 adds durable identity reservations,
|
||||
world-wide account exclusivity, and retained refusals under fenced account
|
||||
control. A reservation spends nothing and is not a spawned organism.
|
||||
Atomic body/checkpoint publication, life binding and session integration remain.
|
||||
Atomic body/checkpoint publication and life binding are in the adapter. The
|
||||
separate profile-3 region host now recovers and durably publishes a new life,
|
||||
without advancing biology. v5 session integration, runtime profile selection,
|
||||
and process-kill qualification remain; the v17 PostgreSQL and host integration
|
||||
tests are waiting on CI confirmation.
|
||||
|
||||
Open from M10 (BACKLOG):
|
||||
- the maintainer's native-client observation, moved to the later 2k drill;
|
||||
|
||||
@@ -229,10 +229,9 @@ speculative body state and prevents the old reservation becoming publishable
|
||||
under a new controller. V5 now gives `existing_life` and `cancelled` durable
|
||||
result values distinct from query-only `pending` and `unknown`.
|
||||
|
||||
Migration v17, its typed query adapter and PostgreSQL tests implement this
|
||||
contract. Host/session execution, concurrent-process crash qualification and
|
||||
readiness integration remain outstanding; v5 stays refused by the running
|
||||
server until those gates pass.
|
||||
Migration v17 and its typed query adapter implement this contract. Host/session
|
||||
execution, concurrent-process crash qualification and readiness integration are
|
||||
separate gates; v5 stays refused by the running server until those gates pass.
|
||||
|
||||
## 11.2c-2d: Object-store publication and recovery adapter
|
||||
|
||||
@@ -269,3 +268,20 @@ recording life ownership. The portable trace alone does not provide durability.
|
||||
New-life request IDs use a separate operation namespace from M10 no-effect probe
|
||||
actions. The future protocol must make that explicit rather than overloading the
|
||||
existing probe result lookup. This avoids silently changing deployed probe rules.
|
||||
|
||||
## 11.2d-3: Primitive region host
|
||||
|
||||
`PrimitiveRegionHost` is separate from the environmental/lineage tick host. It
|
||||
acquires and renews the region lease, becomes ready only after profile-3 recovery,
|
||||
and applies a new-life command as one creation-only checkpoint batch. Each birth
|
||||
advances the checkpoint's committed tick by one without advancing biological
|
||||
state; this is a durable operation boundary, not a simulation tick. The host
|
||||
reserves under the authenticated account-control generation, constructs a
|
||||
detached replay candidate, and returns a born result only after the atomic
|
||||
checkpoint/reservation transaction confirms. Funding and placement refusals are
|
||||
recorded durably. Exact retries query the stored outcome; an ambiguous publish
|
||||
retains the exact prepared commit and pauses admission until reconciliation.
|
||||
|
||||
This first host slice is not connected to `evo-server` or the v5 session protocol.
|
||||
Network admission, routing, process-kill qualification and readiness wiring remain
|
||||
separate implementation gates.
|
||||
|
||||
@@ -98,10 +98,19 @@ Local checks do not close ADR-0097's reference-worker qualification.
|
||||
migration, codec and PostgreSQL tests/docs are updated. Validation is pending:
|
||||
the affected C++ files were compiled directly with MSVC and the updated network
|
||||
and QUIC suites passed (2/2 and 5/5). PostgreSQL 18.6 is unavailable here, so
|
||||
migration and adapter integration tests remain unrun. Host work remains paused
|
||||
until database validation passes.
|
||||
- Remaining 11.2d after 11.2d-2 validation: primitive-capable region host,
|
||||
server admission/execution, crash-boundary qualification and readiness
|
||||
integration. The generic two-profile host still cannot serve profile 3.
|
||||
migration and adapter integration tests remain unrun locally; rely on CI for
|
||||
this database gate before merging or advancing another subsection.
|
||||
- 11.2d-3 implementation: a separate primitive host acquires/renews the fenced
|
||||
region lease, recovers profile 3 before readiness, and executes one authenticated
|
||||
new-life request as a detached replay plus atomic publication. Exact published
|
||||
retries query the durable result; unknown publication retains and retries the
|
||||
exact prepared commit, while uncertain/corrupt state withholds readiness. The
|
||||
host does not advance biology and is not connected to `evo-server` or QUIC yet.
|
||||
Release/NDEBUG translation-unit compilation and the changed integration-test
|
||||
translation unit passed locally. Runtime PostgreSQL tests and full CI are pending;
|
||||
do not enable v5 or claim 11.2d complete until they pass.
|
||||
- Remaining 11.2d: `evo-server` profile selection/provisioning, authenticated v5
|
||||
session admission and routing, process-kill qualification and readiness
|
||||
integration. The generic two-profile host still serves only profiles 1 and 2.
|
||||
|
||||
No deployed world has been provisioned or changed by this work.
|
||||
|
||||
@@ -5677,5 +5677,21 @@ objects; `ctest --preset network` passed (2/2) and `ctest --preset quic` passed
|
||||
|
||||
`psql` and PostgreSQL 18.6 are unavailable in this environment; Docker's Linux
|
||||
engine is also not running. Schema v17 and its PostgreSQL integration, migration,
|
||||
upgrade and privilege tests have not run. Do not continue to 11.2d host/session
|
||||
execution until those database checks pass.
|
||||
upgrade and privilege tests have not run locally. CI must pass these before the
|
||||
host/session subsection advances; the maintainer will confirm that result.
|
||||
|
||||
## 2026-10-10 — M11.2d-3 primitive region host (CI pending)
|
||||
|
||||
Added a separate creation-only `PrimitiveRegionHost`: fenced lease acquisition
|
||||
and renewal, profile-3 recovery before readiness, durable new-life reservation and
|
||||
refusal handling, detached replay, atomic publication and exact retry/query
|
||||
reconciliation. It does not advance biology and is not wired to `evo-server` or
|
||||
QUIC. Added a real-PostgreSQL integration case for recovery, first birth, exact
|
||||
retry and world/account exclusivity.
|
||||
|
||||
Executed: GCC 13.2 Release/NDEBUG compilation with warnings-as-errors for the new
|
||||
host translation unit and changed PostgreSQL test translation unit; `git diff
|
||||
--check`. Both compilations passed. The local Windows shell cannot configure the
|
||||
Linux-only PostgreSQL target, and this turn did not run the PostgreSQL test binary,
|
||||
clang-format, or clang-tidy. Awaiting CI for the database, migration, host test and
|
||||
repository checks; no local runtime pass is claimed.
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
#pragma once
|
||||
|
||||
#include "evo/persistence/checkpoint_store.hpp"
|
||||
#include "evo/server/ops.hpp"
|
||||
|
||||
#include <chrono>
|
||||
#include <expected>
|
||||
#include <functional>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <vector>
|
||||
|
||||
// Authoritative creation-only host for checkpoint profile 3. This host does not
|
||||
// advance biology; each accepted birth is its own fenced, replayable checkpoint.
|
||||
namespace evo::server {
|
||||
using PrimitiveConnect =
|
||||
std::function<std::expected<std::unique_ptr<persistence::Postgres>, persistence::PgFailure>()>;
|
||||
struct PrimitiveHostConfig {
|
||||
persistence::RegionKey key{};
|
||||
std::uint64_t frame{};
|
||||
persistence::LeasePolicy lease{};
|
||||
persistence::Sha256Digest build{};
|
||||
std::vector<persistence::Sha256Digest> retained_builds;
|
||||
};
|
||||
enum class PrimitiveHostError : std::uint8_t { configuration };
|
||||
enum class PrimitiveHostReplyKind : std::uint8_t {
|
||||
born,
|
||||
refused,
|
||||
pending,
|
||||
unknown,
|
||||
routed,
|
||||
unavailable
|
||||
};
|
||||
struct PrimitiveHostReply {
|
||||
PrimitiveHostReplyKind kind{};
|
||||
persistence::LifeReservation reservation{};
|
||||
std::optional<persistence::PrimitiveLifeBinding> published;
|
||||
friend bool operator==(const PrimitiveHostReply&, const PrimitiveHostReply&) = default;
|
||||
};
|
||||
|
||||
class PrimitiveRegionHost {
|
||||
public:
|
||||
[[nodiscard]] static std::expected<std::unique_ptr<PrimitiveRegionHost>,
|
||||
PrimitiveHostError>
|
||||
create(PrimitiveHostConfig config, PrimitiveConnect connect,
|
||||
std::unique_ptr<persistence::FilesystemObjectStore> objects);
|
||||
PrimitiveRegionHost(const PrimitiveRegionHost&) = delete;
|
||||
PrimitiveRegionHost& operator=(const PrimitiveRegionHost&) = delete;
|
||||
~PrimitiveRegionHost();
|
||||
|
||||
// Performs lease upkeep and full profile-3 recovery. It is the only path to
|
||||
// ready; no missing/empty checkpoint is substituted.
|
||||
const HostStatus& step();
|
||||
// Stops requests and best-effort releases the current lease.
|
||||
void drain();
|
||||
// The caller supplies the authenticated account-control generation. The
|
||||
// operation is durable before `born` or a final refusal is returned.
|
||||
[[nodiscard]] PrimitiveHostReply new_life(const persistence::Uuid& account,
|
||||
std::uint64_t generation,
|
||||
std::uint64_t request,
|
||||
std::uint32_t rules);
|
||||
[[nodiscard]] std::optional<persistence::LeaseGrant> authority() const;
|
||||
[[nodiscard]] const HostStatus& status() const noexcept { return status_; }
|
||||
[[nodiscard]] const PrimitiveLives* state() const noexcept { return state_.get(); }
|
||||
|
||||
private:
|
||||
struct Pending;
|
||||
PrimitiveRegionHost(PrimitiveHostConfig config, PrimitiveConnect connect,
|
||||
std::unique_ptr<persistence::FilesystemObjectStore> objects);
|
||||
bool database();
|
||||
void acquire();
|
||||
void upkeep();
|
||||
void refresh();
|
||||
[[nodiscard]] PrimitiveHostReply query(const persistence::Uuid& account,
|
||||
std::uint64_t generation,
|
||||
std::uint64_t request);
|
||||
|
||||
PrimitiveHostConfig config_;
|
||||
PrimitiveConnect connect_;
|
||||
std::unique_ptr<persistence::FilesystemObjectStore> objects_;
|
||||
std::unique_ptr<persistence::Postgres> db_;
|
||||
persistence::LeaseAuthority authority_;
|
||||
std::unique_ptr<PrimitiveLives> state_;
|
||||
std::optional<persistence::CommittedCheckpoint> committed_;
|
||||
std::unique_ptr<Pending> pending_;
|
||||
HostStatus status_{};
|
||||
};
|
||||
} // namespace evo::server
|
||||
@@ -0,0 +1,482 @@
|
||||
#include "evo/server/primitive_region_host.hpp"
|
||||
|
||||
#include "evo/persistence/sha256.hpp"
|
||||
#include "evo/server/lifecycle/primitive_batch.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
namespace evo::server {
|
||||
struct PrimitiveRegionHost::Pending {
|
||||
persistence::Uuid account{};
|
||||
std::uint64_t generation{}, request{};
|
||||
persistence::LifeReservation reservation{};
|
||||
PrimitiveBatch batch{};
|
||||
std::unique_ptr<PrimitiveLives> candidate;
|
||||
persistence::Uuid commit{};
|
||||
std::optional<persistence::PrimitivePreparedCommit> prepared;
|
||||
};
|
||||
|
||||
namespace {
|
||||
using namespace std::chrono_literals;
|
||||
using persistence::LeaseClock;
|
||||
using persistence::LeaseStoreError;
|
||||
|
||||
bool valid(const PrimitiveHostConfig& c) {
|
||||
return c.key.world != 0 && c.key.region != 0 && c.frame != 0 &&
|
||||
persistence::valid(c.lease) && c.build != persistence::Sha256Digest{} &&
|
||||
std::ranges::find(c.retained_builds, c.build) != c.retained_builds.end();
|
||||
}
|
||||
PrimitiveHostReply reply(PrimitiveHostReplyKind kind, persistence::LifeReservation reservation,
|
||||
std::optional<persistence::PrimitiveLifeBinding> published = {}) {
|
||||
return {kind, reservation, std::move(published)};
|
||||
}
|
||||
} // namespace
|
||||
|
||||
PrimitiveRegionHost::PrimitiveRegionHost(PrimitiveHostConfig config, PrimitiveConnect connect,
|
||||
std::unique_ptr<persistence::FilesystemObjectStore> objects)
|
||||
: config_{std::move(config)}, connect_{std::move(connect)}, objects_{std::move(objects)},
|
||||
authority_{config_.lease} {}
|
||||
|
||||
PrimitiveRegionHost::~PrimitiveRegionHost() = default;
|
||||
|
||||
std::expected<std::unique_ptr<PrimitiveRegionHost>, PrimitiveHostError>
|
||||
PrimitiveRegionHost::create(PrimitiveHostConfig config, PrimitiveConnect connect,
|
||||
std::unique_ptr<persistence::FilesystemObjectStore> objects) {
|
||||
if (!valid(config) || !connect || !objects)
|
||||
return std::unexpected{PrimitiveHostError::configuration};
|
||||
return std::unique_ptr<PrimitiveRegionHost>{new PrimitiveRegionHost{
|
||||
std::move(config), std::move(connect), std::move(objects)}};
|
||||
}
|
||||
|
||||
bool PrimitiveRegionHost::database() {
|
||||
if (db_)
|
||||
return true;
|
||||
auto connection = connect_();
|
||||
if (!connection) {
|
||||
++status_.store_failures;
|
||||
return false;
|
||||
}
|
||||
db_ = std::move(*connection);
|
||||
return true;
|
||||
}
|
||||
|
||||
void PrimitiveRegionHost::acquire() {
|
||||
if (!database())
|
||||
return;
|
||||
if (!authority_.may_originate(LeaseClock::now())) {
|
||||
const auto sent = LeaseClock::now();
|
||||
auto lease = persistence::acquire_region_lease(
|
||||
*db_, config_.key, config_.build, config_.lease.duration, config_.lease.renew_timeout);
|
||||
if (!lease) {
|
||||
if (lease.error().kind == LeaseStoreError::not_provisioned ||
|
||||
lease.error().kind == LeaseStoreError::unsupported)
|
||||
status_.phase = HostPhase::quarantined;
|
||||
else if (lease.error().kind != LeaseStoreError::busy)
|
||||
db_.reset();
|
||||
return;
|
||||
}
|
||||
if (lease->frame != config_.frame ||
|
||||
!authority_.acquired(*lease, sent, LeaseClock::now())) {
|
||||
status_.phase = HostPhase::quarantined;
|
||||
return;
|
||||
}
|
||||
}
|
||||
auto recovered = persistence::recover_primitive_checkpoint(
|
||||
*db_, *objects_, config_.key, config_.retained_builds, config_.lease.commit_timeout);
|
||||
if (!recovered) {
|
||||
if (recovered.error().kind == persistence::RecoveryError::store) {
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
return;
|
||||
}
|
||||
authority_.fence();
|
||||
status_.phase = HostPhase::quarantined;
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
return;
|
||||
}
|
||||
if (recovered->manifest.frame != config_.frame ||
|
||||
recovered->manifest.tick_microseconds != 50000 ||
|
||||
recovered->manifest.profile != persistence::CheckpointProfile::primitive_lives_v1) {
|
||||
authority_.fence();
|
||||
status_.phase = HostPhase::quarantined;
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
return;
|
||||
}
|
||||
state_ = std::move(recovered->state);
|
||||
committed_ = std::move(recovered->committed);
|
||||
pending_.reset();
|
||||
status_.epoch = authority_.grant()->epoch;
|
||||
status_.committed_tick = committed_->head.end_tick;
|
||||
status_.simulated_tick = committed_->head.end_tick;
|
||||
status_.committed_sequence = committed_->head.sequence;
|
||||
status_.phase = HostPhase::running;
|
||||
status_.ready = true;
|
||||
}
|
||||
|
||||
void PrimitiveRegionHost::upkeep() {
|
||||
const auto state = authority_.state(LeaseClock::now());
|
||||
if (state == persistence::LeaseState::fenced) {
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
pending_.reset();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::acquiring;
|
||||
return;
|
||||
}
|
||||
if (state != persistence::LeaseState::renew_due || !database())
|
||||
return;
|
||||
const auto grant = *authority_.grant();
|
||||
const auto sent = LeaseClock::now();
|
||||
auto renewed = persistence::renew_region_lease(*db_, config_.key, grant, config_.lease.duration,
|
||||
config_.lease.renew_timeout);
|
||||
if (renewed) {
|
||||
if (!authority_.renewed(grant.epoch, *renewed, sent, LeaseClock::now())) {
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
pending_.reset();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::acquiring;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (renewed.error().kind == LeaseStoreError::unknown) {
|
||||
authority_.renewal_unknown(config_.lease.duration, sent);
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
} else if (renewed.error().kind == LeaseStoreError::fenced) {
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
pending_.reset();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::acquiring;
|
||||
} else {
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
}
|
||||
}
|
||||
|
||||
void PrimitiveRegionHost::refresh() {
|
||||
const auto& lease = authority_.grant();
|
||||
status_.ready = status_.phase == HostPhase::running && state_ && committed_ && lease &&
|
||||
authority_.may_originate(LeaseClock::now());
|
||||
status_.epoch = lease ? lease->epoch : 0;
|
||||
status_.lease_remaining_ms =
|
||||
status_.ready ? static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
authority_.deadline() - LeaseClock::now()).count())
|
||||
: 0;
|
||||
}
|
||||
|
||||
const HostStatus& PrimitiveRegionHost::step() {
|
||||
if (status_.phase == HostPhase::quarantined)
|
||||
return status_;
|
||||
if (state_)
|
||||
upkeep();
|
||||
if (pending_ && authority_.may_begin_commit(LeaseClock::now())) {
|
||||
status_.phase = HostPhase::running;
|
||||
status_.ready = true;
|
||||
const auto account = pending_->account;
|
||||
const auto generation = pending_->generation;
|
||||
const auto request = pending_->request;
|
||||
(void)new_life(account, generation, request, 1);
|
||||
}
|
||||
if (!state_ && status_.phase != HostPhase::quarantined)
|
||||
acquire();
|
||||
refresh();
|
||||
return status_;
|
||||
}
|
||||
|
||||
void PrimitiveRegionHost::drain() {
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::draining;
|
||||
pending_.reset();
|
||||
const auto grant = authority_.grant();
|
||||
if (grant && authority_.may_originate(LeaseClock::now()) && database()) {
|
||||
const auto released = persistence::release_region_lease(
|
||||
*db_, config_.key, *grant, config_.lease.renew_timeout);
|
||||
if (!released)
|
||||
db_.reset();
|
||||
}
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
status_.phase = HostPhase::stopped;
|
||||
status_.epoch = 0;
|
||||
status_.lease_remaining_ms = 0;
|
||||
}
|
||||
|
||||
std::optional<persistence::LeaseGrant> PrimitiveRegionHost::authority() const {
|
||||
if (!status_.ready || !authority_.may_originate(LeaseClock::now()))
|
||||
return std::nullopt;
|
||||
return authority_.grant();
|
||||
}
|
||||
|
||||
PrimitiveHostReply PrimitiveRegionHost::query(const persistence::Uuid& account,
|
||||
std::uint64_t generation,
|
||||
std::uint64_t request) {
|
||||
if (!database() || !authority_.grant())
|
||||
return reply(PrimitiveHostReplyKind::unavailable, {});
|
||||
auto outcome = persistence::query_primitive_life(*db_, config_.key, *authority_.grant(),
|
||||
account, generation, request,
|
||||
config_.lease.commit_timeout);
|
||||
if (!outcome) {
|
||||
if (outcome.error().kind == persistence::LifeReservationError::unknown ||
|
||||
outcome.error().kind == persistence::LifeReservationError::store)
|
||||
db_.reset();
|
||||
if (outcome.error().kind == persistence::LifeReservationError::fenced)
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unknown, {});
|
||||
}
|
||||
if (!*outcome)
|
||||
return reply(PrimitiveHostReplyKind::unknown, {});
|
||||
const auto& value = **outcome;
|
||||
if (value.published)
|
||||
return reply(PrimitiveHostReplyKind::born, value.reservation, value.published);
|
||||
if (value.reservation.region != config_.key.region)
|
||||
return reply(PrimitiveHostReplyKind::routed, value.reservation);
|
||||
if (value.reservation.refusal != persistence::LifeRefusal::none)
|
||||
return reply(PrimitiveHostReplyKind::refused, value.reservation);
|
||||
return reply(PrimitiveHostReplyKind::pending, value.reservation);
|
||||
}
|
||||
|
||||
PrimitiveHostReply PrimitiveRegionHost::new_life(const persistence::Uuid& account,
|
||||
std::uint64_t generation,
|
||||
std::uint64_t request,
|
||||
std::uint32_t rules) {
|
||||
if (!status_.ready || !state_ || !committed_ || !authority_.may_begin_commit(LeaseClock::now()))
|
||||
return reply(PrimitiveHostReplyKind::unavailable, {});
|
||||
if (account == persistence::Uuid{} || generation == 0 || request == 0 || rules == 0)
|
||||
return reply(PrimitiveHostReplyKind::refused, {});
|
||||
if (pending_) {
|
||||
if (pending_->account != account || pending_->generation != generation ||
|
||||
pending_->request != request)
|
||||
return reply(PrimitiveHostReplyKind::pending, pending_->reservation);
|
||||
if (!database()) {
|
||||
status_.phase = HostPhase::resolving;
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unknown, pending_->reservation);
|
||||
}
|
||||
persistence::PrimitivePreparedCommit prepared;
|
||||
if (pending_->prepared) {
|
||||
prepared = *pending_->prepared;
|
||||
} else {
|
||||
// Re-entering the adapter with the same predecessor, trace and commit recreates
|
||||
// the exact canonical bytes when the earlier failure happened before SQL.
|
||||
auto published = persistence::publish_primitive_batch(
|
||||
*db_, *objects_, *authority_.grant(), *committed_, pending_->batch,
|
||||
*pending_->candidate, pending_->commit, config_.build,
|
||||
config_.lease.commit_timeout);
|
||||
if (published) {
|
||||
state_ = std::move(pending_->candidate);
|
||||
committed_ = published->committed;
|
||||
pending_.reset();
|
||||
++status_.commits;
|
||||
status_.committed_sequence = committed_->head.sequence;
|
||||
status_.committed_tick = committed_->head.end_tick;
|
||||
status_.simulated_tick = committed_->head.end_tick;
|
||||
return query(account, generation, request);
|
||||
}
|
||||
if (published.error().kind == persistence::PublicationError::unknown) {
|
||||
pending_->prepared = published.error().prepared;
|
||||
db_.reset();
|
||||
++status_.unknown_outcomes;
|
||||
status_.phase = HostPhase::resolving;
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unknown, pending_->reservation);
|
||||
}
|
||||
if (published.error().kind == persistence::PublicationError::rejected &&
|
||||
published.error().store.kind == LeaseStoreError::fenced) {
|
||||
authority_.fence();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::acquiring;
|
||||
return reply(PrimitiveHostReplyKind::unknown, pending_->reservation);
|
||||
}
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
const auto reservation = pending_->reservation;
|
||||
pending_.reset();
|
||||
authority_.fence();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::quarantined;
|
||||
return reply(PrimitiveHostReplyKind::unavailable, reservation);
|
||||
}
|
||||
auto resolved = persistence::retry_primitive_prepared(
|
||||
*db_, prepared, config_.lease.commit_timeout);
|
||||
if (!resolved) {
|
||||
if (resolved.error().kind == persistence::PublicationError::unknown) {
|
||||
db_.reset();
|
||||
++status_.store_failures;
|
||||
status_.phase = HostPhase::resolving;
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unknown, pending_->reservation);
|
||||
}
|
||||
const auto reservation = pending_->reservation;
|
||||
pending_.reset();
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, reservation);
|
||||
}
|
||||
state_ = std::move(pending_->candidate);
|
||||
committed_ = std::move(*resolved);
|
||||
pending_.reset();
|
||||
++status_.commits;
|
||||
status_.phase = HostPhase::running;
|
||||
status_.committed_sequence = committed_->head.sequence;
|
||||
status_.committed_tick = committed_->head.end_tick;
|
||||
status_.simulated_tick = committed_->head.end_tick;
|
||||
refresh();
|
||||
return query(account, generation, request);
|
||||
}
|
||||
|
||||
if (!database())
|
||||
return reply(PrimitiveHostReplyKind::unavailable, {});
|
||||
auto existing = query(account, generation, request);
|
||||
if (existing.kind != PrimitiveHostReplyKind::unknown &&
|
||||
existing.kind != PrimitiveHostReplyKind::pending)
|
||||
return existing;
|
||||
auto reserved = persistence::reserve_primitive_life(
|
||||
*db_, config_.key, *authority_.grant(), account, generation, request, rules,
|
||||
config_.lease.commit_timeout);
|
||||
if (!reserved) {
|
||||
if (reserved.error().kind == persistence::LifeReservationError::unknown) {
|
||||
db_.reset();
|
||||
++status_.unknown_outcomes;
|
||||
return reply(PrimitiveHostReplyKind::unknown, {});
|
||||
}
|
||||
if (reserved.error().kind == persistence::LifeReservationError::existing_life)
|
||||
return query(account, generation, request);
|
||||
if (reserved.error().kind == persistence::LifeReservationError::fenced) {
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
}
|
||||
return reply(PrimitiveHostReplyKind::unavailable, {});
|
||||
}
|
||||
if (reserved->region != config_.key.region)
|
||||
return reply(PrimitiveHostReplyKind::routed, *reserved);
|
||||
if (reserved->refusal != persistence::LifeRefusal::none)
|
||||
return reply(PrimitiveHostReplyKind::refused, *reserved);
|
||||
const auto refuse = [&](persistence::LifeRefusal reason) {
|
||||
const auto recorded = persistence::reject_primitive_life(
|
||||
*db_, config_.key, *authority_.grant(), account, generation, request, reason,
|
||||
config_.lease.commit_timeout);
|
||||
if (!recorded) {
|
||||
if (recorded.error().kind == persistence::LifeReservationError::unknown)
|
||||
db_.reset();
|
||||
return reply(PrimitiveHostReplyKind::unknown, *reserved);
|
||||
}
|
||||
auto final = *reserved;
|
||||
final.refusal = *recorded;
|
||||
return reply(PrimitiveHostReplyKind::refused, final);
|
||||
};
|
||||
if (state_->lives().size() >= max_primitive_lives ||
|
||||
state_->lives().size() >= state_->config().sites.size())
|
||||
return refuse(persistence::LifeRefusal::no_placement);
|
||||
const auto endowed = biology::primitive_endowment();
|
||||
if (!endowed || state_->balances().empty()) {
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
auto& source = state_->balances().front();
|
||||
const auto enough = [&] {
|
||||
if (source.energy.value < endowed->body.energy.value + endowed->reserve.energy.value +
|
||||
endowed->construction.value)
|
||||
return false;
|
||||
for (std::size_t i = 0; i < source.matter.size(); ++i)
|
||||
if (source.matter[i].value < endowed->body.matter[i].value)
|
||||
return false;
|
||||
return true;
|
||||
};
|
||||
if (!enough())
|
||||
return refuse(persistence::LifeRefusal::insufficient_funding);
|
||||
if (committed_->head.end_tick == UINT64_MAX ||
|
||||
committed_->head.sequence == UINT64_MAX) {
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
const auto decoded = persistence::decode_manifest(committed_->manifest);
|
||||
const auto predecessor_digest = persistence::sha256(committed_->manifest);
|
||||
if (!decoded || !predecessor_digest) {
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
const auto end_tick = committed_->head.end_tick + 1;
|
||||
PrimitiveBatch batch{{config_.key.world, config_.key.region, config_.frame,
|
||||
committed_->head.commit, *predecessor_digest,
|
||||
committed_->head.end_tick, end_tick},
|
||||
{{{account, request}, reserved->identity, generation, {end_tick}}}};
|
||||
auto candidate = replay_primitive_batch(*state_, batch);
|
||||
if (!candidate) {
|
||||
if (candidate.error() == PrimitiveBatchError::state) {
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
authority_.fence();
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
auto commit = authority_.grant()->holder;
|
||||
const auto sequence = committed_->head.sequence + 1;
|
||||
for (unsigned i = 0; i < 8; ++i)
|
||||
commit[8 + i] = static_cast<std::uint8_t>((sequence >> (i * 8U)) & 0xffU);
|
||||
pending_ = std::make_unique<Pending>(Pending{account, generation, request, *reserved,
|
||||
std::move(batch), std::move(*candidate), commit, {}});
|
||||
auto published = persistence::publish_primitive_batch(
|
||||
*db_, *objects_, *authority_.grant(), *committed_, pending_->batch, *pending_->candidate,
|
||||
pending_->commit, config_.build, config_.lease.commit_timeout);
|
||||
if (!published) {
|
||||
const auto& failure = published.error();
|
||||
if (failure.kind == persistence::PublicationError::unknown) {
|
||||
pending_->prepared = failure.prepared;
|
||||
db_.reset();
|
||||
++status_.unknown_outcomes;
|
||||
status_.phase = HostPhase::resolving;
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unknown, *reserved);
|
||||
}
|
||||
if (failure.kind == persistence::PublicationError::rejected &&
|
||||
failure.store.kind == LeaseStoreError::fenced) {
|
||||
authority_.fence();
|
||||
state_.reset();
|
||||
committed_.reset();
|
||||
pending_.reset();
|
||||
status_.ready = false;
|
||||
status_.phase = HostPhase::acquiring;
|
||||
return reply(PrimitiveHostReplyKind::unknown, *reserved);
|
||||
}
|
||||
++status_.store_failures;
|
||||
if (failure.kind == persistence::PublicationError::candidate ||
|
||||
failure.kind == persistence::PublicationError::predecessor ||
|
||||
failure.kind == persistence::PublicationError::receipt ||
|
||||
failure.kind == persistence::PublicationError::rejected) {
|
||||
pending_.reset();
|
||||
authority_.fence();
|
||||
status_.phase = HostPhase::quarantined;
|
||||
status_.ready = false;
|
||||
return reply(PrimitiveHostReplyKind::unavailable, *reserved);
|
||||
}
|
||||
return reply(PrimitiveHostReplyKind::pending, *reserved);
|
||||
}
|
||||
state_ = std::move(pending_->candidate);
|
||||
committed_ = published->committed;
|
||||
pending_.reset();
|
||||
++status_.commits;
|
||||
status_.committed_sequence = committed_->head.sequence;
|
||||
status_.committed_tick = committed_->head.end_tick;
|
||||
status_.simulated_tick = committed_->head.end_tick;
|
||||
return query(account, generation, request);
|
||||
}
|
||||
} // namespace evo::server
|
||||
@@ -5,6 +5,7 @@
|
||||
#include "evo/persistence/region_checkpoint.hpp"
|
||||
#include "evo/persistence/region_lease_store.hpp"
|
||||
#include "evo/persistence/sha256.hpp"
|
||||
#include "evo/server/primitive_region_host.hpp"
|
||||
#include <cstdlib>
|
||||
#include <source_location>
|
||||
#include <stdexcept>
|
||||
@@ -106,6 +107,54 @@ void adapter_publication_and_recovery() {
|
||||
recovered->state->snapshot() == (*candidate)->snapshot());
|
||||
CHECK(release_region_lease(*db, key, *successor_lease, 5s));
|
||||
}
|
||||
|
||||
void primitive_host_fenced_spawn() {
|
||||
auto owner = connect_as(env("PGUSER"));
|
||||
const RegionKey key{9755, 1};
|
||||
const auto build = id<32>(0xD8);
|
||||
CHECK(owner->query("INSERT INTO evolution.retained_builds(build_identity) VALUES "
|
||||
"(decode(repeat('d8',32),'hex')) ON CONFLICT DO NOTHING",
|
||||
{}, 5s));
|
||||
const auto endowed = evo::biology::primitive_endowment();
|
||||
CHECK(endowed);
|
||||
auto funding = endowed->body;
|
||||
funding.energy.value = endowed->reserve.energy.value + endowed->construction.value;
|
||||
auto opening = PrimitiveLives::create({key.world, key.region, 1, 1, funding, {{0, 0}}});
|
||||
CHECK(opening);
|
||||
auto objects = FilesystemObjectStore::open(env("PGHOST"), key, max_primitive_lives_bytes);
|
||||
CHECK(objects);
|
||||
CHECK(provision_primitive_region(*owner, **objects, key, 1, **opening, test_commit(20), build,
|
||||
5s));
|
||||
CHECK(owner->query("INSERT INTO evolution.life_allocators VALUES(9755,1)", {}, 5s));
|
||||
|
||||
auto host_objects = FilesystemObjectStore::open(env("PGHOST"), key,
|
||||
max_primitive_lives_bytes);
|
||||
CHECK(host_objects);
|
||||
const auto pg = PgOptions{env("PGHOST"), {}, env("PGPORT"), env("PGDATABASE"),
|
||||
env("EVO_PG_RUNTIME_USER"), {}, {}};
|
||||
auto made = PrimitiveRegionHost::create(
|
||||
{key, 1, {}, build, {build}},
|
||||
[pg] { return Postgres::connect(pg, 5s); }, std::move(*host_objects));
|
||||
CHECK(made && (*made)->step().ready);
|
||||
auto runtime = connect_as(env("EVO_PG_RUNTIME_USER"));
|
||||
const auto account = id<16>(0x73);
|
||||
auto control = claim_account_control(*runtime, key, *(*made)->authority(), account, 1, 1,
|
||||
20000ms, 5s);
|
||||
CHECK(control);
|
||||
const auto born = (*made)->new_life(account, control->generation, 1, 1);
|
||||
CHECK(born.kind == PrimitiveHostReplyKind::born && born.published &&
|
||||
born.published->identity == 1 && born.published->born == 1 &&
|
||||
born.published->commit_sequence == 1);
|
||||
CHECK((*made)->status().ready && (*made)->state()->lives().size() == 1 &&
|
||||
(*made)->status().committed_tick == 1);
|
||||
const auto exact = (*made)->new_life(account, control->generation, 1, 1);
|
||||
CHECK(exact == born);
|
||||
const auto second = (*made)->new_life(account, control->generation, 2, 1);
|
||||
CHECK(second.kind == PrimitiveHostReplyKind::refused &&
|
||||
second.reservation.refusal == LifeRefusal::existing_life);
|
||||
(*made)->drain();
|
||||
CHECK((*made)->status().phase == HostPhase::stopped && !(*made)->status().ready);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_primitive_publication_tests() {
|
||||
@@ -286,4 +335,5 @@ void run_primitive_publication_tests() {
|
||||
// Exact committed retry survives loss of authority, but cannot originate work.
|
||||
CHECK(publish(*manifest, *trace));
|
||||
adapter_publication_and_recovery();
|
||||
primitive_host_fenced_spawn();
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user