Add M10.4b durable action records (schema v11)
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m47s
Foundation / checks (network) (push) Successful in 9m30s
Foundation / checks (client) (push) Successful in 12m19s
Foundation / checks (tsan) (push) Successful in 7m43s
Foundation / checks (checks) (push) Successful in 31m29s
Foundation / checks (postgres) (push) Successful in 36m39s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m15s
Foundation / server-image (push) Successful in 20m37s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m47s
Foundation / checks (network) (push) Successful in 9m30s
Foundation / checks (client) (push) Successful in 12m19s
Foundation / checks (tsan) (push) Successful in 7m43s
Foundation / checks (checks) (push) Successful in 31m29s
Foundation / checks (postgres) (push) Successful in 36m39s
Foundation / postgres-tools (push) Successful in 19s
Foundation / image (push) Successful in 2m15s
Foundation / server-image (push) Successful in 20m37s
Each batch's actions are recorded in the same statement as its checkpoint: one region_actions row per (world, account, action id) and the block in an append-only table, compared on exact retries and resolution. The region host attaches actions to the ticks they ride and reports acknowledgements and drops, keeping unknown-outcome batches unresolved. Outcomes come from the durable record via ActionLedger, so crashes before or after a commit, unknown publications and cross-region moves all resolve to one answer; a repeated id keeps its first record instead of stalling a region. ADR-0092. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+5
-3
@@ -220,8 +220,10 @@ if(EVO_BUILD_POSTGRES)
|
||||
endif()
|
||||
evo_owned_target(evo-server)
|
||||
if(EVO_BUILD_NETWORK)
|
||||
# M10.3c durable account control for region sessions (ADR-0091, schema v10).
|
||||
add_library(evo_control_ledger_postgres STATIC src/server/session/postgres_control_ledger.cpp)
|
||||
# M10.3c durable account control and M10.4b durable action records for region sessions
|
||||
# (ADR-0091 schema v10, ADR-0092 schema v11).
|
||||
add_library(evo_control_ledger_postgres STATIC src/server/session/postgres_control_ledger.cpp
|
||||
src/server/session/postgres_action_ledger.cpp)
|
||||
target_link_libraries(evo_control_ledger_postgres PUBLIC evo::postgres evo_network_protocol)
|
||||
evo_owned_target(evo_control_ledger_postgres)
|
||||
endif()
|
||||
@@ -232,7 +234,7 @@ if(EVO_BUILD_POSTGRES)
|
||||
if(BUILD_TESTING)
|
||||
# Contention tests use one thread and one connection per contender.
|
||||
add_executable(evo-postgres-tests tests/postgres_tests.cpp tests/region_lease_store_tests.cpp
|
||||
tests/account_control_store_tests.cpp
|
||||
tests/account_control_store_tests.cpp tests/region_action_tests.cpp
|
||||
tests/region_checkpoint_store_tests.cpp tests/region_host_tests.cpp
|
||||
tests/region_server_process_tests.cpp tests/region_recovery_tests.cpp
|
||||
tests/region_ops_tests.cpp tests/region_clone_tests.cpp
|
||||
|
||||
+3
-2
@@ -522,8 +522,9 @@ hardware (10.9). 10.2 split: 10.2a selection, pins and build of msquic/OpenSSL d
|
||||
10.3 split (ADR-0091): 10.3a account ids, ECDSA grants, identity adapter and region
|
||||
session control done; 10.3b QUIC session login (schemas/network/session-v1.md) done;
|
||||
10.3c durable per-world account control (schema v10) done. 10.4 split (ADR-0092):
|
||||
10.4a intention/action protocol and admission done; 10.4b durable action records and
|
||||
10.4c evo-server wiring with a headless client follow. 10.5-10.9 planned.
|
||||
10.4a intention/action protocol and admission done; 10.4b durable action records
|
||||
(schema v11, same transaction as the checkpoint) done; 10.4c evo-server wiring with a
|
||||
headless client follows. 10.5-10.9 planned.
|
||||
|
||||
**Entry:** M9 client/session evidence and M7/M8 dev/stage durability. Read
|
||||
NETWORK_ARCHITECTURE, PERSISTENCE_ARCHITECTURE and ADR-0017. The M9 transport is
|
||||
|
||||
+5
-2
@@ -28,8 +28,11 @@ control is fenced or cannot be renewed. M10.4a (ADR-0092) adds
|
||||
intentions (datagram receipts and a cumulative simulated watermark) and account-scoped
|
||||
actions with stable ids, receipts and final outcomes to session-v1, with admission
|
||||
(window, rate, sequence, duplicates, unresolved bound) and client support; intentions and
|
||||
actions have no biological effect yet. Next: M10.4b durable action records (migration
|
||||
v11) and M10.4c evo-server wiring.
|
||||
actions have no biological effect yet. M10.4b makes
|
||||
action outcomes durable: schema v11 records each batch's actions in the same statement as
|
||||
its checkpoint, the host attaches actions to the ticks they ride, and outcomes are read
|
||||
from the durable record after crashes before or after a commit, unknown publications and
|
||||
cross-region moves. Next: M10.4c evo-server wiring with a headless client.
|
||||
|
||||
## Previous gate - 2026-10-02
|
||||
|
||||
|
||||
@@ -59,3 +59,33 @@ the wire field must never be cast from it).
|
||||
- The simulated watermark says the targeted tick ran while the intention was scheduled;
|
||||
with no movement rule it changed nothing. 10.6 gives intentions an effect.
|
||||
- Nothing is wired into evo-server yet (10.4c); nothing deploys M10 sessions.
|
||||
|
||||
## M10.4b durable action records (2026-10-03)
|
||||
Schema v11 (`migrations/0011_region_actions.sql`, [region-actions-v1](../../schemas/region-actions-v1.md)):
|
||||
- **Same transaction as the checkpoint.** The region host attaches each submitted action to
|
||||
the first simulated tick at or after its scheduled tick and publishes the batch's action
|
||||
block with the three-argument `publish_region_checkpoint`; the records exist exactly when
|
||||
the checkpoint does. Committed history stays immutable: blocks live in their own
|
||||
append-only table, and an exact retry or resolution must repeat the block.
|
||||
- **Outcomes come from the record.** `find_region_action` (under the caller's exact region
|
||||
lease) returns the committed record of (world, account, action id). Found: final
|
||||
`committed_no_effect` with that record's tick and commit. Absent once this authority is
|
||||
recovered and ready: never committed by any earlier epoch (its commits are final).
|
||||
- **One record per id, without stalling a region.** Found during testing: an account that
|
||||
moves regions while its action is pending in the old one can have the same id admitted by
|
||||
the new region (the record does not exist yet). Refusing the second commit would have
|
||||
quarantined a healthy region. The database instead keeps the first record and lets the
|
||||
later batch commit; `IntentAdmission::committed` reports each outcome from the record, so
|
||||
every region answers the same tick and commit for an id.
|
||||
- **Host semantics.** `submit_action` (bounded: 256 queued plus riding), `ActionAck` per
|
||||
committed action, `DroppedAction` with `final` = certainly never committed. Actions of a
|
||||
batch whose publication outcome was unknown when authority was lost are not final; the
|
||||
session layer keeps them pending until the record decides.
|
||||
- **Session layer.** `ActionLedger` (interface) with `PostgresActionLedger`; admission
|
||||
checks the record before admitting an id not in memory (`duplicate` with its outcome, or
|
||||
`not_ready` when the record cannot be read); queries resolve unknown-outcome and evicted
|
||||
actions from the record once ready. `RegionSessions::committed/dropped` take the host's
|
||||
reports.
|
||||
- Limits: action rows are retained (no pruning policy yet); clones (ADR-0067) do not copy
|
||||
them; the two-argument publication of a commit that has an action block would still
|
||||
return its receipt (the adapter never does this).
|
||||
|
||||
@@ -4237,3 +4237,42 @@ tests; format check; Windows MSVC 19.44 preset `quic` build and CTest 4/4 (binar
|
||||
timestamps checked).
|
||||
Not done: durable action records and crash resolution (10.4b), evo-server wiring and
|
||||
process kill tests (10.4c).
|
||||
|
||||
## 2026-10-03 — M10.4b durable action records (ADR-0092, schema v11)
|
||||
|
||||
Implemented migration v11 (`region_actions`, `region_commit_actions`, `check_action_block`,
|
||||
`commit_actions`, three-argument publication, six-argument resolution,
|
||||
`find_region_action`; apply chain and audit), `encode_actions`, the store overloads,
|
||||
actions in `PreparedCommit`/publication/resolution, host `submit_action` with
|
||||
`ActionAck`/`DroppedAction`, `ActionLedger` in admission and `PostgresActionLedger`.
|
||||
Database tests (`tests/region_action_tests.cpp`, regions 7/131-135, commit ids tagged 0xA7):
|
||||
publication with actions and lookups (including another account's and an absent id);
|
||||
exact retry and resolution with the block; a changed block conflicting (retry and
|
||||
resolution); nine invalid blocks refused (tick outside, tick 0, kind, zero account, zero id,
|
||||
unsorted, duplicate, one id at two ticks, wrong length) and the encoder refusing the same;
|
||||
a later batch carrying an already recorded id committing with the first record kept; a
|
||||
batch without actions; lookups fenced with an old lease; the PostgreSQL action ledger.
|
||||
Host: acknowledgement with the tick each action rode, late action riding the next tick,
|
||||
duplicate, invalid and full submissions, not ready; crash before the transaction (successor
|
||||
finds no record), crash after the commit before any acknowledgement (record durable,
|
||||
successor at commit 1); unknown publication outcome resolved by exact bytes including the
|
||||
actions. Unit tests: duplicate from the record, unavailable record refusing admission,
|
||||
refused attempts costing no lookup, exact commit tick, final and unknown drops, unknown
|
||||
decided once ready (committed or not committed), evicted outcome found on record.
|
||||
`postgres-check.sh`: history 1..11, future-version probe 12, v11 checksum drift, audit
|
||||
refusing SELECT on both new tables and EXECUTE on `check_action_block`.
|
||||
|
||||
Found and fixed during development: updating `region_commits` with the block was refused by
|
||||
the immutable-history trigger (blocks now live in their own append-only table); a repeated
|
||||
action id would have failed the later commit and quarantined a healthy region (now the first
|
||||
record is kept and outcomes come from it); the encoder and SQL accepted one id at two ticks
|
||||
in one block; test fixtures colliding with other tests' world, commit ids and accounts.
|
||||
|
||||
Executed: Linux GCC 14 `tools/ci/postgres-check.sh` complete (18 integration and rehearsal
|
||||
suites, migration and audit probes); clang-tidy 19 on the changed persistence, host,
|
||||
ledger, session and test sources (Clang builds with network+PostgreSQL and with quic);
|
||||
format check; GCC preset `quic` CTest 4/4, session tests 10/10 repeated; Clang 19 ASan/UBSan
|
||||
session and network tests; Windows MSVC 19.44 preset `quic` build and CTest 4/4 (binaries
|
||||
rebuilt, timestamps checked).
|
||||
Not done: evo-server wiring, headless client and process kill tests with real sessions
|
||||
(10.4c); ASan of the PostgreSQL suite; a real two-region move with a pending action.
|
||||
|
||||
@@ -190,6 +190,16 @@ renewal requires the same generation, region and epoch and an unexpired control
|
||||
(`EVL12` otherwise). The privilege audit includes all three functions. Lock order: upgrade
|
||||
lock (shared), region lease row, account control row.
|
||||
|
||||
## Action records (schema v11, ADR-0092)
|
||||
|
||||
`publish_region_checkpoint(manifest, decisions, actions)` records a batch's action block
|
||||
([region-actions-v1](../../schemas/region-actions-v1.md)) in the same statement as its
|
||||
checkpoint: one `evolution.region_actions` row per new (world, account, action id), and the
|
||||
block in the append-only `region_commit_actions` (committed history itself is never
|
||||
updated). Exact retries and `resolve_region_commit(..., actions)` compare the block.
|
||||
`find_region_action` reads one record under the caller's exact region lease. The privilege
|
||||
audit covers all three functions; the tables and `check_action_block` are owner-only.
|
||||
|
||||
## Host lease discipline
|
||||
|
||||
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
#include <deque>
|
||||
#include <expected>
|
||||
#include <optional>
|
||||
#include <span>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
@@ -45,11 +46,32 @@ struct AccountOutcome {
|
||||
ActionOutcome outcome;
|
||||
};
|
||||
enum class IntentConfigError : std::uint8_t { invalid };
|
||||
// Durable action records (10.4b): the region host implements it over its lease (schema v11);
|
||||
// tests use fakes. `find` answers with the committed outcome, nothing when the action was
|
||||
// never committed, or `unavailable` when it cannot decide.
|
||||
enum class ActionLedgerError : std::uint8_t { unavailable };
|
||||
class ActionLedger {
|
||||
public:
|
||||
ActionLedger() = default;
|
||||
ActionLedger(const ActionLedger&) = delete;
|
||||
ActionLedger& operator=(const ActionLedger&) = delete;
|
||||
virtual ~ActionLedger() = default;
|
||||
[[nodiscard]] virtual std::expected<std::optional<ActionOutcome>, ActionLedgerError>
|
||||
find(const AccountId& account, std::uint64_t action_id) = 0;
|
||||
|
||||
protected:
|
||||
ActionLedger(ActionLedger&&) = default;
|
||||
ActionLedger& operator=(ActionLedger&&) = default;
|
||||
};
|
||||
|
||||
class IntentAdmission {
|
||||
public:
|
||||
[[nodiscard]] static std::expected<IntentAdmission, IntentConfigError>
|
||||
create(IntentLimits limits);
|
||||
// Durable lookups for duplicates and unresolved actions; null keeps memory only.
|
||||
void attach(ActionLedger* ledger) noexcept {
|
||||
ledger_ = ledger;
|
||||
}
|
||||
// Session lifecycle, driven by session control (after Welcome, on any close).
|
||||
[[nodiscard]] bool open(std::uint64_t session, const AccountId& account);
|
||||
void close(std::uint64_t session);
|
||||
@@ -59,18 +81,26 @@ class IntentAdmission {
|
||||
const AdmissionFacts& facts);
|
||||
[[nodiscard]] ActionReceipt action(std::uint64_t session, const SessionAction& m,
|
||||
const AdmissionFacts& facts);
|
||||
// What this process knows about an action of the session's account.
|
||||
[[nodiscard]] ActionOutcome query(std::uint64_t session, const ActionQuery& m) const;
|
||||
// An action of the session's account: final, pending or unknown. With a ledger, an action
|
||||
// whose batch outcome was unknown is decided from its durable record once this authority
|
||||
// is ready, and an action no longer in memory is looked up.
|
||||
[[nodiscard]] ActionOutcome query(std::uint64_t session, const ActionQuery& m,
|
||||
const AdmissionFacts& facts);
|
||||
// Watermarks to send after the host simulated through facts.server_tick.
|
||||
[[nodiscard]] std::vector<std::pair<std::uint64_t, SimulatedThrough>>
|
||||
simulated(const AdmissionFacts& facts);
|
||||
// Scheduled actions whose tick is <= `tick`: they ride that tick's batch from now on.
|
||||
[[nodiscard]] std::vector<CarriedAction> carry(std::uint64_t tick);
|
||||
// A commit covering ticks up to `end_tick` is durable: its carried actions are final.
|
||||
[[nodiscard]] std::vector<AccountOutcome> committed(std::uint64_t end_tick,
|
||||
// These carried actions are durable in `commit_sequence`; `tick` is the tick each rode.
|
||||
// With a ledger, each outcome is taken from the action's durable record.
|
||||
[[nodiscard]] std::vector<AccountOutcome> committed(std::span<const CarriedAction> actions,
|
||||
std::uint64_t commit_sequence);
|
||||
// The host dropped these carried actions. Final: never committed. Otherwise their batch's
|
||||
// outcome was unknown: they stay pending until their durable record decides them.
|
||||
[[nodiscard]] std::vector<AccountOutcome> dropped(std::span<const CarriedAction> actions,
|
||||
bool final);
|
||||
// Authority lost or epoch advanced: actions never carried are final (not_committed);
|
||||
// carried ones stay unresolved until a durable record decides them (10.4b).
|
||||
// carried ones stay pending until committed, dropped or decided by their record.
|
||||
[[nodiscard]] std::vector<AccountOutcome> abandon();
|
||||
// The current session of an account, for delivering outcomes.
|
||||
[[nodiscard]] std::optional<std::uint64_t> session_of(const AccountId& account) const;
|
||||
@@ -92,8 +122,16 @@ class IntentAdmission {
|
||||
struct Action {
|
||||
CarriedAction carried;
|
||||
Stage stage{};
|
||||
bool unknown{}; // Carried in a batch whose outcome was unknown when authority was lost.
|
||||
};
|
||||
explicit IntentAdmission(IntentLimits limits) : limits_{limits} {}
|
||||
// A ledger answer flattened: unavailable (or no ledger), absent, or committed.
|
||||
enum class Record : std::uint8_t { unavailable, absent, committed };
|
||||
struct Found {
|
||||
Record state{Record::unavailable};
|
||||
ActionOutcome outcome;
|
||||
};
|
||||
[[nodiscard]] Found lookup(const AccountId& account, std::uint64_t action_id) const;
|
||||
Session* find(std::uint64_t session);
|
||||
const Session* find(std::uint64_t session) const;
|
||||
void count_window(Session& s, std::uint64_t server_tick) const;
|
||||
@@ -103,5 +141,6 @@ class IntentAdmission {
|
||||
std::vector<Session> sessions_;
|
||||
std::vector<Action> actions_;
|
||||
std::deque<AccountOutcome> finals_;
|
||||
ActionLedger* ledger_{};
|
||||
};
|
||||
} // namespace evo::network
|
||||
|
||||
@@ -34,6 +34,7 @@ struct PreparedCommit {
|
||||
Uuid commit{};
|
||||
std::vector<std::uint8_t> manifest;
|
||||
std::vector<std::uint8_t> decisions;
|
||||
std::vector<std::uint8_t> actions; // Schema v11 block; empty: published without actions.
|
||||
};
|
||||
struct PublicationFailure {
|
||||
PublicationError kind{};
|
||||
@@ -92,7 +93,8 @@ publish_environmental_batch(Postgres& db, const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::EnvironmentalCommunity>& batch,
|
||||
Uuid commit, const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout, const PublishHooks& hooks = {});
|
||||
std::chrono::milliseconds timeout, const PublishHooks& hooks = {},
|
||||
std::span<const ActionRecord> actions = {});
|
||||
// Loads and fully verifies the committed head before any readiness (ADR-0065):
|
||||
// manifest digest and chain fields, retained build, payload length/SHA-256 and full
|
||||
// restore; then the head's tail: its trace object binds to the committed
|
||||
@@ -119,7 +121,8 @@ publish_lineage_batch(Postgres& db, const FilesystemObjectStore& objects, const
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::LineageRegion>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout, const PublishHooks& hooks = {});
|
||||
std::chrono::milliseconds timeout, const PublishHooks& hooks = {},
|
||||
std::span<const ActionRecord> actions = {});
|
||||
[[nodiscard]] std::expected<RecoveredLineageCheckpoint, RecoveryFailure>
|
||||
recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
|
||||
@@ -57,6 +57,22 @@ inline constexpr std::size_t max_batch_decisions = 8;
|
||||
return trace_header_bytes + (static_cast<std::size_t>(ticks) * trace_step_bytes);
|
||||
}
|
||||
|
||||
// M10.4b durable action records (schema v11, ADR-0092): the actions that rode a batch,
|
||||
// committed in the same statement as its checkpoint. Not part of the trace or manifest.
|
||||
inline constexpr std::size_t action_record_bytes = 34;
|
||||
inline constexpr std::size_t max_batch_actions = 1024;
|
||||
struct ActionRecord {
|
||||
Uuid account{};
|
||||
std::uint64_t action_id{}, tick{};
|
||||
std::uint16_t kind{1}; // 1 = probe (no effect).
|
||||
friend bool operator==(const ActionRecord&, const ActionRecord&) = default;
|
||||
};
|
||||
// Canonical block: sorted by (tick, account, id); refuses zero accounts or ids, other kinds,
|
||||
// ticks outside (begin, end], duplicates and more than max_batch_actions records.
|
||||
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_actions(std::span<const ActionRecord> actions, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick);
|
||||
|
||||
// One step per tick, begin_tick+1..end_tick, at most 64.
|
||||
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_trace(const TraceHeader& header, std::span<const TraceStep> steps);
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <optional>
|
||||
#include <span>
|
||||
#include <vector>
|
||||
|
||||
@@ -67,6 +68,14 @@ publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::chrono::milliseconds timeout);
|
||||
|
||||
// Schema v11: the same publication with the batch's action block (encode_actions), recorded
|
||||
// in the same statement. The block is part of the commit identity. An action id already on
|
||||
// record in the world keeps its first record (find_region_action is the authority).
|
||||
[[nodiscard]] std::expected<RegionHead, LeaseStoreFailure>
|
||||
publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::span<const std::uint8_t> actions, std::chrono::milliseconds timeout);
|
||||
|
||||
enum class Resolution : std::uint8_t {
|
||||
committed, // Exact bytes are durable; receipt holds the head. Grants no authority.
|
||||
absent, // Not committed now; proves nothing about a request still in flight.
|
||||
@@ -81,6 +90,22 @@ resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions, std::chrono::milliseconds timeout);
|
||||
|
||||
// Schema v11 resolution, comparing the action block too.
|
||||
[[nodiscard]] std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::span<const std::uint8_t> actions, std::chrono::milliseconds timeout);
|
||||
// The committed record of an account's action in the lease's world (schema v11), under the
|
||||
// caller's exact lease. Nothing when the action was never committed.
|
||||
struct CommittedAction {
|
||||
std::uint64_t region{}, tick{}, commit_sequence{};
|
||||
friend bool operator==(const CommittedAction&, const CommittedAction&) = default;
|
||||
};
|
||||
[[nodiscard]] std::expected<std::optional<CommittedAction>, LeaseStoreFailure>
|
||||
find_region_action(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
|
||||
std::uint64_t action_id, std::chrono::milliseconds timeout);
|
||||
|
||||
struct CommittedDecision {
|
||||
std::uint64_t commit_sequence{};
|
||||
BatchDecision decision{};
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
#pragma once
|
||||
#include "evo/network/intent_admission.hpp"
|
||||
#include "evo/persistence/region_lease_store.hpp"
|
||||
#include <chrono>
|
||||
#include <functional>
|
||||
#include <optional>
|
||||
#include <utility>
|
||||
|
||||
// M10.4b durable action records over the region's PostgreSQL lease (ADR-0092, schema v11).
|
||||
// Linux only. Owned by the region host's thread, on the host's own connection. `lease`
|
||||
// returns the host's current region lease, or nothing while it holds none; every lookup
|
||||
// then reports `unavailable`, as does any store failure.
|
||||
namespace evo::server {
|
||||
class PostgresActionLedger final : public network::ActionLedger {
|
||||
public:
|
||||
using LeaseSource = std::function<std::optional<persistence::LeaseGrant>()>;
|
||||
PostgresActionLedger(persistence::Postgres& db, persistence::RegionKey key, LeaseSource lease,
|
||||
std::chrono::milliseconds timeout = std::chrono::milliseconds{2000})
|
||||
: db_{&db}, key_{key}, lease_{std::move(lease)}, timeout_{timeout} {}
|
||||
[[nodiscard]] std::expected<std::optional<network::ActionOutcome>, network::ActionLedgerError>
|
||||
find(const network::AccountId& account, std::uint64_t action_id) override;
|
||||
|
||||
private:
|
||||
persistence::Postgres* db_;
|
||||
persistence::RegionKey key_;
|
||||
LeaseSource lease_;
|
||||
std::chrono::milliseconds timeout_;
|
||||
};
|
||||
} // namespace evo::server
|
||||
@@ -50,6 +50,20 @@ struct Dropped {
|
||||
TickInput input{};
|
||||
Unacknowledged reason{};
|
||||
};
|
||||
// M10.4b (ADR-0092): an action is durable with the batch that simulated its tick.
|
||||
struct ActionAck {
|
||||
persistence::ActionRecord record{}; // `tick` is the tick it rode.
|
||||
std::uint64_t commit_sequence{};
|
||||
friend bool operator==(const ActionAck&, const ActionAck&) = default;
|
||||
};
|
||||
// `final`: certainly never committed. Otherwise its batch's publication outcome was unknown
|
||||
// when authority was lost; the durable record decides (find_region_action) once a later
|
||||
// authority has recovered.
|
||||
struct DroppedAction {
|
||||
persistence::ActionRecord record{};
|
||||
bool final{};
|
||||
Unacknowledged reason{};
|
||||
};
|
||||
|
||||
// Commit boundaries for fault injection (M7.7 kill tests).
|
||||
enum class CrashPoint : std::uint8_t {
|
||||
@@ -68,6 +82,7 @@ struct HostConfig {
|
||||
std::chrono::milliseconds flush_interval{1000}; // Commit a non-empty batch this often.
|
||||
std::size_t max_queued_inputs{64};
|
||||
std::size_t max_queued_bytes{65536};
|
||||
std::size_t max_queued_actions{256}; // Queued plus riding the open batch.
|
||||
std::uint32_t max_catch_up_ticks{8}; // Per step; debt is never skipped.
|
||||
persistence::Sha256Digest build{};
|
||||
std::vector<persistence::Sha256Digest> retained_builds;
|
||||
@@ -95,6 +110,10 @@ template <typename State> class RegionHost {
|
||||
|
||||
// Provisional admission only; nothing is final until a FinalAck.
|
||||
[[nodiscard]] std::expected<void, AdmissionError> submit(const TickInput& input);
|
||||
// An action rides the first simulated tick >= record.tick and is committed with that
|
||||
// batch, in the same statement (schema v11). Final only through an ActionAck.
|
||||
[[nodiscard]] std::expected<void, AdmissionError>
|
||||
submit_action(const persistence::ActionRecord& record);
|
||||
// Advances the state machine once: lease upkeep, recovery, resolution, commits
|
||||
// and up to max_catch_up_ticks ticks. Blocks at most for bounded I/O.
|
||||
const HostStatus& step(SchedulerTime now);
|
||||
@@ -104,6 +123,8 @@ template <typename State> class RegionHost {
|
||||
[[nodiscard]] std::vector<FinalAck> take_acks();
|
||||
[[nodiscard]] std::vector<FinalOutcome> take_outcomes();
|
||||
[[nodiscard]] std::vector<Dropped> take_dropped();
|
||||
[[nodiscard]] std::vector<ActionAck> take_action_acks();
|
||||
[[nodiscard]] std::vector<DroppedAction> take_dropped_actions();
|
||||
[[nodiscard]] const HostStatus& status() const noexcept {
|
||||
return status_;
|
||||
}
|
||||
@@ -148,6 +169,10 @@ template <typename State> class RegionHost {
|
||||
std::vector<FinalAck> acks_;
|
||||
std::vector<FinalOutcome> outcomes_;
|
||||
std::vector<Dropped> dropped_;
|
||||
std::deque<persistence::ActionRecord> action_queue_;
|
||||
std::vector<persistence::ActionRecord> batch_actions_; // Riding the open batch.
|
||||
std::vector<ActionAck> action_acks_;
|
||||
std::vector<DroppedAction> dropped_actions_;
|
||||
SchedulerTime origin_{}, batch_started_{};
|
||||
// Store failures back off: a failed commit is retried after retry_delay_, which
|
||||
// doubles to a cap and resets on the next confirmed commit.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
#include <expected>
|
||||
#include <memory>
|
||||
#include <optional>
|
||||
#include <span>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
@@ -45,6 +46,8 @@ struct RegionSessionsConfig {
|
||||
network::ControlLedger* ledger{};
|
||||
std::chrono::milliseconds renew_interval{5000}, safety_margin{2000};
|
||||
network::IntentLimits intents{}; // max_sessions follows control.max_sessions.
|
||||
// Durable action records (10.4b), owned by the host; null keeps outcomes in memory only.
|
||||
network::ActionLedger* actions{};
|
||||
};
|
||||
// The region host's authority, read once per poll.
|
||||
struct HostFacts {
|
||||
@@ -74,9 +77,12 @@ class RegionSessions {
|
||||
// Actions scheduled for ticks <= `tick`, to be recorded durably with the batch that
|
||||
// simulates that tick (10.4b). Each action is returned once.
|
||||
[[nodiscard]] std::vector<network::CarriedAction> carry(std::uint64_t tick);
|
||||
// A commit covering ticks up to `end_tick` is durable: carried actions become final and
|
||||
// their outcomes are sent to each account's current session.
|
||||
void committed(std::uint64_t end_tick, std::uint64_t commit_sequence);
|
||||
// These carried actions are durable in `commit_sequence` (each with the tick it rode):
|
||||
// their outcomes go to each account's current session.
|
||||
void committed(std::span<const network::CarriedAction> actions, std::uint64_t commit_sequence);
|
||||
// The host dropped these carried actions: final ones are reported not_committed; the
|
||||
// others await their durable record (their batch outcome was unknown).
|
||||
void dropped(std::span<const network::CarriedAction> actions, bool final);
|
||||
// Closes every connection with `shutdown` (flushed by later polls).
|
||||
void shutdown(std::chrono::steady_clock::time_point now);
|
||||
[[nodiscard]] std::vector<RegionSessionEvent> take_events();
|
||||
|
||||
@@ -0,0 +1,170 @@
|
||||
-- M10.4b durable action records (ADR-0092). An action rides the batch that simulates its
|
||||
-- tick and becomes durable in the same statement as that batch's checkpoint. The action
|
||||
-- block is part of the commit's identity: an exact retry must repeat it byte for byte.
|
||||
--
|
||||
-- Action block: N records of 34 bytes, strictly ascending by (tick, account, action id):
|
||||
-- account uuid (16 raw bytes), action id u64 LE (nonzero), tick u64 LE in
|
||||
-- (begin_tick, end_tick], kind u16 LE (1 = probe). At most 1024 records.
|
||||
-- (world, account, action id) is unique across the world: an id is recorded once, by the
|
||||
-- first commit that carries it. A later batch carrying the same id (an account that moved
|
||||
-- regions while its action was pending in the old one) commits normally without a second
|
||||
-- record; the session layer reports outcomes from the record, never from the batch.
|
||||
-- Committed history stays immutable: each commit's block is kept in its own append-only
|
||||
-- table (a commit without a row has an empty block).
|
||||
CREATE TABLE evolution.region_commit_actions (
|
||||
commit_id uuid PRIMARY KEY REFERENCES evolution.region_commits(commit_id),
|
||||
actions bytea NOT NULL CHECK (octet_length(actions) > 0)
|
||||
);
|
||||
REVOKE ALL ON evolution.region_commit_actions FROM PUBLIC;
|
||||
|
||||
CREATE TABLE evolution.region_actions (
|
||||
world_id evolution.u64 NOT NULL,
|
||||
account_id uuid NOT NULL CHECK (account_id <> '00000000-0000-0000-0000-000000000000'),
|
||||
action_id evolution.u64 NOT NULL CHECK (action_id > 0),
|
||||
region_id evolution.u64 NOT NULL,
|
||||
tick evolution.u64 NOT NULL CHECK (tick > 0),
|
||||
kind smallint NOT NULL CHECK (kind = 1),
|
||||
commit_sequence evolution.u64 NOT NULL CHECK (commit_sequence > 0),
|
||||
PRIMARY KEY (world_id, account_id, action_id)
|
||||
);
|
||||
REVOKE ALL ON evolution.region_actions FROM PUBLIC;
|
||||
|
||||
-- The action block a commit was published with (empty when none).
|
||||
CREATE FUNCTION evolution.commit_actions(p_commit uuid)
|
||||
RETURNS bytea LANGUAGE sql STABLE SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
|
||||
SELECT coalesce((SELECT a.actions FROM evolution.region_commit_actions a
|
||||
WHERE a.commit_id = p_commit), ''::bytea);
|
||||
$$;
|
||||
|
||||
-- Validates an action block against a manifest's tick range; returns the record count.
|
||||
CREATE FUNCTION evolution.check_action_block(p_actions bytea, p_begin numeric, p_end numeric)
|
||||
RETURNS integer LANGUAGE plpgsql IMMUTABLE SECURITY INVOKER
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
n integer := coalesce(octet_length(p_actions), -1);
|
||||
account bytea;
|
||||
id numeric;
|
||||
tick numeric;
|
||||
kind numeric;
|
||||
prev_tick numeric := -1;
|
||||
prev_account bytea := ''::bytea;
|
||||
prev_id numeric := -1;
|
||||
seen text[] := '{}';
|
||||
BEGIN
|
||||
IF n < 0 OR n % 34 <> 0 OR n / 34 > 1024 THEN
|
||||
RAISE EXCEPTION 'invalid action block' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
FOR i IN 0 .. n / 34 - 1 LOOP
|
||||
account := substring(p_actions FROM 34 * i + 1 FOR 16);
|
||||
id := evolution.manifest_uint(p_actions, 34 * i + 16, 8);
|
||||
tick := evolution.manifest_uint(p_actions, 34 * i + 24, 8);
|
||||
kind := evolution.manifest_uint(p_actions, 34 * i + 32, 2);
|
||||
-- Canonical order: strictly ascending (tick, account bytes, id); no duplicates.
|
||||
IF account = decode(repeat('00', 16), 'hex') OR id = 0 OR kind <> 1
|
||||
OR tick <= p_begin OR tick > p_end
|
||||
OR (tick, account, id) <= (prev_tick, prev_account, prev_id) THEN
|
||||
RAISE EXCEPTION 'invalid action record' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- An action id appears once in a block, whatever its tick.
|
||||
IF encode(account, 'hex') || ':' || id::text = ANY (seen) THEN
|
||||
RAISE EXCEPTION 'invalid action record' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
seen := seen || (encode(account, 'hex') || ':' || id::text);
|
||||
prev_tick := tick;
|
||||
prev_account := account;
|
||||
prev_id := id;
|
||||
END LOOP;
|
||||
RETURN n / 34;
|
||||
END $$;
|
||||
|
||||
CREATE FUNCTION evolution.publish_region_checkpoint(p_manifest bytea, p_decisions bytea,
|
||||
p_actions bytea)
|
||||
RETURNS TABLE (region_frame evolution.u64, head_sequence evolution.u64, head_commit uuid,
|
||||
head_digest text, head_tick evolution.u64)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
c evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
|
||||
existing boolean;
|
||||
r record;
|
||||
n integer;
|
||||
BEGIN
|
||||
n := evolution.check_action_block(p_actions, c.begin_tick, c.end_tick);
|
||||
-- An exact retry repeats the actions too; any difference is a reused identity.
|
||||
existing := EXISTS (SELECT 1 FROM evolution.region_commits x WHERE x.commit_id = c.commit_id);
|
||||
IF existing AND evolution.commit_actions(c.commit_id) <> p_actions THEN
|
||||
RAISE EXCEPTION 'commit identity reused' USING ERRCODE = 'EVL07';
|
||||
END IF;
|
||||
-- The checkpoint itself: lease, head, decisions and exact-retry receipt (schema v5).
|
||||
SELECT * INTO r FROM evolution.publish_region_checkpoint(p_manifest, p_decisions);
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'publication returned no head' USING ERRCODE = 'XX000';
|
||||
END IF;
|
||||
IF NOT existing THEN -- First publication, not a retry: record the actions.
|
||||
FOR i IN 0 .. n - 1 LOOP
|
||||
INSERT INTO evolution.region_actions(world_id, account_id, action_id, region_id, tick,
|
||||
kind, commit_sequence)
|
||||
VALUES (c.world_id, encode(substring(p_actions FROM 34 * i + 1 FOR 16), 'hex')::uuid,
|
||||
evolution.manifest_uint(p_actions, 34 * i + 16, 8), c.region_id,
|
||||
evolution.manifest_uint(p_actions, 34 * i + 24, 8), 1, c.commit_sequence)
|
||||
ON CONFLICT (world_id, account_id, action_id) DO NOTHING;
|
||||
END LOOP;
|
||||
IF n > 0 THEN
|
||||
INSERT INTO evolution.region_commit_actions(commit_id, actions)
|
||||
VALUES (c.commit_id, p_actions);
|
||||
END IF;
|
||||
END IF;
|
||||
RETURN QUERY SELECT r.region_frame, r.head_sequence, r.head_commit, r.head_digest,
|
||||
r.head_tick;
|
||||
END $$;
|
||||
|
||||
-- Resolution after an ambiguous publication, including the action block.
|
||||
CREATE FUNCTION evolution.resolve_region_commit(p_world evolution.u64, p_region evolution.u64,
|
||||
p_commit uuid, p_manifest bytea, p_decisions bytea, p_actions bytea)
|
||||
RETURNS TABLE (status text, region_frame evolution.u64, head_sequence evolution.u64,
|
||||
head_commit uuid, head_digest text, head_tick evolution.u64)
|
||||
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF p_actions IS NULL THEN
|
||||
RAISE EXCEPTION 'invalid resolution request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_commits x WHERE x.commit_id = p_commit)
|
||||
AND evolution.commit_actions(p_commit) <> p_actions THEN
|
||||
RETURN QUERY SELECT 'conflict'::text, NULL::evolution.u64, NULL::evolution.u64, NULL::uuid,
|
||||
NULL::text, NULL::evolution.u64;
|
||||
RETURN;
|
||||
END IF;
|
||||
RETURN QUERY SELECT * FROM evolution.resolve_region_commit(p_world, p_region, p_commit,
|
||||
p_manifest, p_decisions);
|
||||
END $$;
|
||||
|
||||
-- The committed record of one action of an account in the caller's world, under the
|
||||
-- caller's exact region lease. No row: the action is not committed (and, once the caller's
|
||||
-- authority is recovered and ready, never was by an earlier epoch).
|
||||
CREATE FUNCTION evolution.find_region_action(p_world evolution.u64, p_region evolution.u64,
|
||||
p_holder uuid, p_epoch evolution.u64, p_account uuid, p_action evolution.u64)
|
||||
RETURNS TABLE (action_region evolution.u64, action_tick evolution.u64,
|
||||
action_commit evolution.u64)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF p_account IS NULL OR p_action IS NULL THEN
|
||||
RAISE EXCEPTION 'invalid action request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
PERFORM evolution.check_region_lease(p_world, p_region, p_holder, p_epoch);
|
||||
RETURN QUERY SELECT a.region_id, a.tick, a.commit_sequence FROM evolution.region_actions a
|
||||
WHERE a.world_id = p_world AND a.account_id = p_account AND a.action_id = p_action;
|
||||
END $$;
|
||||
|
||||
REVOKE ALL ON FUNCTION evolution.check_action_block(bytea, numeric, numeric),
|
||||
evolution.commit_actions(uuid),
|
||||
evolution.publish_region_checkpoint(bytea, bytea, bytea),
|
||||
evolution.resolve_region_commit(evolution.u64, evolution.u64, uuid, bytea, bytea, bytea),
|
||||
evolution.find_region_action(evolution.u64, evolution.u64, uuid, evolution.u64, uuid,
|
||||
evolution.u64)
|
||||
FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION
|
||||
evolution.publish_region_checkpoint(bytea, bytea, bytea),
|
||||
evolution.resolve_region_commit(evolution.u64, evolution.u64, uuid, bytea, bytea, bytea),
|
||||
evolution.find_region_action(evolution.u64, evolution.u64, uuid, evolution.u64, uuid,
|
||||
evolution.u64)
|
||||
TO :"runtime_role";
|
||||
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (11, :'migration_sha_11');
|
||||
+25
-4
@@ -45,28 +45,41 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|
||||
|| ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10' AS at_v10 \gset
|
||||
\if :at_v10
|
||||
|| ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10' AS at_v10,
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6'
|
||||
|| ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8'
|
||||
|| ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10'
|
||||
|| ',11:' || :'migration_sha_11' AS at_v11 \gset
|
||||
\if :at_v11
|
||||
\elif :at_v10
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v9
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v8
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v7
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v6
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v5
|
||||
\ir 0006_commit_lookup.sql
|
||||
\ir 0007_region_clones.sql
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v4
|
||||
\ir pre_v5_guard.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
@@ -75,6 +88,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v3
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir pre_v5_guard.sql
|
||||
@@ -84,6 +98,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v2
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
@@ -94,6 +109,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\elif :at_v1
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
@@ -105,6 +121,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\else
|
||||
\echo 'Unknown or changed migration history'
|
||||
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
|
||||
@@ -121,6 +138,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0008_upgrade_barrier.sql
|
||||
\ir 0009_serialized_authority.sql
|
||||
\ir 0010_account_control.sql
|
||||
\ir 0011_region_actions.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
|
||||
@@ -155,7 +173,10 @@ BEGIN
|
||||
('evolution.load_region_commit(evolution.u64,evolution.u64,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.claim_account_control(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,evolution.u64,evolution.u64,integer)'::regprocedure::oid),
|
||||
('evolution.renew_account_control(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,evolution.u64,integer)'::regprocedure::oid),
|
||||
('evolution.release_account_control(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,evolution.u64)'::regprocedure::oid)
|
||||
('evolution.release_account_control(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.publish_region_checkpoint(bytea,bytea,bytea)'::regprocedure::oid),
|
||||
('evolution.resolve_region_commit(evolution.u64,evolution.u64,uuid,bytea,bytea,bytea)'::regprocedure::oid),
|
||||
('evolution.find_region_action(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,evolution.u64)'::regprocedure::oid)
|
||||
), violations(role_oid, reason) AS (
|
||||
SELECT r.oid, 'privileged attribute or owner identity' FROM pg_catalog.pg_roles r
|
||||
JOIN closure c ON c.oid = r.oid
|
||||
@@ -179,7 +200,7 @@ BEGIN
|
||||
OR EXISTS (SELECT 1 FROM pg_catalog.pg_namespace n WHERE n.nspname = 'public'
|
||||
AND pg_catalog.has_schema_privilege(c.oid, n.oid, 'CREATE'))
|
||||
UNION ALL
|
||||
-- Only the fifteen guarded functions are executable; provisioning/internal are not.
|
||||
-- Only the eighteen guarded functions are executable; provisioning/internal are not.
|
||||
SELECT c.oid, 'EXECUTE outside the fenced function set' FROM closure c
|
||||
CROSS JOIN pg_catalog.pg_proc p
|
||||
WHERE p.pronamespace = 'evolution'::regnamespace
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
# Region action block, version 1
|
||||
|
||||
Durable record of the actions that rode one checkpoint batch (ADR-0092, schema v11,
|
||||
migrations/0011_region_actions.sql). Encoded by `encode_actions`
|
||||
(include/evo/persistence/region_batch.hpp), checked again by
|
||||
`evolution.check_action_block`, published by the three-argument
|
||||
`publish_region_checkpoint` in the same statement as the batch's checkpoint. The block is
|
||||
not part of the trace or manifest, so simulation evidence and replay are unchanged; it is
|
||||
part of the commit's identity (an exact retry or resolution must repeat it byte for byte).
|
||||
|
||||
N records of 34 bytes, integers little-endian, N <= 1024:
|
||||
|
||||
| Offset | Size | Field | Rule |
|
||||
|---|---|---|---|
|
||||
| 0 | 16 | account | AccountId bytes, not all zero |
|
||||
| 16 | 8 | action_id | nonzero |
|
||||
| 24 | 8 | tick | the tick the action rode, in (begin_tick, end_tick] of the batch |
|
||||
| 32 | 2 | kind | 1 (probe, no effect) |
|
||||
|
||||
Records are strictly ascending by (tick, account bytes, action_id), and an
|
||||
(account, action_id) pair appears once. An empty block publishes exactly as before v11.
|
||||
|
||||
Each record also becomes a row of `evolution.region_actions` keyed by
|
||||
(world, account, action_id). The first commit that carries an id records it; a later batch
|
||||
carrying the same id (an account that moved regions while the action was pending in the
|
||||
old one) commits normally and adds no second record. `find_region_action` is therefore the
|
||||
authority for an action's outcome.
|
||||
@@ -145,28 +145,60 @@ ActionReceipt IntentAdmission::action(std::uint64_t session, const SessionAction
|
||||
r.status = AdmitStatus::outside_window;
|
||||
else if (unresolved_of(*s) >= limits_.max_unresolved)
|
||||
r.status = AdmitStatus::too_many_unresolved;
|
||||
else if (ledger_) {
|
||||
// Committed earlier (another epoch, process or region): never admitted again.
|
||||
const auto found = lookup(account, m.action_id);
|
||||
if (found.state == Record::unavailable) {
|
||||
r.status = AdmitStatus::not_ready; // Uniqueness cannot be confirmed now.
|
||||
} else if (found.state == Record::committed) {
|
||||
r.status = AdmitStatus::duplicate;
|
||||
r.tick = found.outcome.tick;
|
||||
remember(account, found.outcome);
|
||||
}
|
||||
}
|
||||
if (r.status == AdmitStatus::admitted)
|
||||
actions_.push_back({{account, m.action_id, m.tick, ActionKind::probe}, Stage::scheduled});
|
||||
return r;
|
||||
}
|
||||
|
||||
ActionOutcome IntentAdmission::query(std::uint64_t session, const ActionQuery& m) const {
|
||||
ActionOutcome IntentAdmission::query(std::uint64_t session, const ActionQuery& m,
|
||||
const AdmissionFacts& facts) {
|
||||
ActionOutcome o{m.action_id, ActionResult::unknown, 0, 0};
|
||||
const auto* s = find(session);
|
||||
if (!s)
|
||||
return o;
|
||||
const auto account = s->account;
|
||||
const auto pending = std::ranges::find_if(actions_, [&](const Action& a) {
|
||||
return a.carried.account == s->account && a.carried.action_id == m.action_id;
|
||||
return a.carried.account == account && a.carried.action_id == m.action_id;
|
||||
});
|
||||
if (pending != actions_.end()) {
|
||||
o.result = ActionResult::pending;
|
||||
o.tick = pending->carried.tick;
|
||||
// A batch outcome left unknown by a lost authority is decided by the durable record
|
||||
// once this authority is ready: every earlier epoch's commits are then final.
|
||||
if (pending->unknown && ledger_ && facts.ready)
|
||||
if (const auto found = lookup(account, m.action_id);
|
||||
found.state != Record::unavailable) {
|
||||
o = found.state == Record::committed
|
||||
? found.outcome
|
||||
: ActionOutcome{m.action_id, ActionResult::not_committed, 0, 0};
|
||||
actions_.erase(pending);
|
||||
remember(account, o);
|
||||
}
|
||||
return o;
|
||||
}
|
||||
const auto final = std::ranges::find_if(finals_, [&](const AccountOutcome& f) {
|
||||
return f.account == s->account && f.outcome.action_id == m.action_id;
|
||||
return f.account == account && f.outcome.action_id == m.action_id;
|
||||
});
|
||||
return final != finals_.end() ? final->outcome : o;
|
||||
if (final != finals_.end())
|
||||
return final->outcome;
|
||||
// No longer in memory: a committed action is still on record.
|
||||
if (ledger_ && facts.ready)
|
||||
if (const auto found = lookup(account, m.action_id); found.state == Record::committed) {
|
||||
remember(account, found.outcome);
|
||||
return found.outcome;
|
||||
}
|
||||
return o;
|
||||
}
|
||||
|
||||
std::vector<std::pair<std::uint64_t, SimulatedThrough>>
|
||||
@@ -206,23 +238,51 @@ void IntentAdmission::remember(const AccountId& account, const ActionOutcome& ou
|
||||
finals_.pop_front();
|
||||
}
|
||||
|
||||
std::vector<AccountOutcome> IntentAdmission::committed(std::uint64_t end_tick,
|
||||
std::vector<AccountOutcome> IntentAdmission::committed(std::span<const CarriedAction> actions,
|
||||
std::uint64_t commit_sequence) {
|
||||
std::vector<AccountOutcome> out;
|
||||
std::erase_if(actions_, [&](const Action& a) {
|
||||
if (a.stage != Stage::carried || a.carried.tick > end_tick)
|
||||
return false;
|
||||
const ActionOutcome o{a.carried.action_id, ActionResult::committed_no_effect,
|
||||
a.carried.tick, commit_sequence};
|
||||
out.push_back({a.carried.account, o});
|
||||
remember(a.carried.account, o);
|
||||
return true;
|
||||
});
|
||||
for (const auto& c : actions) {
|
||||
ActionOutcome o{c.action_id, ActionResult::committed_no_effect, c.tick, commit_sequence};
|
||||
// The durable record is the authority: an id carried by two batches (an account that
|
||||
// moved regions with the action pending) keeps the record of the first commit.
|
||||
if (ledger_)
|
||||
if (const auto found = lookup(c.account, c.action_id); found.state == Record::committed)
|
||||
o = found.outcome;
|
||||
std::erase_if(actions_, [&](const Action& a) {
|
||||
return a.carried.account == c.account && a.carried.action_id == c.action_id;
|
||||
});
|
||||
out.push_back({c.account, o});
|
||||
remember(c.account, o);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<AccountOutcome> IntentAdmission::dropped(std::span<const CarriedAction> actions,
|
||||
bool final) {
|
||||
std::vector<AccountOutcome> out;
|
||||
for (const auto& c : actions) {
|
||||
const auto it = std::ranges::find_if(actions_, [&](const Action& a) {
|
||||
return a.carried.account == c.account && a.carried.action_id == c.action_id;
|
||||
});
|
||||
if (!final) {
|
||||
if (it != actions_.end())
|
||||
it->unknown = true;
|
||||
continue;
|
||||
}
|
||||
if (it != actions_.end())
|
||||
actions_.erase(it);
|
||||
const ActionOutcome o{c.action_id, ActionResult::not_committed, 0, 0};
|
||||
out.push_back({c.account, o});
|
||||
remember(c.account, o);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<AccountOutcome> IntentAdmission::abandon() {
|
||||
std::vector<AccountOutcome> out;
|
||||
for (auto& a : actions_)
|
||||
if (a.stage == Stage::carried)
|
||||
a.unknown = true; // Decided by a commit report, a drop report or its record.
|
||||
std::erase_if(actions_, [&](const Action& a) {
|
||||
if (a.stage != Stage::scheduled)
|
||||
return false;
|
||||
@@ -236,6 +296,21 @@ std::vector<AccountOutcome> IntentAdmission::abandon() {
|
||||
return out;
|
||||
}
|
||||
|
||||
IntentAdmission::Found IntentAdmission::lookup(const AccountId& account,
|
||||
std::uint64_t action_id) const {
|
||||
Found f;
|
||||
if (!ledger_)
|
||||
return f;
|
||||
const auto answer = ledger_->find(account, action_id);
|
||||
if (!answer.has_value())
|
||||
return f;
|
||||
const auto& record = answer.value();
|
||||
f.state = record.has_value() ? Record::committed : Record::absent;
|
||||
if (record.has_value())
|
||||
f.outcome = record.value();
|
||||
return f;
|
||||
}
|
||||
|
||||
std::optional<std::uint64_t> IntentAdmission::session_of(const AccountId& account) const {
|
||||
const auto it =
|
||||
std::ranges::find_if(sessions_, [&](const Session& s) { return s.account == account; });
|
||||
|
||||
@@ -49,7 +49,7 @@ publish_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGra
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<typename Codec::State>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t budget, milliseconds timeout,
|
||||
const PublishHooks& hooks) {
|
||||
const PublishHooks& hooks, std::span<const ActionRecord> actions) {
|
||||
const auto previous = decode_manifest(predecessor.manifest);
|
||||
if (!previous || previous->profile != Codec::profile || !batch.sealed() ||
|
||||
batch.steps().empty() || batch.begin_tick() != previous->end_tick)
|
||||
@@ -81,7 +81,8 @@ publish_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGra
|
||||
batch.begin_tick(),
|
||||
batch.end_tick()};
|
||||
auto trace = encode_trace(header, batch.steps());
|
||||
if (!decision_bytes || !trace)
|
||||
auto action_bytes = encode_actions(actions, batch.begin_tick(), batch.end_tick());
|
||||
if (!decision_bytes || !trace || !action_bytes)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const auto payload_digest = sha256(*payload);
|
||||
const auto trace_digest = sha256(*trace);
|
||||
@@ -104,10 +105,10 @@ publish_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGra
|
||||
if (auto stored = put_verified(objects, *trace, {*trace_digest, trace->size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
|
||||
PreparedCommit prepared{previous->key, commit, *bytes, *decision_bytes};
|
||||
PreparedCommit prepared{previous->key, commit, *bytes, *decision_bytes, *action_bytes};
|
||||
if (hooks.before_transaction)
|
||||
hooks.before_transaction();
|
||||
auto published = publish_region_checkpoint(db, *bytes, *decision_bytes, timeout);
|
||||
auto published = retry_prepared(db, prepared, timeout);
|
||||
auto committed = confirm(published, std::move(*bytes), *manifest, std::move(prepared));
|
||||
if (!committed)
|
||||
return std::unexpected{committed.error()};
|
||||
@@ -128,9 +129,10 @@ publish_environmental_batch(Postgres& db, const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::EnvironmentalCommunity>& batch,
|
||||
Uuid commit, const Sha256Digest& build, std::size_t restore_budget,
|
||||
milliseconds timeout, const PublishHooks& hooks) {
|
||||
milliseconds timeout, const PublishHooks& hooks,
|
||||
std::span<const ActionRecord> actions) {
|
||||
return publish_batch<EnvironmentalCodec>(db, objects, lease, predecessor, batch, commit, build,
|
||||
restore_budget, timeout, hooks);
|
||||
restore_budget, timeout, hooks, actions);
|
||||
}
|
||||
std::expected<RecoveredCheckpoint, RecoveryFailure>
|
||||
recover_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
@@ -152,9 +154,9 @@ publish_lineage_batch(Postgres& db, const FilesystemObjectStore& objects, const
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::LineageRegion>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget, milliseconds timeout,
|
||||
const PublishHooks& hooks) {
|
||||
const PublishHooks& hooks, std::span<const ActionRecord> actions) {
|
||||
return publish_batch<LineageCodec>(db, objects, lease, predecessor, batch, commit, build,
|
||||
restore_budget, timeout, hooks);
|
||||
restore_budget, timeout, hooks, actions);
|
||||
}
|
||||
std::expected<RecoveredLineageCheckpoint, RecoveryFailure>
|
||||
recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
@@ -165,11 +167,18 @@ recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, R
|
||||
|
||||
std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_prepared(Postgres& db, const PreparedCommit& prepared, milliseconds timeout) {
|
||||
if (!prepared.actions.empty())
|
||||
return resolve_region_commit(db, prepared.key, prepared.commit, prepared.manifest,
|
||||
prepared.decisions, prepared.actions, timeout);
|
||||
return resolve_region_commit(db, prepared.key, prepared.commit, prepared.manifest,
|
||||
prepared.decisions, timeout);
|
||||
}
|
||||
std::expected<RegionHead, LeaseStoreFailure>
|
||||
retry_prepared(Postgres& db, const PreparedCommit& prepared, milliseconds timeout) {
|
||||
// Without actions the schema v5 publication is used, exactly as before v11.
|
||||
if (!prepared.actions.empty())
|
||||
return publish_region_checkpoint(db, prepared.manifest, prepared.decisions,
|
||||
prepared.actions, timeout);
|
||||
return publish_region_checkpoint(db, prepared.manifest, prepared.decisions, timeout);
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include <algorithm>
|
||||
#include <tuple>
|
||||
#include <utility>
|
||||
|
||||
namespace evo::persistence {
|
||||
namespace {
|
||||
@@ -223,4 +225,39 @@ std::vector<BatchDecision> derive_decisions(std::span<const TraceStep> steps,
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_actions(std::span<const ActionRecord> actions, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick) {
|
||||
if (actions.size() > max_batch_actions)
|
||||
return std::unexpected{BatchError::format};
|
||||
std::vector<ActionRecord> sorted(actions.begin(), actions.end());
|
||||
const auto key = [](const ActionRecord& a) {
|
||||
return std::tuple{a.tick, a.account, a.action_id};
|
||||
};
|
||||
std::ranges::sort(sorted, {}, key);
|
||||
// An action id appears once, whatever its tick.
|
||||
std::vector<std::pair<Uuid, std::uint64_t>> ids;
|
||||
ids.reserve(sorted.size());
|
||||
for (const auto& a : sorted)
|
||||
ids.emplace_back(a.account, a.action_id);
|
||||
std::ranges::sort(ids);
|
||||
if (std::ranges::adjacent_find(ids) != ids.end())
|
||||
return std::unexpected{BatchError::format};
|
||||
std::vector<std::uint8_t> out;
|
||||
out.reserve(sorted.size() * action_record_bytes);
|
||||
for (std::size_t i = 0; i < sorted.size(); ++i) {
|
||||
const auto& a = sorted[i];
|
||||
if (a.account == Uuid{} || a.action_id == 0 || a.kind != 1 || a.tick <= begin_tick ||
|
||||
a.tick > end_tick || (i > 0 && key(sorted[i - 1]) == key(a)))
|
||||
return std::unexpected{BatchError::format};
|
||||
out.insert(out.end(), a.account.begin(), a.account.end());
|
||||
for (unsigned b = 0; b < 8; ++b)
|
||||
out.push_back(static_cast<std::uint8_t>(a.action_id >> (8U * b)));
|
||||
for (unsigned b = 0; b < 8; ++b)
|
||||
out.push_back(static_cast<std::uint8_t>(a.tick >> (8U * b)));
|
||||
out.push_back(static_cast<std::uint8_t>(a.kind));
|
||||
out.push_back(static_cast<std::uint8_t>(a.kind >> 8U));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -281,6 +281,77 @@ resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
return ResolvedCommit{Resolution::committed, *head};
|
||||
}
|
||||
|
||||
std::expected<RegionHead, LeaseStoreFailure>
|
||||
publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::span<const std::uint8_t> actions, milliseconds timeout) {
|
||||
const std::array<std::string, 3> p{bytea(manifest), bytea(decisions), bytea(actions)};
|
||||
auto rows = db.query(
|
||||
"SELECT * FROM evolution.publish_region_checkpoint($1::bytea, $2::bytea, $3::bytea)", p,
|
||||
timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
auto head = parse_head(*rows);
|
||||
if (!head)
|
||||
return std::unexpected{malformed(true)};
|
||||
return *head;
|
||||
}
|
||||
|
||||
std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::span<const std::uint8_t> actions, milliseconds timeout) {
|
||||
const std::array<std::string, 6> p{std::to_string(key.world), std::to_string(key.region),
|
||||
uuid_text(commit), bytea(manifest),
|
||||
bytea(decisions), bytea(actions)};
|
||||
auto rows = db.query("SELECT * FROM evolution.resolve_region_commit($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::uuid, $4::bytea, $5::bytea, $6::bytea)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
if (rows->size() != 1 || (*rows)[0].size() != 6)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto& status_text = (*rows)[0][0];
|
||||
if (!status_text)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto status = *status_text;
|
||||
if (status == "absent")
|
||||
return ResolvedCommit{Resolution::absent, {}};
|
||||
if (status == "conflict")
|
||||
return ResolvedCommit{Resolution::conflict, {}};
|
||||
const PgRows receipt{
|
||||
std::vector<std::optional<std::string>>((*rows)[0].begin() + 1, (*rows)[0].end())};
|
||||
auto head = status == "committed" ? parse_head(receipt) : std::nullopt;
|
||||
if (!head)
|
||||
return std::unexpected{malformed(false)};
|
||||
return ResolvedCommit{Resolution::committed, *head};
|
||||
}
|
||||
|
||||
std::expected<std::optional<CommittedAction>, LeaseStoreFailure>
|
||||
find_region_action(Postgres& db, RegionKey key, const LeaseGrant& lease, const Uuid& account,
|
||||
std::uint64_t action_id, milliseconds timeout) {
|
||||
const std::array<std::string, 6> p{std::to_string(key.world), std::to_string(key.region),
|
||||
uuid_text(lease.holder), std::to_string(lease.epoch),
|
||||
uuid_text(account), std::to_string(action_id)};
|
||||
auto rows = db.query("SELECT * FROM evolution.find_region_action($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::uuid, $4::evolution.u64, $5::uuid, "
|
||||
"$6::evolution.u64)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
if (rows->empty())
|
||||
return std::optional<CommittedAction>{};
|
||||
if (rows->size() != 1 || (*rows)[0].size() != 3)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto region = parse_integer<std::uint64_t>((*rows)[0][0]);
|
||||
const auto tick = parse_integer<std::uint64_t>((*rows)[0][1]);
|
||||
const auto sequence = parse_integer<std::uint64_t>((*rows)[0][2]);
|
||||
if (!region || !tick || !sequence)
|
||||
return std::unexpected{malformed(false)};
|
||||
return std::optional<CommittedAction>{CommittedAction{*region, *tick, *sequence}};
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
decisions_from(const PgRows& rows) {
|
||||
|
||||
@@ -100,6 +100,27 @@ std::expected<void, AdmissionError> RegionHost<State>::submit(const TickInput& i
|
||||
return {};
|
||||
}
|
||||
|
||||
template <typename State>
|
||||
std::expected<void, AdmissionError>
|
||||
RegionHost<State>::submit_action(const persistence::ActionRecord& record) {
|
||||
if (record.account == persistence::Uuid{} || record.action_id == 0 || record.kind != 1 ||
|
||||
record.tick == 0)
|
||||
return std::unexpected{AdmissionError::invalid};
|
||||
if (draining_)
|
||||
return std::unexpected{AdmissionError::draining};
|
||||
if (!status_.ready)
|
||||
return std::unexpected{AdmissionError::not_ready};
|
||||
const auto same = [&](const persistence::ActionRecord& a) {
|
||||
return a.account == record.account && a.action_id == record.action_id;
|
||||
};
|
||||
if (std::ranges::any_of(action_queue_, same) || std::ranges::any_of(batch_actions_, same))
|
||||
return std::unexpected{AdmissionError::duplicate};
|
||||
if (action_queue_.size() + batch_actions_.size() >= config_.max_queued_actions)
|
||||
return std::unexpected{AdmissionError::full};
|
||||
action_queue_.push_back(record);
|
||||
return {};
|
||||
}
|
||||
|
||||
template <typename State> bool RegionHost<State>::database() {
|
||||
if (db_)
|
||||
return true;
|
||||
@@ -118,6 +139,13 @@ void RegionHost<State>::lose_authority(Unacknowledged reason, HostPhase next) {
|
||||
dropped_.push_back({a.input, reason});
|
||||
for (auto& q : queue_)
|
||||
dropped_.push_back({q, reason});
|
||||
// Actions of a batch whose publication outcome is unknown may be durable: not final.
|
||||
for (const auto& a : batch_actions_)
|
||||
dropped_actions_.push_back({a, !prepared_.has_value(), reason});
|
||||
for (const auto& a : action_queue_)
|
||||
dropped_actions_.push_back({a, true, reason});
|
||||
batch_actions_.clear();
|
||||
action_queue_.clear();
|
||||
applied_.clear();
|
||||
queue_.clear();
|
||||
batch_.reset(); // References state_: reset first.
|
||||
@@ -248,6 +276,9 @@ void RegionHost<State>::confirmed(const persistence::PublishedBatch& published,
|
||||
applied_.clear();
|
||||
for (const auto& d : published.decisions)
|
||||
outcomes_.push_back({d, head.sequence});
|
||||
for (const auto& a : batch_actions_)
|
||||
action_acks_.push_back({a, head.sequence});
|
||||
batch_actions_.clear();
|
||||
prepared_.reset();
|
||||
batch_started_ = now;
|
||||
retry_at_ = {};
|
||||
@@ -272,9 +303,9 @@ template <typename State> void RegionHost<State>::commit(SchedulerTime now) {
|
||||
persistence::PublishHooks hooks;
|
||||
if (crash)
|
||||
hooks.before_transaction = [&crash] { crash(CrashPoint::before_transaction); };
|
||||
auto published =
|
||||
Ops<State>::publish(*db_, *objects_, *grant, *committed_, *batch_, commit_id, config_.build,
|
||||
config_.restore_budget, config_.lease.commit_timeout, hooks);
|
||||
auto published = Ops<State>::publish(*db_, *objects_, *grant, *committed_, *batch_, commit_id,
|
||||
config_.build, config_.restore_budget,
|
||||
config_.lease.commit_timeout, hooks, batch_actions_);
|
||||
if (published) {
|
||||
confirmed(*published, now);
|
||||
return;
|
||||
@@ -387,6 +418,14 @@ template <typename State> void RegionHost<State>::simulate(SchedulerTime now) {
|
||||
continue;
|
||||
}
|
||||
applied_.push_back({input, report->tick});
|
||||
// Actions due at or before this tick ride it (late ones ride the first tick run).
|
||||
std::erase_if(action_queue_, [&](persistence::ActionRecord action) {
|
||||
if (action.tick > report->tick)
|
||||
return false;
|
||||
action.tick = report->tick;
|
||||
batch_actions_.push_back(action);
|
||||
return true;
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -482,6 +521,9 @@ template <typename State> void RegionHost<State>::drain() {
|
||||
for (auto& q : queue_)
|
||||
dropped_.push_back({q, Unacknowledged::drained});
|
||||
queue_.clear();
|
||||
for (const auto& a : action_queue_)
|
||||
dropped_actions_.push_back({a, true, Unacknowledged::drained});
|
||||
action_queue_.clear();
|
||||
if (status_.phase == HostPhase::running)
|
||||
status_.phase = HostPhase::draining;
|
||||
else if (status_.phase == HostPhase::acquiring)
|
||||
@@ -498,6 +540,12 @@ template <typename State> std::vector<FinalOutcome> RegionHost<State>::take_outc
|
||||
template <typename State> std::vector<Dropped> RegionHost<State>::take_dropped() {
|
||||
return std::exchange(dropped_, {});
|
||||
}
|
||||
template <typename State> std::vector<ActionAck> RegionHost<State>::take_action_acks() {
|
||||
return std::exchange(action_acks_, {});
|
||||
}
|
||||
template <typename State> std::vector<DroppedAction> RegionHost<State>::take_dropped_actions() {
|
||||
return std::exchange(dropped_actions_, {});
|
||||
}
|
||||
|
||||
template class RegionHost<processes::EnvironmentalCommunity>;
|
||||
template class RegionHost<processes::LineageRegion>;
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
#include "evo/server/postgres_action_ledger.hpp"
|
||||
|
||||
namespace evo::server {
|
||||
std::expected<std::optional<network::ActionOutcome>, network::ActionLedgerError>
|
||||
PostgresActionLedger::find(const network::AccountId& account, std::uint64_t action_id) {
|
||||
const auto lease = lease_();
|
||||
if (!lease)
|
||||
return std::unexpected(network::ActionLedgerError::unavailable);
|
||||
const auto found =
|
||||
persistence::find_region_action(*db_, key_, *lease, account.bytes, action_id, timeout_);
|
||||
if (!found.has_value())
|
||||
return std::unexpected(network::ActionLedgerError::unavailable);
|
||||
const auto& record = found.value();
|
||||
if (!record.has_value())
|
||||
return std::optional<network::ActionOutcome>{};
|
||||
return std::optional<network::ActionOutcome>{
|
||||
network::ActionOutcome{action_id, network::ActionResult::committed_no_effect,
|
||||
record.value().tick, record.value().commit_sequence}};
|
||||
}
|
||||
} // namespace evo::server
|
||||
@@ -282,14 +282,14 @@ struct RegionSessions::Impl {
|
||||
send(c, receipt);
|
||||
// A duplicate of a final action: answer with the outcome as well.
|
||||
if (receipt.status == network::AdmitStatus::duplicate)
|
||||
if (const auto o = intents.query(c.session, {m->action_id});
|
||||
if (const auto o = intents.query(c.session, {m->action_id}, admission());
|
||||
o.result == network::ActionResult::committed_no_effect ||
|
||||
o.result == network::ActionResult::not_committed)
|
||||
send(c, o);
|
||||
return;
|
||||
}
|
||||
if (const auto* m = std::get_if<network::ActionQuery>(&*decoded))
|
||||
return send(c, intents.query(c.session, *m));
|
||||
return send(c, intents.query(c.session, *m, admission()));
|
||||
if (const auto* m = std::get_if<network::SessionClose>(&*decoded);
|
||||
m && m->reason == SessionCloseReason::client_closed) {
|
||||
close(c, SessionCloseReason::client_closed, now);
|
||||
@@ -324,6 +324,7 @@ RegionSessions::create(RegionSessionsConfig config) {
|
||||
auto intents = network::IntentAdmission::create(config.intents);
|
||||
if (!intents)
|
||||
return std::unexpected(RegionSessionsError::configuration);
|
||||
intents->attach(config.actions);
|
||||
auto endpoint = network::quic::Endpoint::serve(config.transport);
|
||||
if (!endpoint)
|
||||
return std::unexpected(RegionSessionsError::transport);
|
||||
@@ -407,8 +408,13 @@ std::vector<network::CarriedAction> RegionSessions::carry(std::uint64_t tick) {
|
||||
return impl_->intents.carry(tick);
|
||||
}
|
||||
|
||||
void RegionSessions::committed(std::uint64_t end_tick, std::uint64_t commit_sequence) {
|
||||
impl_->deliver(impl_->intents.committed(end_tick, commit_sequence));
|
||||
void RegionSessions::committed(std::span<const network::CarriedAction> actions,
|
||||
std::uint64_t commit_sequence) {
|
||||
impl_->deliver(impl_->intents.committed(actions, commit_sequence));
|
||||
}
|
||||
|
||||
void RegionSessions::dropped(std::span<const network::CarriedAction> actions, bool final) {
|
||||
impl_->deliver(impl_->intents.dropped(actions, final));
|
||||
}
|
||||
|
||||
void RegionSessions::shutdown(Clock::time_point now) {
|
||||
|
||||
+93
-14
@@ -412,6 +412,85 @@ void session_codec() {
|
||||
CHECK(refused(decode_session(encode_session(w)), SessionCodecError::invalid_value));
|
||||
}
|
||||
}
|
||||
// Scripted durable action records (10.4b).
|
||||
struct FakeActions final : ActionLedger {
|
||||
std::expected<std::optional<ActionOutcome>, ActionLedgerError>
|
||||
find(const AccountId& who, std::uint64_t action_id) override {
|
||||
++lookups;
|
||||
if (unavailable)
|
||||
return std::unexpected(ActionLedgerError::unavailable);
|
||||
for (const auto& [account, outcome] : records)
|
||||
if (account == who && outcome.action_id == action_id)
|
||||
return std::optional{outcome};
|
||||
return std::optional<ActionOutcome>{};
|
||||
}
|
||||
std::vector<std::pair<AccountId, ActionOutcome>> records;
|
||||
bool unavailable{};
|
||||
int lookups{};
|
||||
};
|
||||
|
||||
void durable_actions() {
|
||||
auto a = *IntentAdmission::create(IntentLimits{});
|
||||
FakeActions ledger;
|
||||
a.attach(&ledger);
|
||||
const auto alice = account(1);
|
||||
CHECK(a.open(1, alice));
|
||||
AdmissionFacts f{true, 5, 100};
|
||||
const auto action = [&](std::uint64_t id, std::uint64_t tick) {
|
||||
return a.action(1, SessionAction{1, 5, id, 1, tick}, f);
|
||||
};
|
||||
// Committed by an earlier epoch, process or region: a duplicate with its outcome.
|
||||
const ActionOutcome old{9, ActionResult::committed_no_effect, 40, 3};
|
||||
ledger.records.emplace_back(alice, old);
|
||||
auto r = action(9, 101);
|
||||
CHECK(r.status == AdmitStatus::duplicate && r.tick == 40);
|
||||
CHECK(a.query(1, {9}, f) == old);
|
||||
// Uniqueness that cannot be confirmed is not admitted.
|
||||
ledger.unavailable = true;
|
||||
CHECK(action(10, 101).status == AdmitStatus::not_ready);
|
||||
ledger.unavailable = false;
|
||||
const int before = ledger.lookups;
|
||||
CHECK(action(10, 101).status == AdmitStatus::admitted && ledger.lookups == before + 1);
|
||||
// A refused attempt never costs a lookup.
|
||||
CHECK(action(11, 150).status == AdmitStatus::outside_window && ledger.lookups == before + 1);
|
||||
// Exact commit: the tick reported is the tick the action rode.
|
||||
auto riding = a.carry(101);
|
||||
CHECK(riding.size() == 1);
|
||||
riding[0].tick = 102;
|
||||
auto outcomes = a.committed(riding, 6);
|
||||
CHECK(outcomes.size() == 1 &&
|
||||
outcomes[0].outcome == ActionOutcome{10, ActionResult::committed_no_effect, 102, 6});
|
||||
// A final drop is not_committed; an unknown one stays pending until its record decides.
|
||||
f.server_tick = 108;
|
||||
CHECK(action(12, 109).status == AdmitStatus::admitted);
|
||||
CHECK(action(13, 109).status == AdmitStatus::admitted);
|
||||
riding = a.carry(109);
|
||||
CHECK(riding.size() == 2);
|
||||
outcomes = a.dropped(std::span{riding}.first(1), true);
|
||||
CHECK(outcomes.size() == 1 && outcomes[0].outcome.result == ActionResult::not_committed);
|
||||
CHECK(a.dropped(std::span{riding}.subspan(1), false).empty());
|
||||
f.ready = false; // Recovering: nothing is decided yet.
|
||||
CHECK(a.query(1, {13}, f).result == ActionResult::pending);
|
||||
f.ready = true;
|
||||
ledger.records.emplace_back(alice,
|
||||
ActionOutcome{13, ActionResult::committed_no_effect, 109, 7});
|
||||
CHECK(a.query(1, {13}, f) == ActionOutcome{13, ActionResult::committed_no_effect, 109, 7});
|
||||
// The same for an action whose record is absent once the new authority is ready.
|
||||
f.server_tick = 116;
|
||||
CHECK(action(14, 117).status == AdmitStatus::admitted);
|
||||
riding = a.carry(117);
|
||||
CHECK(a.dropped(riding, false).empty());
|
||||
CHECK(a.query(1, {14}, f) == ActionOutcome{14, ActionResult::not_committed, 0, 0});
|
||||
// Outcomes evicted from memory are found again on record.
|
||||
auto small = IntentLimits{};
|
||||
small.max_finals = 1;
|
||||
auto b = *IntentAdmission::create(small);
|
||||
b.attach(&ledger);
|
||||
CHECK(b.open(1, alice));
|
||||
CHECK(b.query(1, {13}, f).result == ActionResult::committed_no_effect);
|
||||
CHECK(b.query(1, {99}, f).result == ActionResult::unknown);
|
||||
}
|
||||
|
||||
void intent_admission() {
|
||||
IntentLimits limits;
|
||||
limits.max_unresolved = 6;
|
||||
@@ -465,7 +544,7 @@ void intent_admission() {
|
||||
f.server_tick = 200;
|
||||
auto r = action(1, 10, 203);
|
||||
CHECK(r.status == AdmitStatus::admitted && r.tick == 203 && r.server_tick == 200);
|
||||
CHECK(a.query(1, {10}).result == ActionResult::pending);
|
||||
CHECK(a.query(1, {10}, f).result == ActionResult::pending);
|
||||
CHECK(action(1, 0, 203).status == AdmitStatus::invalid_action);
|
||||
CHECK(action(1, 11, 203, 9).status == AdmitStatus::invalid_action);
|
||||
{ // Wire kind 257 must not truncate to a known kind.
|
||||
@@ -481,23 +560,23 @@ void intent_admission() {
|
||||
const auto carried = a.carry(203);
|
||||
CHECK(carried.size() == 1 && carried[0] == CarriedAction{alice, 10, 203, ActionKind::probe});
|
||||
CHECK(a.carry(203).empty()); // Each action is carried once.
|
||||
CHECK(a.committed(202, 7).empty());
|
||||
auto outcomes = a.committed(203, 8);
|
||||
CHECK(a.committed({}, 7).empty());
|
||||
auto outcomes = a.committed(carried, 8);
|
||||
CHECK(outcomes.size() == 1 && outcomes[0].account == alice &&
|
||||
outcomes[0].outcome == ActionOutcome{10, ActionResult::committed_no_effect, 203, 8});
|
||||
CHECK(a.query(1, {10}) == outcomes[0].outcome);
|
||||
CHECK(a.query(2, {10}).result == ActionResult::unknown); // Bob's account has no action 10.
|
||||
CHECK(a.query(1, {10}, f) == outcomes[0].outcome);
|
||||
CHECK(a.query(2, {10}, f).result == ActionResult::unknown); // Bob's account has no action 10.
|
||||
// A reconnect of the same account (new session) sees the same outcome.
|
||||
a.close(1);
|
||||
CHECK(a.query(1, {10}).result == ActionResult::unknown); // Closed session asks nothing.
|
||||
CHECK(a.query(1, {10}, f).result == ActionResult::unknown); // Closed session asks nothing.
|
||||
CHECK(a.open(3, alice) && a.session_of(alice) == 3u);
|
||||
CHECK(a.query(3, {10}) == outcomes[0].outcome);
|
||||
CHECK(a.query(3, {10}, f) == outcomes[0].outcome);
|
||||
f.server_tick = 300;
|
||||
CHECK(action(3, 10, 301).status == AdmitStatus::duplicate); // Never admitted twice.
|
||||
// Unresolved cap counts intentions and actions; abandon finalizes only uncarried actions.
|
||||
CHECK(action(3, 20, 301).status == AdmitStatus::admitted);
|
||||
CHECK(action(3, 21, 308).status == AdmitStatus::admitted);
|
||||
(void)a.carry(301);
|
||||
const auto riding = a.carry(301);
|
||||
for (std::uint64_t i = 0; i < 4; ++i)
|
||||
CHECK(a.intention(3, SessionIntention{3, 5, i + 1, 302 + i, 0, 0}, f).status ==
|
||||
AdmitStatus::admitted);
|
||||
@@ -507,18 +586,17 @@ void intent_admission() {
|
||||
outcomes = a.abandon();
|
||||
CHECK(outcomes.size() == 1 &&
|
||||
outcomes[0].outcome == ActionOutcome{21, ActionResult::not_committed, 0, 0});
|
||||
CHECK(a.query(3, {20}).result == ActionResult::pending); // Carried: awaits its record.
|
||||
CHECK(a.query(3, {20}, f).result == ActionResult::pending); // Carried: awaits its record.
|
||||
CHECK(a.unresolved(3) == 1);
|
||||
CHECK(a.committed(301, 9).size() == 1); // Its batch turned out durable.
|
||||
CHECK(a.committed(riding, 9).size() == 1); // Its batch turned out durable.
|
||||
// Final outcomes are kept within a bound; older ones become unknown here.
|
||||
for (std::uint64_t id = 30; id < 34; ++id) {
|
||||
f.server_tick += 8;
|
||||
CHECK(action(3, id, f.server_tick + 1).status == AdmitStatus::admitted);
|
||||
(void)a.carry(f.server_tick + 1);
|
||||
CHECK(a.committed(f.server_tick + 1, id).size() == 1);
|
||||
CHECK(a.committed(a.carry(f.server_tick + 1), id).size() == 1);
|
||||
}
|
||||
CHECK(a.query(3, {10}).result == ActionResult::unknown);
|
||||
CHECK(a.query(3, {33}).result == ActionResult::committed_no_effect);
|
||||
CHECK(a.query(3, {10}, f).result == ActionResult::unknown);
|
||||
CHECK(a.query(3, {33}, f).result == ActionResult::committed_no_effect);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
@@ -528,5 +606,6 @@ void control_tests() {
|
||||
control_rules();
|
||||
session_codec();
|
||||
intent_admission();
|
||||
durable_actions();
|
||||
}
|
||||
} // namespace evo::network_test
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
#include <stdexcept>
|
||||
void run_region_lease_store_tests();
|
||||
void run_account_control_store_tests();
|
||||
void run_region_action_tests();
|
||||
void run_region_checkpoint_store_tests();
|
||||
void run_region_host_tests();
|
||||
void run_region_server_process_tests();
|
||||
@@ -118,6 +119,7 @@ int main() {
|
||||
std::puts("postgres adapter integration passed");
|
||||
run_region_lease_store_tests();
|
||||
run_account_control_store_tests();
|
||||
run_region_action_tests();
|
||||
std::puts("region lease fencing integration passed");
|
||||
run_region_checkpoint_store_tests();
|
||||
std::puts("region checkpoint publication integration passed");
|
||||
|
||||
@@ -406,7 +406,7 @@ void intentions_and_actions(const Files& f) {
|
||||
CHECK(h.server->carry(102).empty());
|
||||
const auto carried = h.server->carry(103);
|
||||
CHECK(carried.size() == 1 && carried[0] == CarriedAction{alice, 7, 103, ActionKind::probe});
|
||||
h.server->committed(103, 4);
|
||||
h.server->committed(carried, 4);
|
||||
CHECK(h.until([&] { return !(outcomes = a->take_action_outcomes()).empty(); }));
|
||||
CHECK(outcomes[0] == ActionOutcome{7, ActionResult::committed_no_effect, 103, 4});
|
||||
// A retry gets a duplicate receipt and the same final outcome.
|
||||
|
||||
@@ -0,0 +1,412 @@
|
||||
// Real PostgreSQL M10.4b durable action records (ADR-0092, schema v11): actions committed in
|
||||
// the same statement as their batch's checkpoint, exact retries and resolution including
|
||||
// the action block, invalid blocks, an action id never committed twice, and host crashes
|
||||
// before and after the commit. Regions 7/131..135 and commit ids tagged 0xA7 are used only
|
||||
// here. Run by
|
||||
// tools/ci/postgres-check.sh.
|
||||
#include "evo/persistence/checkpoint_store.hpp"
|
||||
#include "evo/persistence/region_lease_store.hpp"
|
||||
#include "evo/server/region_host.hpp"
|
||||
#include "fixtures/environmental_region.hpp"
|
||||
#ifdef EVO_POSTGRES_SESSIONS
|
||||
#include "evo/server/postgres_action_ledger.hpp"
|
||||
#endif
|
||||
#include <array>
|
||||
#include <cstdio>
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
#include <source_location>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
void run_region_action_tests();
|
||||
|
||||
namespace {
|
||||
using namespace evo::persistence;
|
||||
using namespace evo::server;
|
||||
using namespace std::chrono_literals;
|
||||
using evo::processes::LineageRegion;
|
||||
namespace fs = std::filesystem;
|
||||
void check(bool value, std::source_location at = std::source_location::current()) {
|
||||
if (!value)
|
||||
throw std::runtime_error("region action test line " + std::to_string(at.line()));
|
||||
}
|
||||
#define CHECK(...) check(static_cast<bool>(__VA_ARGS__))
|
||||
template <typename T, typename E> bool holds(const std::expected<T, E>& e, const T& v) {
|
||||
return e.has_value() && *e == v;
|
||||
}
|
||||
constexpr std::uint64_t world = 7; // The fixtures' world.
|
||||
constexpr std::uint64_t base_region = 130;
|
||||
constexpr std::size_t budget = 8000000;
|
||||
|
||||
std::string env(const char* key) {
|
||||
const auto* value = std::getenv(key);
|
||||
if (!value)
|
||||
throw std::runtime_error("missing test environment");
|
||||
return value;
|
||||
}
|
||||
PgOptions options(const std::string& user) {
|
||||
return {env("PGHOST"), {}, env("PGPORT"), env("PGDATABASE"), user, {}, {}};
|
||||
}
|
||||
std::unique_ptr<Postgres> connect_as(const std::string& user) {
|
||||
auto made = Postgres::connect(options(user), 5s);
|
||||
CHECK(made);
|
||||
return std::move(*made);
|
||||
}
|
||||
template <std::size_t N> std::array<std::uint8_t, N> filled(std::uint8_t value) {
|
||||
std::array<std::uint8_t, N> out{};
|
||||
out.fill(value);
|
||||
return out;
|
||||
}
|
||||
const Sha256Digest build = filled<32>(0xB1);
|
||||
const Uuid alice = filled<16>(0xA1), bob = filled<16>(0xB0);
|
||||
const Uuid carol = filled<16>(0xC1), dave = filled<16>(0xD0); // Host tests only.
|
||||
// Commit ids are unique across the database: tag this file's.
|
||||
Uuid tagged(std::uint8_t n) {
|
||||
Uuid id = filled<16>(0xA7);
|
||||
id.back() = n;
|
||||
return id;
|
||||
}
|
||||
|
||||
struct Region {
|
||||
RegionKey key;
|
||||
fs::path root;
|
||||
CommittedCheckpoint genesis;
|
||||
};
|
||||
Region provisioned(std::uint64_t region) {
|
||||
region += base_region;
|
||||
Region r{{world, region}, fs::path(env("PGHOST")) / ("actions-" + std::to_string(region)), {}};
|
||||
fs::remove_all(r.root);
|
||||
fs::create_directory(r.root);
|
||||
fs::permissions(r.root, fs::perms::owner_all, fs::perm_options::replace);
|
||||
auto owner = connect_as(env("PGUSER"));
|
||||
CHECK(owner->query("INSERT INTO evolution.retained_builds(build_identity) VALUES "
|
||||
"(decode(repeat('b1', 32), 'hex')) ON CONFLICT DO NOTHING",
|
||||
{}, 5s));
|
||||
auto objects = FilesystemObjectStore::open(r.root, r.key, max_lineage_payload_bytes);
|
||||
CHECK(objects);
|
||||
auto genesis = provision_lineage_region(
|
||||
*owner, **objects, r.key, evo::test::region_identity.frame, *evo::test::lineage_region(),
|
||||
tagged(static_cast<std::uint8_t>(region)), build, budget, 5s);
|
||||
if (!genesis)
|
||||
std::fprintf(stderr, "provision failed: kind %d sqlstate %s\n",
|
||||
static_cast<int>(genesis.error().kind),
|
||||
genesis.error().store.cause.sqlstate.data());
|
||||
CHECK(genesis);
|
||||
r.genesis = *genesis;
|
||||
return r;
|
||||
}
|
||||
std::vector<std::uint8_t> record(const Uuid& account, std::uint64_t id, std::uint64_t tick,
|
||||
std::uint16_t kind = 1) {
|
||||
std::vector<std::uint8_t> out(account.begin(), account.end());
|
||||
for (unsigned b = 0; b < 8; ++b)
|
||||
out.push_back(static_cast<std::uint8_t>(id >> (8U * b)));
|
||||
for (unsigned b = 0; b < 8; ++b)
|
||||
out.push_back(static_cast<std::uint8_t>(tick >> (8U * b)));
|
||||
out.push_back(static_cast<std::uint8_t>(kind));
|
||||
out.push_back(static_cast<std::uint8_t>(kind >> 8U));
|
||||
return out;
|
||||
}
|
||||
std::vector<std::uint8_t> join(std::initializer_list<std::vector<std::uint8_t>> parts) {
|
||||
std::vector<std::uint8_t> out;
|
||||
for (const auto& p : parts)
|
||||
out.insert(out.end(), p.begin(), p.end());
|
||||
return out;
|
||||
}
|
||||
// Owner read of the durable record: (region, tick, commit) or empty.
|
||||
std::string stored(Postgres& owner, const Uuid& account, std::uint64_t id) {
|
||||
const std::array<std::string, 2> p{std::string{"\\x"} +
|
||||
[&] {
|
||||
std::string h;
|
||||
constexpr char digits[] = "0123456789abcdef";
|
||||
for (const auto b : account) {
|
||||
h.push_back(digits[b >> 4U]);
|
||||
h.push_back(digits[b & 0xFU]);
|
||||
}
|
||||
return h;
|
||||
}(),
|
||||
std::to_string(id)};
|
||||
auto rows = owner.query("SELECT region_id || ':' || tick || ':' || commit_sequence FROM "
|
||||
"evolution.region_actions WHERE world_id = 7 AND account_id = "
|
||||
"encode($1::bytea, 'hex')::uuid AND action_id = $2::evolution.u64",
|
||||
p, 5s);
|
||||
CHECK(rows);
|
||||
return rows->empty() ? std::string{} : (*rows)[0][0].value_or("");
|
||||
}
|
||||
|
||||
void store_rules() {
|
||||
auto r = provisioned(1);
|
||||
auto db = connect_as(env("EVO_PG_RUNTIME_USER"));
|
||||
auto objects = FilesystemObjectStore::open(r.root, r.key, max_lineage_payload_bytes);
|
||||
CHECK(objects);
|
||||
auto lease = acquire_region_lease(*db, r.key, build, 20000ms, 5s);
|
||||
CHECK(lease);
|
||||
auto state = evo::test::lineage_region();
|
||||
RegionBatch<LineageRegion> batch{*state};
|
||||
for (std::uint64_t tick = 1; tick <= 10; ++tick)
|
||||
CHECK(batch.advance(evo::test::region_input(tick, false)));
|
||||
batch.seal();
|
||||
const std::vector<ActionRecord> actions{{alice, 2, 10, 1}, {bob, 1, 3, 1}, {alice, 1, 3, 1}};
|
||||
auto first = publish_lineage_batch(*db, **objects, *lease, r.genesis, batch, tagged(0x01),
|
||||
build, budget, 5s, {}, actions);
|
||||
if (!first)
|
||||
std::fprintf(stderr, "publish failed: kind %d store %d sqlstate %s\n",
|
||||
static_cast<int>(first.error().kind),
|
||||
static_cast<int>(first.error().store.kind),
|
||||
first.error().store.cause.sqlstate.data());
|
||||
CHECK(first && first->committed.head.sequence == 1);
|
||||
CHECK(batch.confirm(10));
|
||||
auto found = find_region_action(*db, r.key, *lease, alice, 1, 5s);
|
||||
CHECK(found && *found == CommittedAction{131, 3, 1});
|
||||
CHECK(holds(find_region_action(*db, r.key, *lease, bob, 1, 5s),
|
||||
std::optional{CommittedAction{131, 3, 1}}));
|
||||
CHECK(holds(find_region_action(*db, r.key, *lease, alice, 2, 5s),
|
||||
std::optional{CommittedAction{131, 10, 1}}));
|
||||
CHECK(holds(find_region_action(*db, r.key, *lease, alice, 3, 5s),
|
||||
std::optional<CommittedAction>{}));
|
||||
|
||||
// Exact retry and resolution include the action block.
|
||||
const auto block = encode_actions(actions, 0, 10);
|
||||
CHECK(block);
|
||||
PreparedCommit prepared{r.key, tagged(0x01), first->committed.manifest, {}, *block};
|
||||
CHECK(retry_prepared(*db, prepared, 5s) == first->committed.head);
|
||||
auto resolved = resolve_prepared(*db, prepared, 5s);
|
||||
CHECK(resolved && resolved->status == Resolution::committed);
|
||||
auto changed = prepared;
|
||||
changed.actions = join({record(alice, 1, 3), record(alice, 2, 10)}); // One missing.
|
||||
auto retried = retry_prepared(*db, changed, 5s);
|
||||
CHECK(!retried && retried.error().kind == LeaseStoreError::commit_conflict);
|
||||
resolved = resolve_prepared(*db, changed, 5s);
|
||||
CHECK(resolved && resolved->status == Resolution::conflict);
|
||||
// Every invalid block is refused before anything else.
|
||||
for (const auto& bad :
|
||||
{record(alice, 9, 11), record(alice, 9, 0), record(alice, 9, 5, 2), record(Uuid{}, 9, 5),
|
||||
record(alice, 0, 5), join({record(bob, 9, 5), record(alice, 9, 5)}),
|
||||
join({record(alice, 9, 5), record(alice, 9, 5)}),
|
||||
join({record(alice, 9, 6), record(alice, 9, 5)}),
|
||||
join({record(alice, 9, 5), record(alice, 9, 6)}), // One id at two ticks.
|
||||
std::vector<std::uint8_t>(33, 1)}) {
|
||||
auto p = prepared;
|
||||
p.actions = bad;
|
||||
retried = retry_prepared(*db, p, 5s);
|
||||
CHECK(!retried && retried.error().kind == LeaseStoreError::invalid);
|
||||
}
|
||||
// The local encoder refuses the same blocks it would send.
|
||||
CHECK(!encode_actions(std::vector<ActionRecord>{{alice, 9, 11, 1}}, 0, 10));
|
||||
CHECK(!encode_actions(std::vector<ActionRecord>{{alice, 9, 5, 1}, {alice, 9, 6, 1}}, 0, 10));
|
||||
|
||||
// An action id is recorded once in a world: a later batch carrying it commits, and the
|
||||
// first record stands.
|
||||
RegionBatch<LineageRegion> next{*state};
|
||||
for (std::uint64_t tick = 11; tick <= 20; ++tick)
|
||||
CHECK(next.advance(evo::test::region_input(tick, false)));
|
||||
next.seal();
|
||||
const std::vector<ActionRecord> repeat{{alice, 1, 12, 1}, {alice, 3, 12, 1}};
|
||||
auto second = publish_lineage_batch(*db, **objects, *lease, first->committed, next,
|
||||
tagged(0x03), build, budget, 5s, {}, repeat);
|
||||
CHECK(second && second->committed.head.sequence == 2);
|
||||
CHECK(holds(find_region_action(*db, r.key, *lease, alice, 1, 5s),
|
||||
std::optional{CommittedAction{131, 3, 1}}));
|
||||
// A batch without actions still publishes as before v11.
|
||||
CHECK(next.confirm(20));
|
||||
RegionBatch<LineageRegion> plain{*state};
|
||||
for (std::uint64_t tick = 21; tick <= 25; ++tick)
|
||||
CHECK(plain.advance(evo::test::region_input(tick, false)));
|
||||
plain.seal();
|
||||
CHECK(publish_lineage_batch(*db, **objects, *lease, second->committed, plain, tagged(0x04),
|
||||
build, budget, 5s));
|
||||
// Lookups are fenced like everything else under the lease.
|
||||
CHECK(release_region_lease(*db, r.key, *lease, 5s));
|
||||
auto later = acquire_region_lease(*db, r.key, build, 20000ms, 5s);
|
||||
CHECK(later);
|
||||
auto fenced = find_region_action(*db, r.key, *lease, alice, 1, 5s);
|
||||
CHECK(!fenced && fenced.error().kind == LeaseStoreError::fenced);
|
||||
CHECK(holds(find_region_action(*db, r.key, *later, alice, 3, 5s),
|
||||
std::optional{CommittedAction{131, 12, 2}}));
|
||||
#ifdef EVO_POSTGRES_SESSIONS
|
||||
// The session layer's ledger over the same lookup.
|
||||
std::optional<LeaseGrant> current = *later;
|
||||
PostgresActionLedger ledger{*db, r.key, [&] { return current; }};
|
||||
evo::network::AccountId who;
|
||||
who.bytes = alice;
|
||||
CHECK(holds(ledger.find(who, 3),
|
||||
std::optional{evo::network::ActionOutcome{
|
||||
3, evo::network::ActionResult::committed_no_effect, 12, 2}}));
|
||||
CHECK(holds(ledger.find(who, 99), std::optional<evo::network::ActionOutcome>{}));
|
||||
current.reset();
|
||||
CHECK(!ledger.find(who, 3));
|
||||
#endif
|
||||
CHECK(release_region_lease(*db, r.key, *later, 5s));
|
||||
}
|
||||
|
||||
// Host-level: fast lease (4 s), 1 s commits, as in the host tests.
|
||||
const LeasePolicy fast{4000ms, 800ms, 400ms, 400ms, 1000ms};
|
||||
struct Crash {};
|
||||
HostConfig config(const Region& r) {
|
||||
HostConfig c;
|
||||
c.key = r.key;
|
||||
c.frame = evo::test::region_identity.frame;
|
||||
c.lease = fast;
|
||||
c.build = build;
|
||||
c.retained_builds = {build};
|
||||
c.max_queued_actions = 3;
|
||||
return c;
|
||||
}
|
||||
std::unique_ptr<RegionHost<LineageRegion>> host(const Region& r, HostConfig c) {
|
||||
auto objects = FilesystemObjectStore::open(r.root, r.key, max_lineage_payload_bytes);
|
||||
CHECK(objects);
|
||||
auto made = RegionHost<LineageRegion>::create(
|
||||
std::move(c), [] { return Postgres::connect(options(env("EVO_PG_RUNTIME_USER")), 2s); },
|
||||
std::move(*objects));
|
||||
CHECK(made);
|
||||
return std::move(*made);
|
||||
}
|
||||
TickInput input(std::uint64_t tick, std::uint8_t tag) {
|
||||
Uuid request{};
|
||||
request[0] = tag;
|
||||
for (unsigned i = 0; i < 8; ++i)
|
||||
request[8 + i] = static_cast<std::uint8_t>((tick >> (8U * i)) & 0xFFU);
|
||||
return {request, evo::test::region_input(tick, false)};
|
||||
}
|
||||
template <typename Done>
|
||||
const HostStatus& step_until(RegionHost<LineageRegion>& h, SchedulerTime& t, Done done,
|
||||
int limit = 1000) {
|
||||
for (int i = 0; i < limit; ++i) {
|
||||
t += 50ms;
|
||||
const auto& s = h.step(t);
|
||||
if (done(s))
|
||||
return s;
|
||||
std::this_thread::sleep_for(10ms);
|
||||
}
|
||||
CHECK(false);
|
||||
return h.status();
|
||||
}
|
||||
|
||||
void host_actions() {
|
||||
auto owner = connect_as(env("PGUSER"));
|
||||
{
|
||||
// Normal path: acknowledged with the tick each rode; late actions ride the next tick.
|
||||
const auto r = provisioned(2);
|
||||
auto h = host(r, config(r));
|
||||
CHECK(h->submit_action({carol, 1, 3, 1}).error() == AdmissionError::not_ready);
|
||||
SchedulerTime t = std::chrono::steady_clock::now();
|
||||
CHECK(h->step(t).ready);
|
||||
CHECK(h->submit_action({carol, 1, 3, 1}));
|
||||
CHECK(h->submit_action({carol, 1, 4, 1}).error() == AdmissionError::duplicate);
|
||||
CHECK(h->submit_action({carol, 0, 4, 1}).error() == AdmissionError::invalid);
|
||||
CHECK(h->submit_action({carol, 5, 4, 2}).error() == AdmissionError::invalid);
|
||||
CHECK(h->submit_action({dave, 1, 1, 1}));
|
||||
CHECK(h->submit_action({dave, 2, 2, 1}));
|
||||
CHECK(h->submit_action({dave, 3, 2, 1}).error() == AdmissionError::full);
|
||||
for (std::uint64_t tick = 1; tick <= 30; ++tick)
|
||||
CHECK(h->submit(input(tick, 0x71)));
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.committed_sequence >= 1; });
|
||||
auto acks = h->take_action_acks();
|
||||
CHECK(acks.size() == 3);
|
||||
for (const auto& a : acks)
|
||||
CHECK(a.commit_sequence == 1 &&
|
||||
a.record.tick == (a.record.account == dave ? a.record.action_id : 3u));
|
||||
CHECK(stored(*owner, carol, 1) == "132:3:1");
|
||||
// Submitted after tick 20 ran for tick 5: rides the first tick simulated next.
|
||||
const auto ran = h->status().simulated_tick;
|
||||
CHECK(h->submit_action({carol, 7, 5, 1}));
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.committed_sequence >= 2; });
|
||||
acks = h->take_action_acks();
|
||||
CHECK(acks.size() == 1 && acks[0].record.tick == ran + 1);
|
||||
CHECK(h->take_dropped_actions().empty());
|
||||
}
|
||||
{
|
||||
// Crash before the transaction: never committed; a successor finds no record.
|
||||
const auto r = provisioned(3);
|
||||
bool armed = false;
|
||||
auto c = config(r);
|
||||
c.crash_hook = [&](CrashPoint p) {
|
||||
if (armed && p == CrashPoint::before_transaction)
|
||||
throw Crash{};
|
||||
};
|
||||
auto h = host(r, c);
|
||||
SchedulerTime t = std::chrono::steady_clock::now();
|
||||
CHECK(h->step(t).ready);
|
||||
for (std::uint64_t tick = 1; tick <= 30; ++tick)
|
||||
CHECK(h->submit(input(tick, 0x72)));
|
||||
CHECK(h->submit_action({carol, 10, 5, 1}));
|
||||
armed = true;
|
||||
bool crashed = false;
|
||||
try {
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.committed_sequence >= 1; });
|
||||
} catch (const Crash&) {
|
||||
crashed = true;
|
||||
}
|
||||
CHECK(crashed);
|
||||
h.reset();
|
||||
auto successor = host(r, config(r));
|
||||
step_until(*successor, t, [](const HostStatus& s) { return s.ready; });
|
||||
CHECK(successor->status().committed_sequence == 0 && stored(*owner, carol, 10).empty());
|
||||
}
|
||||
{
|
||||
// Crash after the commit, before any acknowledgement: the record is durable.
|
||||
const auto r = provisioned(4);
|
||||
bool armed = false;
|
||||
auto c = config(r);
|
||||
c.crash_hook = [&](CrashPoint p) {
|
||||
if (armed && p == CrashPoint::after_commit)
|
||||
throw Crash{};
|
||||
};
|
||||
auto h = host(r, c);
|
||||
SchedulerTime t = std::chrono::steady_clock::now();
|
||||
CHECK(h->step(t).ready);
|
||||
for (std::uint64_t tick = 1; tick <= 30; ++tick)
|
||||
CHECK(h->submit(input(tick, 0x73)));
|
||||
CHECK(h->submit_action({carol, 11, 5, 1}));
|
||||
armed = true;
|
||||
bool crashed = false;
|
||||
try {
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.committed_sequence >= 1; });
|
||||
} catch (const Crash&) {
|
||||
crashed = true;
|
||||
}
|
||||
CHECK(crashed && h->take_action_acks().empty());
|
||||
h.reset();
|
||||
CHECK(stored(*owner, carol, 11) == "134:5:1");
|
||||
auto successor = host(r, config(r));
|
||||
step_until(*successor, t, [](const HostStatus& s) { return s.ready; });
|
||||
CHECK(successor->status().committed_sequence == 1);
|
||||
CHECK(successor->submit_action({carol, 11, 40, 1})); // The host does not deduplicate
|
||||
// across processes: admission does (find_region_action), and the database refuses a
|
||||
// second commit of the id (store_rules).
|
||||
}
|
||||
{
|
||||
// Unknown publication outcome: resolved by the exact bytes, actions included.
|
||||
const auto r = provisioned(5);
|
||||
auto h = host(r, config(r));
|
||||
SchedulerTime t = std::chrono::steady_clock::now();
|
||||
CHECK(h->step(t).ready);
|
||||
for (std::uint64_t tick = 1; tick <= 30; ++tick)
|
||||
CHECK(h->submit(input(tick, 0x74)));
|
||||
CHECK(h->submit_action({dave, 12, 2, 1}));
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.simulated_tick >= 2; });
|
||||
auto lock = connect_as(env("PGUSER"));
|
||||
const std::array<std::string, 2> key{std::to_string(r.key.world),
|
||||
std::to_string(r.key.region)};
|
||||
CHECK(lock->query("BEGIN", {}, 5s));
|
||||
CHECK(lock->query("SELECT 1 FROM evolution.region_leases WHERE world_id = "
|
||||
"$1::evolution.u64 AND region_id = $2::evolution.u64 FOR UPDATE",
|
||||
key, 5s));
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.phase == HostPhase::resolving; });
|
||||
CHECK(h->take_action_acks().empty() && h->take_dropped_actions().empty());
|
||||
CHECK(lock->query("ROLLBACK", {}, 5s));
|
||||
step_until(*h, t, [](const HostStatus& s) { return s.phase == HostPhase::running; });
|
||||
const auto acks = h->take_action_acks();
|
||||
CHECK(acks.size() == 1 && acks[0].record.action_id == 12 && acks[0].commit_sequence == 1);
|
||||
CHECK(stored(*owner, dave, 12) == "135:2:1");
|
||||
}
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_action_tests() {
|
||||
store_rules();
|
||||
host_actions();
|
||||
std::puts("region action integration passed");
|
||||
}
|
||||
@@ -582,7 +582,7 @@ void durable_outcomes_and_ambiguity() {
|
||||
// receipt and records nothing twice; resolution confirms it under any epoch.
|
||||
auto death_block = encode_decisions(died->decisions, 0, 20);
|
||||
CHECK(death_block);
|
||||
const PreparedCommit lost{r.key, filled<16>(0xF1), died->committed.manifest, *death_block};
|
||||
const PreparedCommit lost{r.key, filled<16>(0xF1), died->committed.manifest, *death_block, {}};
|
||||
auto retried = retry_prepared(*r.runtime, lost, 5s);
|
||||
CHECK(retried && *retried == died->committed.head);
|
||||
auto resolved = resolve_prepared(*r.runtime, lost, 5s);
|
||||
|
||||
@@ -176,6 +176,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
" -v migration_sha_7=$(s 0007_region_clones.sql) -v migration_sha_8=invalid"
|
||||
" -v migration_sha_9=$(s 0009_serialized_authority.sql)"
|
||||
" -v migration_sha_10=$(s 0010_account_control.sql)"
|
||||
" -v migration_sha_11=$(s 0011_region_actions.sql)"
|
||||
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
|
||||
" > /dev/null 2>&1") != 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
|
||||
@@ -191,7 +192,7 @@ void upgrade_across_migration_and_barrier() {
|
||||
CHECK(shell("PGDATABASE=" + database +
|
||||
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
|
||||
"tools/migrate-postgres.sh\" > /dev/null") == 0);
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10");
|
||||
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11");
|
||||
CHECK(renew_region_lease(*runtime, r.key, *lease, 20000ms, 5s));
|
||||
confirm(publish_next(20)); // 41..60, now v8.
|
||||
CHECK(head.head.end_tick == 60);
|
||||
|
||||
@@ -61,7 +61,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
|
||||
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11
|
||||
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
|
||||
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
|
||||
v2_sha="${v2_sha%% *}"
|
||||
@@ -82,7 +82,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 10:1,2,3,4,5,6,7,8,9,10
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 11:1,2,3,4,5,6,7,8,9,10,11
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
|
||||
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
|
||||
@@ -136,7 +136,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
|
||||
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
|
||||
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
|
||||
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
|
||||
@@ -169,14 +169,20 @@ expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.check_region_lease(evo
|
||||
expect_audit_failure 'ALTER ROLE evo_runtime_login CREATEDB' 'ALTER ROLE evo_runtime_login NOCREATEDB'
|
||||
expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.provision_region_checkpoint(bytea) TO evo_runtime_test' 'REVOKE EXECUTE ON FUNCTION evolution.provision_region_checkpoint(bytea) FROM evo_runtime_test'
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.region_commits TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commits FROM evo_runtime_test'
|
||||
# Schema v11: action records are reachable only through the fenced functions.
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.region_actions TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_actions FROM evo_runtime_test'
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.region_commit_actions TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commit_actions FROM evo_runtime_test'
|
||||
expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.check_action_block(bytea, numeric, numeric) TO evo_runtime_test' 'REVOKE EXECUTE ON FUNCTION evolution.check_action_block(bytea, numeric, numeric) FROM evo_runtime_test'
|
||||
# Schema v10: account controls are reachable only through the three fenced functions.
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.account_controls TO evo_runtime_test' 'REVOKE SELECT ON evolution.account_controls FROM evo_runtime_test'
|
||||
expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) FROM evo_runtime_test' 'GRANT EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) TO evo_runtime_test'
|
||||
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
|
||||
# Unknown future versions and checksum drift must not be repaired silently.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=11 WHERE version=10'
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=12 WHERE version=11'
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=11 WHERE version=12"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=11"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=10 WHERE version=11"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=10"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=9"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# exported). Real evo-server processes on world 500:
|
||||
# 1. Failed migration: a migration that takes the migration lock, alters a history
|
||||
# table, holds its locks for 3 s and then fails runs while build b1 serves.
|
||||
# Nothing it did remains (schema still v10, no new column); the host keeps its
|
||||
# Nothing it did remains (schema still v11, no new column); the host keeps its
|
||||
# authority, and every acknowledgement stays durable.
|
||||
# 2. Barrier (ADR-0068 operator order): build b2 is retained, and a b2 host is started
|
||||
# while b1 still holds the lease (it waits, never ready). upgrade_barrier(b1) fences
|
||||
@@ -61,7 +61,7 @@ SQL
|
||||
[ "$code" != 0 ] && grep -q 'division by zero' "$work/migration.log" || fail "the rehearsal migration did not fail"
|
||||
[ "$(q -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'region_commits' AND column_name = 'rehearsal_column'")" = 0 ] ||
|
||||
fail "the failed migration left a column"
|
||||
[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 10 ] || fail "schema version moved"
|
||||
[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 11 ] || fail "schema version moved"
|
||||
wait_ack "$work/b1.log" $((before + 40)) || fail "b1 did not resume after the failed migration"
|
||||
kill -0 "$b1_pid" 2> /dev/null || fail "b1 exited during the failed migration"
|
||||
bash "$root/tools/migrate-postgres.sh" > /dev/null || fail "migrate after the failure"
|
||||
|
||||
@@ -32,11 +32,13 @@ digest_9="$(sha256sum "$root/migrations/0009_serialized_authority.sql")"
|
||||
digest_9="${digest_9%% *}"
|
||||
digest_10="$(sha256sum "$root/migrations/0010_account_control.sql")"
|
||||
digest_10="${digest_10%% *}"
|
||||
digest_11="$(sha256sum "$root/migrations/0011_region_actions.sql")"
|
||||
digest_11="${digest_11%% *}"
|
||||
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
|
||||
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
|
||||
--set="migration_sha_6=$digest_6" --set="migration_sha_7=$digest_7" \
|
||||
--set="migration_sha_8=$digest_8" --set="migration_sha_9=$digest_9" \
|
||||
--set="migration_sha_10=$digest_10" \
|
||||
--set="migration_sha_10=$digest_10" --set="migration_sha_11=$digest_11" \
|
||||
--set="runtime_role=$runtime_role" \
|
||||
--file="$root/migrations/apply.sql"
|
||||
|
||||
Reference in New Issue
Block a user