Complete M7.5 durable batch outcomes and commit resolution
ADR-0063: manifest v2 (360 bytes) binds a per-tick validation trace object and a canonical decision block by SHA-256. Publication replays the trace from the committed predecessor's verified payload and requires every report and the candidate bytes, derives deaths deterministically, uploads payload and trace with verified read-back, then sends manifest and decisions to one fenced statement. RegionBatch seals at a terminal transition or 64 ticks and refuses ticks until durable confirmation. Schema v5 records the checkpoint, its decisions and outbox rows atomically; a unique index keeps one terminal outcome per actor (EVL08), triggers make commits/decisions immutable (EVL09), an exact retry of a committed ID returns the original receipt, changed bytes conflict, and read-only resolve_region_commit answers committed/absent/ conflict. Outbox pending/acknowledge give idempotent at-least-once delivery. Recovery reconciles durable deaths with restored lifecycle (and archive deaths for the lineage profile). Validated: Windows CTest 58/58; Linux GCC Debug/Release, clang-tidy and ASan/UBSan CTest 58/58 each; PostgreSQL suites Release, analysis, ASan (lost response, committed-after-timeout, terminated orphan retry, changed bytes, repeated/partial/reordered decisions, gap, outbox redelivery, history quarantine); five mutations each caught. No Docker/Gitea CI, process kill, broker or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+6
-1
@@ -78,7 +78,7 @@ target_include_directories(evo_region_lease PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/
|
||||
target_compile_features(evo_region_lease PUBLIC cxx_std_23)
|
||||
evo_owned_target(evo_region_lease)
|
||||
# Portable checkpoint manifest codec and environmental-v1 payload validation.
|
||||
add_library(evo_region_checkpoint STATIC src/persistence/region_checkpoint.cpp)
|
||||
add_library(evo_region_checkpoint STATIC src/persistence/region_checkpoint.cpp src/persistence/region_batch.cpp)
|
||||
add_library(evo::region_checkpoint ALIAS evo_region_checkpoint)
|
||||
target_link_libraries(evo_region_checkpoint PUBLIC evo::region_lease evo::processes)
|
||||
evo_owned_target(evo_region_checkpoint)
|
||||
@@ -123,6 +123,11 @@ if(BUILD_TESTING)
|
||||
evo_owned_target(evo-lineage-region-tests)
|
||||
add_test(NAME lineage-region-contracts COMMAND evo-lineage-region-tests)
|
||||
set_tests_properties(lineage-region-contracts PROPERTIES TIMEOUT 60)
|
||||
add_executable(evo-region-batch-tests tests/region_batch_tests.cpp)
|
||||
target_link_libraries(evo-region-batch-tests PRIVATE evo::region_checkpoint)
|
||||
evo_owned_target(evo-region-batch-tests)
|
||||
add_test(NAME region-batch-contracts COMMAND evo-region-batch-tests)
|
||||
set_tests_properties(region-batch-contracts PROPERTIES TIMEOUT 60)
|
||||
add_executable(evo-region-checkpoint-tests tests/region_checkpoint_tests.cpp)
|
||||
target_link_libraries(evo-region-checkpoint-tests PRIVATE evo::region_checkpoint)
|
||||
evo_owned_target(evo-region-checkpoint-tests)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: evolution-db-migrate-v4
|
||||
name: evolution-db-migrate-v5
|
||||
namespace: evolution-dev
|
||||
spec:
|
||||
backoffLimit: 0
|
||||
|
||||
@@ -542,3 +542,15 @@ adds:
|
||||
Recovery restores retained genomes, ancestry and dead organisms, and continues
|
||||
identically. Next: M7.5 durable decisions, history and outbox rows in the same
|
||||
transaction, and commit-ID resolution.
|
||||
|
||||
M7.5 implemented and locally qualified (2026-09-28), under ADR-0063.
|
||||
- Manifest v2 binds a per-tick validation trace and the decision block.
|
||||
- Staging replays the trace from the committed predecessor payload.
|
||||
- `RegionBatch` seals at a death or at 64 ticks and pauses until the commit is
|
||||
confirmed.
|
||||
- Schema v5 stores decisions and outbox rows atomically with each checkpoint,
|
||||
with one terminal outcome per actor and immutable history.
|
||||
- Exact retries return the original receipt; read-only resolution covers
|
||||
ambiguous outcomes.
|
||||
- Recovery reconciles decisions with the restored state.
|
||||
Next: M7.6 hosts bounded authoritative ticks (apps/evo-server, region_host).
|
||||
|
||||
@@ -812,3 +812,27 @@ and ASan/UBSan CTest each ran 57/57. The PostgreSQL suites passed 2/2/5 times.
|
||||
|
||||
Next: M7.5 durable decisions, history and outbox rows, and commit-ID resolution.
|
||||
Retained ancestry still needs M7.5 history and M7.7/M7.9 kill-recovery evidence.
|
||||
|
||||
## M7.5 durable batch outcomes (2026-09-28)
|
||||
|
||||
ADR-0063 introduces manifest v2, which binds a per-tick validation trace object
|
||||
and a decision block.
|
||||
- Staging replays the trace from the committed predecessor's verified payload and
|
||||
must reproduce every report and the candidate bytes.
|
||||
- `RegionBatch` seals at a death or at 64 ticks and refuses ticks until the commit
|
||||
is confirmed.
|
||||
- Schema v5 records each checkpoint, its deaths and their outbox rows in one
|
||||
statement:
|
||||
- one terminal outcome per actor;
|
||||
- committed history is immutable;
|
||||
- exact retries return the original receipt;
|
||||
- read-only resolution answers committed, absent or conflict.
|
||||
- Recovery reconciles durable deaths with the restored lifecycle.
|
||||
|
||||
Real-PostgreSQL tests cover a lost response, a timeout that later committed, a
|
||||
terminated orphan that was retried, changed bytes, repeated or partial or
|
||||
reordered decisions, gaps, outbox redelivery, and history disagreement. Five
|
||||
mutations were each caught. Windows and Linux ran 58/58; PostgreSQL suites passed
|
||||
under Release, analysis and ASan.
|
||||
|
||||
Next: M7.6 region host. No host, broker, deployment or process-kill test exists yet.
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
# ADR-0063: Batch evidence, durable outcomes and commit resolution
|
||||
|
||||
## Status
|
||||
Accepted for M7.5, 2026-09-28. Builds on ADR-0058 (full checkpoint batches),
|
||||
ADR-0060 (fencing) and ADR-0061/0062 (publication, profiles). No region host,
|
||||
broker or deployment is added; M7.6 hosts drive these APIs.
|
||||
|
||||
## Context
|
||||
ADR-0058 requires irreversible outcomes to become final only through the same
|
||||
fenced transaction as the state that produced them, with exact reconciliation of
|
||||
ambiguous commits. M7.4 published full checkpoints but recorded no decisions, no
|
||||
outbox and no batch trace. Its retries could only compare the head, and the
|
||||
database had no record of why a death happened.
|
||||
|
||||
The environmental profiles have exactly two actors, whose only permanent outcome
|
||||
is death, and deaths are terminal. The PostgreSQL adapter bounds each parameter
|
||||
to 4 KiB and each call to 16 parameters (ADR-0059).
|
||||
|
||||
## Decision
|
||||
**Manifest v2** (360 bytes) appends to v1 the trace length and SHA-256, the
|
||||
decision count and the decision-block SHA-256; one reserved field must be zero.
|
||||
The commit digest therefore binds the whole batch. v1 manifests from M7.4 commits
|
||||
still decode, but only v2 is written.
|
||||
|
||||
**Validation trace** (schemas/region-batch.md) records the ordered
|
||||
input/report pair for every tick. Its 96-byte header binds the profile, region,
|
||||
predecessor commit and digest, and the tick range, so a trace cannot be replayed
|
||||
elsewhere. Each tick is a 304-byte step, up to 19,552 bytes total. It is stored as
|
||||
an immutable object beside the payload.
|
||||
|
||||
**Staging replays** the trace from the committed predecessor's verified payload
|
||||
and must reproduce every report and the candidate bytes exactly. A missing or
|
||||
corrupt predecessor stops publication (`predecessor`).
|
||||
|
||||
**Decisions** are 24-byte records (tick, actor, outcome, archived organism),
|
||||
derived deterministically from terminal transitions. They are ordered by
|
||||
(tick, actor), with at most one per actor and ticks inside the batch. Their
|
||||
organism is 0 for environmental-v1 and the bound organism for the lineage
|
||||
profile. The block is capped at 8 records, which keeps it within one adapter
|
||||
parameter; these profiles can produce at most 2. The decision identity is
|
||||
(region, outcome, actor, tick), with no hashing needed.
|
||||
|
||||
`RegionBatch` records at most one pending batch per state:
|
||||
- It seals itself at the first terminal transition, so the whole region pauses
|
||||
at an irreversible outcome. It also seals at 64 ticks, or when the host flushes.
|
||||
- A sealed batch refuses ticks until `confirm(end_tick)`.
|
||||
- Outcomes are provisional until a receipt arrives.
|
||||
|
||||
**Schema v5**:
|
||||
- `publish_region_checkpoint(manifest, decisions)` validates the decision block
|
||||
against the manifest digest, then checks for an existing commit ID. Identical
|
||||
bytes return the original receipt without mutation or authority; anything else
|
||||
is EVL07.
|
||||
- A new commit then takes the fenced path. In the same statement it inserts the
|
||||
commit (with its decision block), the head, the decisions and the outbox rows.
|
||||
A repeated death is EVL08, backed by a unique index on terminal outcomes.
|
||||
- `resolve_region_commit` is read-only and answers committed (with the receipt),
|
||||
absent or conflict.
|
||||
- `load_region_decisions` supports recovery reconciliation.
|
||||
- `pending_region_outbox` and `acknowledge_region_outbox` give at-least-once
|
||||
delivery; acknowledgement is idempotent.
|
||||
- Triggers forbid UPDATE, DELETE and TRUNCATE of commits and decisions, and allow
|
||||
outbox rows only a first delivery timestamp (EVL09).
|
||||
- The runtime role's audited function set grows to ten.
|
||||
|
||||
**Recovery** loads decisions and requires lifecycle agreement:
|
||||
- one death per terminal actor, none for an active actor;
|
||||
- no death after the head tick, and organisms match the profile;
|
||||
- for the lineage profile, each death equals the archive's death at the same tick.
|
||||
|
||||
Disagreement quarantines the region with `history`.
|
||||
|
||||
**Ambiguity**: after `unknown`, the host keeps its batch sealed and resolves the
|
||||
exact prepared bytes.
|
||||
- Committed: adopt the receipt.
|
||||
- Absent: retry the same bytes, and only while the lease is valid.
|
||||
- Conflict: fail closed. A timeout is never treated as a rollback.
|
||||
|
||||
## Alternatives Considered
|
||||
- **The whole request in the database**: exceeds adapter bounds and duplicates
|
||||
object storage; the trace digest binds it instead.
|
||||
- **Hashed event IDs**: need SHA-256 in portable code; the structured identity is
|
||||
unique by construction.
|
||||
- **Retry by reloading the head**: cannot tell a lost response from a concurrent
|
||||
successor, and cannot detect changed bytes.
|
||||
- **Idempotency after the lease check**: would refuse a truthful receipt to a
|
||||
writer whose lease expired after its commit.
|
||||
|
||||
## Benefits
|
||||
- Deaths are durable exactly once, with their causing state.
|
||||
- Lost responses resolve without guessing.
|
||||
- The chain carries verifiable per-tick evidence.
|
||||
- Delivery can be repeated without touching the simulation.
|
||||
|
||||
## Costs
|
||||
- Every batch restores and replays its predecessor: about 7 ms in local Release
|
||||
for these payloads, versus about 4 ms without replay in M7.4.
|
||||
- Two objects are written per batch, and orphans double.
|
||||
- History tables only grow.
|
||||
|
||||
## Risks
|
||||
- The database still cannot validate the trace or the payload contents; a
|
||||
compromised host with a valid lease can publish a self-consistent fiction
|
||||
(ADR-0058).
|
||||
- Owner roles can disable the triggers.
|
||||
- The outbox has no consumer yet.
|
||||
- Profiles with births or more outcome types need their own identity and closure
|
||||
rules.
|
||||
- An absent resolution does not prove a delayed original can never arrive; the
|
||||
unique ID and head guard make the two converge.
|
||||
|
||||
## Consequences
|
||||
- M7.6 hosts must use `RegionBatch`, flush by deadline, pause while sealed, and
|
||||
resolve before any further tick after `unknown`.
|
||||
- M7.7 exercises kills at each boundary.
|
||||
- A broker or projection consumer is later operations work and must deduplicate
|
||||
by the decision identity.
|
||||
- Manifest v3, or new decision kinds, require new profile or history values and a
|
||||
migration.
|
||||
@@ -73,3 +73,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
|
||||
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
|
||||
- [ADR-0062: Lineage checkpoint profile](0062-lineage-checkpoint-profile.md) - accepted for M7.4b archive codec, LineageRegion closure and history-capable profile 2.
|
||||
- [ADR-0063: Batch outcomes](0063-batch-outcomes.md) - accepted for M7.5 manifest v2, validation trace replay, durable decisions/outbox and commit resolution.
|
||||
|
||||
@@ -186,3 +186,18 @@ archive codec and the `environmental-lineage-v1` profile validate:
|
||||
- the living set, identity high-waters and retained dead members.
|
||||
Recovery restores all of these. Durable history rows (M7.5) and process-kill
|
||||
recovery (M7.7/M7.9) remain before the retained-ancestry exit claim.
|
||||
|
||||
M7.5 (ADR-0063) executed its rows against real PostgreSQL:
|
||||
- A rejected publication (repeated death, partial or reordered decision block,
|
||||
gap) leaves the head, commits and decisions unchanged. Publication is a single
|
||||
statement; an injected failure between insert and commit is not yet tested.
|
||||
- A lost response resolves to the original receipt, and an exact retry adds no
|
||||
second death.
|
||||
- A client timeout while queued resolved as committed after the lock cleared,
|
||||
and was adopted.
|
||||
- A terminated orphan resolved as absent and was retried with the same bytes.
|
||||
- Changed bytes conflict.
|
||||
- Outbox redelivery is idempotent and never touches the simulation.
|
||||
Recovery now reconciles durable deaths with the restored lifecycle, so the
|
||||
checkpoint and durable history must agree. Kill tests at each boundary remain
|
||||
M7.7.
|
||||
|
||||
@@ -2213,3 +2213,55 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
- checkpoint publication 3.5-5.0 ms for 23-37 KiB payloads.
|
||||
- Not executed: Docker/Gitea CI, remote storage, process kill, reproduction within
|
||||
a batch (the fixture has none), durable history rows (M7.5) and TSan.
|
||||
|
||||
## M7.5 durable batch outcomes - 2026-09-28
|
||||
|
||||
- Windows MSVC Release CTest passed 58/58, including the new `region-batch-contracts`:
|
||||
- recorder sealing at a death or at 64 ticks, early flush, confirm matching, and
|
||||
a refused tick recording nothing;
|
||||
- trace round trip, size (19,552 bytes at 64 ticks), range, contiguity and
|
||||
header binding;
|
||||
- decision order, uniqueness, range, and refusal of unknown values.
|
||||
The manifest v2 golden layout, v1 decoding and batch-evidence structural rules
|
||||
were also covered.
|
||||
- WSL Linux: the `checks` build set (GCC14 Debug and Release, clang19 analysis,
|
||||
clang19 ASan/UBSan) passed CTest 58/58 each, and the repository format check
|
||||
passed.
|
||||
- The first PostgreSQL clang-tidy build flagged a returned const-reference
|
||||
parameter and two unchecked optional accesses; all three were fixed. Final
|
||||
PostgreSQL suites: Release 2 runs, analysis 2, ASan/UBSan 4, all passing.
|
||||
- Real-database coverage:
|
||||
- Durable death at tick 20, with decisions and outbox rows in the same commit.
|
||||
- Outbox: redelivered, acknowledged true then false, then empty; invalid limit
|
||||
and unknown rows refused.
|
||||
- A lost response retried with identical bytes returns the same receipt and
|
||||
records no second death; resolution says committed.
|
||||
- A changed manifest or changed decisions conflict (EVL07/22023). Other regions
|
||||
and unknown IDs resolve as conflict and absent.
|
||||
- A repeated death is EVL08. A forged same-size block, a partial block and a
|
||||
reordered block are 22023. A gap is EVL03. None of these moved the head or
|
||||
decisions.
|
||||
- A 300 ms client timeout while queued on the lease row became unknown; once the
|
||||
lock cleared the server committed, resolution returned committed and the retry
|
||||
was idempotent.
|
||||
- A terminated orphan request resolved as absent, and the same bytes were
|
||||
retried and committed.
|
||||
- A missing predecessor payload is `predecessor`. State advanced outside the
|
||||
batch fails replay. Batches from the wrong predecessor and unsealed batches are
|
||||
refused before upload.
|
||||
- Recovery reconciles decisions. An owner-inserted death the state does not
|
||||
show quarantines with `history`, and cannot be deleted (EVL09).
|
||||
- Commit and decision UPDATE and DELETE are EVL09 even for the owner; the
|
||||
defense-in-depth test disables and restores the trigger explicitly.
|
||||
- Lineage profile: a death is recorded with the archived organism id and
|
||||
reconciled against the archive.
|
||||
- Migrations: clean→v5, v1→v5 and v3→v5 with an existing v1-manifest commit;
|
||||
tampered checksums for v2 through v5 and an unknown v6 are refused.
|
||||
- Mutations (scratch SQL copies; host source restored and rerun green), each
|
||||
caught: removing idempotent retry, outcome finality, the decision-digest check,
|
||||
decision recording, or recovery reconciliation.
|
||||
- Local measurements, including predecessor replay: Release 6.6-8.1 ms per batch;
|
||||
analysis about 20-23 ms; ASan about 19-21 ms. Manifest publish p50 about
|
||||
0.75-0.86 ms.
|
||||
- Not executed: Docker/Gitea CI, remote object store, process kill, a broker
|
||||
consumer, an injected failure between commit insert and statement end, and TSan.
|
||||
|
||||
@@ -68,7 +68,7 @@ every role transitively in it, and rolls back on any violation:
|
||||
`pg_write_all_data`);
|
||||
- any table privilege in `evolution`, including via PUBLIC;
|
||||
- CREATE on the schema, the database or `public`;
|
||||
- EXECUTE outside the six fenced functions (schema v3).
|
||||
- EXECUTE outside the ten fenced functions (schema v5).
|
||||
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
||||
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
||||
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
||||
@@ -112,6 +112,16 @@ payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
|
||||
profile, history, RNG and durability declarations. Provision history-capable
|
||||
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
|
||||
|
||||
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
|
||||
- `region_decisions` and `region_outbox`, with one terminal outcome per actor;
|
||||
- `resolve_region_commit`, `load_region_decisions`, `pending_region_outbox` and
|
||||
`acknowledge_region_outbox`, bringing the runtime function set to ten.
|
||||
Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
|
||||
a first delivery time. Owners can disable triggers, so treat that as an audited
|
||||
break-glass action. EVL08 means an outcome was already final. After an
|
||||
ambiguous publication, resolve the exact prepared bytes before doing anything
|
||||
else.
|
||||
|
||||
There is no operator CLI for provisioning yet, and no real world has been
|
||||
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
have no payload.
|
||||
|
||||
@@ -1,30 +1,49 @@
|
||||
#pragma once
|
||||
#include "evo/persistence/object_store.hpp"
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include "evo/persistence/region_checkpoint.hpp"
|
||||
#include "evo/persistence/region_lease_store.hpp"
|
||||
#include <chrono>
|
||||
#include <expected>
|
||||
#include <memory>
|
||||
#include <span>
|
||||
#include <vector>
|
||||
|
||||
// Linux publication and recovery of environmental-v1 checkpoints (ADR-0061).
|
||||
// Order: validate candidate -> immutable upload -> verified read-back -> fenced
|
||||
// manifest transaction. Nothing before the transaction has authority; a failure
|
||||
// there leaves the previous head and at most an unreferenced orphan object.
|
||||
// Linux publication and recovery of region checkpoints (ADR-0061/0062/0063).
|
||||
// Batch order: replay the recorded trace from the committed predecessor payload
|
||||
// and require the candidate bytes -> derive decisions -> immutable payload and
|
||||
// trace uploads with verified read-back -> one fenced transaction recording
|
||||
// checkpoint, decisions and outbox. Nothing before the transaction has authority:
|
||||
// a failure there leaves the previous head and at most unreferenced orphan objects.
|
||||
// Permanent outcomes are final only when a receipt (publish or resolve) confirms them.
|
||||
namespace evo::persistence {
|
||||
enum class PublicationError : std::uint8_t {
|
||||
candidate, // State/manifest not a valid successor; nothing uploaded.
|
||||
digest, // Digest provider failure; nothing uploaded.
|
||||
upload, // Object write failed; no manifest was sent.
|
||||
verify, // Read-back differs from the candidate; no manifest was sent.
|
||||
rejected, // Database refused (see store); the upload is an orphan.
|
||||
unknown, // Outcome unknown (see store); stay paused and reconcile the head.
|
||||
receipt, // Committed receipt disagrees with the local manifest: fail closed.
|
||||
candidate, // State, batch or manifest is not a valid successor; nothing uploaded.
|
||||
predecessor, // Committed predecessor payload missing/corrupt/invalid: quarantine.
|
||||
digest, // Digest provider failure; nothing uploaded.
|
||||
upload, // Object write failed; no manifest was sent.
|
||||
verify, // Read-back differs from the candidate; no manifest was sent.
|
||||
rejected, // Database refused (see store); uploads are orphans.
|
||||
unknown, // Outcome unknown (see store): stay paused; resolve `prepared`.
|
||||
receipt, // Committed receipt disagrees with the local manifest: fail closed.
|
||||
};
|
||||
// The exact bytes sent to the database. After `unknown`, resolve or retry only these.
|
||||
struct PreparedCommit {
|
||||
RegionKey key{};
|
||||
Uuid commit{};
|
||||
std::vector<std::uint8_t> manifest;
|
||||
std::vector<std::uint8_t> decisions;
|
||||
};
|
||||
struct PublicationFailure {
|
||||
PublicationError kind{};
|
||||
LeaseStoreFailure store{}; // Meaningful for rejected/unknown.
|
||||
ObjectError object{}; // Meaningful for upload/verify.
|
||||
ObjectError object{}; // Meaningful for upload/verify/predecessor.
|
||||
PreparedCommit prepared; // Present for rejected/unknown.
|
||||
};
|
||||
// Durably confirmed batch: its decisions are now final and may be acknowledged.
|
||||
struct PublishedBatch {
|
||||
CommittedCheckpoint committed;
|
||||
std::vector<BatchDecision> decisions;
|
||||
};
|
||||
enum class RecoveryError : std::uint8_t {
|
||||
store, // Database unavailable/failed (see store).
|
||||
@@ -34,6 +53,7 @@ enum class RecoveryError : std::uint8_t {
|
||||
missing_payload, // Referenced object absent: quarantine, no older head.
|
||||
corrupt_payload, // Length/digest mismatch: quarantine, no older head.
|
||||
payload, // Verified bytes fail full state validation: quarantine.
|
||||
history, // Durable decisions disagree with the restored lifecycle: quarantine.
|
||||
};
|
||||
struct RecoveryFailure {
|
||||
RecoveryError kind{};
|
||||
@@ -43,11 +63,13 @@ struct RecoveredCheckpoint {
|
||||
CommittedCheckpoint committed;
|
||||
CheckpointManifest manifest;
|
||||
std::unique_ptr<processes::EnvironmentalCommunity> state;
|
||||
std::vector<CommittedDecision> decisions;
|
||||
};
|
||||
struct RecoveredLineageCheckpoint {
|
||||
CommittedCheckpoint committed;
|
||||
CheckpointManifest manifest;
|
||||
std::unique_ptr<processes::LineageRegion> state;
|
||||
std::vector<CommittedDecision> decisions;
|
||||
};
|
||||
|
||||
// Owner/operator connection only: uploads a tick-0 state and records genesis.
|
||||
@@ -55,17 +77,17 @@ struct RecoveredLineageCheckpoint {
|
||||
Postgres& owner, const FilesystemObjectStore& objects, RegionKey key, std::uint64_t frame,
|
||||
const processes::EnvironmentalCommunity& opening, Uuid commit, const Sha256Digest& build,
|
||||
std::size_t restore_budget, std::chrono::milliseconds timeout);
|
||||
// Publishes `state` as the successor of `predecessor` under `lease`. The caller
|
||||
// must hold the committed predecessor it simulated from and a lease that may
|
||||
// begin a commit (LeaseAuthority::may_begin_commit).
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, PublicationFailure>
|
||||
publish_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
|
||||
const processes::EnvironmentalCommunity& state, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Loads and fully verifies the committed head. Readiness additionally requires a
|
||||
// fresh valid lease acquired before this call (M7.7); this does not grant one.
|
||||
// Publishes a sealed batch that began at `predecessor`, under `lease` (which the
|
||||
// caller has checked with LeaseAuthority::may_begin_commit). On success call
|
||||
// batch.confirm(end_tick); only then are its decisions final.
|
||||
[[nodiscard]] std::expected<PublishedBatch, PublicationFailure>
|
||||
publish_environmental_batch(Postgres& db, const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::EnvironmentalCommunity>& batch,
|
||||
Uuid commit, const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout);
|
||||
// Loads and fully verifies the committed head and reconciles durable decisions.
|
||||
// Readiness additionally requires a fresh valid lease acquired before this call.
|
||||
[[nodiscard]] std::expected<RecoveredCheckpoint, RecoveryFailure>
|
||||
recover_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
@@ -79,12 +101,22 @@ provision_lineage_region(Postgres& owner, const FilesystemObjectStore& objects,
|
||||
std::uint64_t frame, const processes::LineageRegion& opening, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, PublicationFailure> publish_lineage_checkpoint(
|
||||
Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const processes::LineageRegion& state, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget, std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<PublishedBatch, PublicationFailure>
|
||||
publish_lineage_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::LineageRegion>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget,
|
||||
std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<RecoveredLineageCheckpoint, RecoveryFailure>
|
||||
recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
std::size_t restore_budget, std::chrono::milliseconds timeout);
|
||||
|
||||
// After an `unknown` publication: resolve exactly the prepared bytes. A committed
|
||||
// result yields the final batch; absent may be retried with the same prepared bytes
|
||||
// only while the lease is still valid; conflict fails closed.
|
||||
[[nodiscard]] std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_prepared(Postgres& db, const PreparedCommit& prepared, std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<RegionHead, LeaseStoreFailure>
|
||||
retry_prepared(Postgres& db, const PreparedCommit& prepared, std::chrono::milliseconds timeout);
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
#pragma once
|
||||
#include "evo/persistence/region_lease.hpp"
|
||||
#include "evo/processes/environmental_community.hpp"
|
||||
#include <array>
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <span>
|
||||
#include <vector>
|
||||
|
||||
// M7.5 batch evidence (ADR-0063, schemas/region-batch.md): the ordered validation
|
||||
// trace of host inputs and reports for one checkpoint batch, and the permanent
|
||||
// outcomes (decisions) derived from it. Portable and storage-free.
|
||||
namespace evo::persistence {
|
||||
enum class BatchError : std::uint8_t {
|
||||
format, // Malformed trace or decision bytes.
|
||||
sealed, // The batch holds a pending permanent outcome or is full.
|
||||
empty, // Nothing to commit.
|
||||
state, // The simulation refused the tick; nothing was recorded.
|
||||
mismatch, // Replay, receipt or reconciliation disagrees with the record.
|
||||
};
|
||||
// Encoded as u32; decoding refuses unknown values rather than truncating them.
|
||||
enum class Actor : std::uint8_t { producer = 0, consumer = 1 };
|
||||
enum class Outcome : std::uint8_t { death = 1 };
|
||||
|
||||
// One permanent outcome. Identity is (region, outcome, actor, tick); a death is
|
||||
// terminal, so each actor has at most one. organism is the archived organism for
|
||||
// the lineage profile and 0 for environmental-v1.
|
||||
struct BatchDecision {
|
||||
std::uint64_t tick{};
|
||||
Actor actor{};
|
||||
Outcome outcome{Outcome::death};
|
||||
std::uint64_t organism{};
|
||||
friend bool operator==(const BatchDecision&, const BatchDecision&) = default;
|
||||
};
|
||||
struct TraceStep {
|
||||
processes::EnvironmentalInput input{};
|
||||
processes::EnvironmentalReport report{};
|
||||
friend bool operator==(const TraceStep&, const TraceStep&) = default;
|
||||
};
|
||||
// Binds a trace to its place in the chain so it cannot be replayed elsewhere.
|
||||
struct TraceHeader {
|
||||
std::uint32_t profile{};
|
||||
RegionKey key{};
|
||||
Uuid predecessor_commit{};
|
||||
Sha256Digest predecessor_digest{};
|
||||
std::uint64_t begin_tick{}, end_tick{};
|
||||
friend bool operator==(const TraceHeader&, const TraceHeader&) = default;
|
||||
};
|
||||
|
||||
inline constexpr std::size_t trace_header_bytes = 96;
|
||||
inline constexpr std::size_t trace_step_bytes = 304;
|
||||
inline constexpr std::size_t decision_bytes = 24;
|
||||
// Two actors, each dies at most once; the wire cap leaves room for later profiles.
|
||||
inline constexpr std::size_t max_batch_decisions = 8;
|
||||
[[nodiscard]] constexpr std::size_t trace_bytes_for(std::uint64_t ticks) noexcept {
|
||||
return trace_header_bytes + (static_cast<std::size_t>(ticks) * trace_step_bytes);
|
||||
}
|
||||
|
||||
// One step per tick, begin_tick+1..end_tick, at most 64.
|
||||
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_trace(const TraceHeader& header, std::span<const TraceStep> steps);
|
||||
struct DecodedTrace {
|
||||
TraceHeader header;
|
||||
std::vector<TraceStep> steps;
|
||||
};
|
||||
[[nodiscard]] std::expected<DecodedTrace, BatchError>
|
||||
decode_trace(std::span<const std::uint8_t> bytes);
|
||||
|
||||
// Sorted by (tick, actor), unique actors, death only, ticks within the batch.
|
||||
[[nodiscard]] std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_decisions(std::span<const BatchDecision> decisions, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick);
|
||||
[[nodiscard]] std::expected<std::vector<BatchDecision>, BatchError>
|
||||
decode_decisions(std::span<const std::uint8_t> bytes, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick);
|
||||
// Every terminal transition in the trace yields exactly one decision, in order.
|
||||
[[nodiscard]] std::vector<BatchDecision> derive_decisions(std::span<const TraceStep> steps,
|
||||
std::uint64_t producer_organism,
|
||||
std::uint64_t consumer_organism);
|
||||
|
||||
// Records one pending batch over a single-owner state (EnvironmentalCommunity or
|
||||
// LineageRegion). Seals itself after any tick with a terminal transition, or at
|
||||
// 64 ticks; a sealed batch refuses further ticks until confirm() records durable
|
||||
// confirmation. Nothing here is durable: outcomes are provisional until then.
|
||||
template <typename State> class RegionBatch {
|
||||
public:
|
||||
explicit RegionBatch(State& state) : state_(&state), begin_(tick_of(state)) {}
|
||||
[[nodiscard]] std::expected<processes::EnvironmentalReport, BatchError>
|
||||
advance(const processes::EnvironmentalInput& input) {
|
||||
if (sealed_)
|
||||
return std::unexpected{BatchError::sealed};
|
||||
const auto tick = tick_of(*state_) + 1;
|
||||
auto report = state_->advance(tick, input);
|
||||
if (!report)
|
||||
return std::unexpected{BatchError::state};
|
||||
steps_.push_back({input, *report});
|
||||
if (report->terminal_transition == 1 || report->consumer_transition == 1 ||
|
||||
steps_.size() == max_ticks)
|
||||
sealed_ = true;
|
||||
return *report;
|
||||
}
|
||||
// Host deadline or shutdown: stop and commit what exists.
|
||||
void seal() noexcept {
|
||||
sealed_ = true;
|
||||
}
|
||||
// Durable confirmation for exactly this batch: the next batch starts here.
|
||||
[[nodiscard]] std::expected<void, BatchError> confirm(std::uint64_t committed_end_tick) {
|
||||
if (!sealed_ || steps_.empty() || committed_end_tick != end_tick())
|
||||
return std::unexpected{BatchError::mismatch};
|
||||
begin_ = committed_end_tick;
|
||||
steps_.clear();
|
||||
sealed_ = false;
|
||||
return {};
|
||||
}
|
||||
[[nodiscard]] bool sealed() const noexcept {
|
||||
return sealed_;
|
||||
}
|
||||
[[nodiscard]] std::uint64_t begin_tick() const noexcept {
|
||||
return begin_;
|
||||
}
|
||||
[[nodiscard]] std::uint64_t end_tick() const noexcept {
|
||||
return begin_ + steps_.size();
|
||||
}
|
||||
[[nodiscard]] std::span<const TraceStep> steps() const noexcept {
|
||||
return steps_;
|
||||
}
|
||||
[[nodiscard]] const State& state() const noexcept {
|
||||
return *state_;
|
||||
}
|
||||
|
||||
private:
|
||||
static constexpr std::size_t max_ticks = 64;
|
||||
static std::uint64_t tick_of(const State& s) {
|
||||
if constexpr (requires { s.community(); })
|
||||
return s.community().progress().tick;
|
||||
else
|
||||
return s.progress().tick;
|
||||
}
|
||||
State* state_;
|
||||
std::uint64_t begin_;
|
||||
std::vector<TraceStep> steps_;
|
||||
bool sealed_{};
|
||||
};
|
||||
} // namespace evo::persistence
|
||||
@@ -1,4 +1,5 @@
|
||||
#pragma once
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include "evo/persistence/region_lease.hpp"
|
||||
#include "evo/processes/environmental_community.hpp"
|
||||
#include "evo/processes/lineage_region.hpp"
|
||||
@@ -34,7 +35,18 @@ struct VersionBundle {
|
||||
};
|
||||
inline constexpr VersionBundle environmental_v1_versions{};
|
||||
|
||||
// Batch evidence bound by manifest v2 (ADR-0063); all zero at genesis.
|
||||
struct BatchDigests {
|
||||
std::uint64_t trace_bytes{};
|
||||
Sha256Digest trace_sha256{};
|
||||
std::uint32_t decision_count{};
|
||||
Sha256Digest decisions_sha256{}; // Zero exactly when there are no decisions.
|
||||
friend bool operator==(const BatchDigests&, const BatchDigests&) = default;
|
||||
};
|
||||
|
||||
struct CheckpointManifest {
|
||||
// 2 is the only version written; 1 (M7.4, no batch evidence) still decodes.
|
||||
std::uint32_t manifest_version{2};
|
||||
CheckpointProfile profile{CheckpointProfile::environmental_v1};
|
||||
RegionKey key{};
|
||||
std::uint64_t frame{};
|
||||
@@ -53,10 +65,12 @@ struct CheckpointManifest {
|
||||
DurabilityPolicy durability{DurabilityPolicy::single_volume_development};
|
||||
std::uint64_t payload_bytes{};
|
||||
Sha256Digest payload_sha256{};
|
||||
BatchDigests batch{};
|
||||
friend bool operator==(const CheckpointManifest&, const CheckpointManifest&) = default;
|
||||
};
|
||||
|
||||
inline constexpr std::size_t checkpoint_manifest_bytes = 280;
|
||||
inline constexpr std::size_t checkpoint_manifest_v1_bytes = 280;
|
||||
inline constexpr std::size_t checkpoint_manifest_bytes = 360;
|
||||
inline constexpr std::size_t max_environmental_payload_bytes = std::size_t{1024} * 1024;
|
||||
inline constexpr std::size_t max_lineage_payload_bytes = processes::max_lineage_region_bytes;
|
||||
// Payload cap by profile; zero for an unknown profile.
|
||||
@@ -95,7 +109,7 @@ genesis_manifest(RegionKey key, std::uint64_t frame, Uuid commit, std::uint64_t
|
||||
successor_manifest(const CheckpointManifest& predecessor, const Sha256Digest& predecessor_digest,
|
||||
Uuid commit, const LeaseGrant& lease, std::uint64_t end_tick,
|
||||
const Sha256Digest& build, std::uint64_t payload_bytes,
|
||||
const Sha256Digest& payload_sha256);
|
||||
const Sha256Digest& payload_sha256, const BatchDigests& batch);
|
||||
|
||||
// Fully restores payload bytes (envelope checksum, bounds and replay audit), then
|
||||
// requires manifest identity, end tick, tick quantum, envelope format 2 and a
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
#pragma once
|
||||
#include "evo/persistence/postgres.hpp"
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include "evo/persistence/region_lease.hpp"
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
@@ -18,7 +19,9 @@ enum class LeaseStoreError : std::uint8_t {
|
||||
not_provisioned, // No genesis; runtime can never create one.
|
||||
exhausted, // Epoch or commit sequence would wrap: fail closed.
|
||||
unsupported, // Profile/version/build not admitted, or rules/quantum changed.
|
||||
commit_conflict, // Commit ID already used by another manifest (resolve at M7.5).
|
||||
commit_conflict, // Commit ID already holds other bytes: never retry it.
|
||||
outcome_final, // A decision repeats an already committed permanent outcome.
|
||||
immutable, // Attempted rewrite of committed history (owner paths only).
|
||||
invalid, // Rejected arguments; nothing changed.
|
||||
store, // Other SQL/configuration failure; nothing committed.
|
||||
unknown, // May have executed; reconnect and reconcile before reuse.
|
||||
@@ -47,11 +50,45 @@ release_region_lease(Postgres& db, RegionKey key, const LeaseGrant& lease,
|
||||
load_region_head(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
|
||||
[[nodiscard]] std::expected<CommittedCheckpoint, LeaseStoreFailure>
|
||||
load_region_checkpoint(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
|
||||
// Canonical manifest v1 bytes; the database parses identity, lease and chain from
|
||||
// them, locks the lease then the head and requires the exact committed predecessor.
|
||||
// The payload must already be verified in the object store. head_mismatch after
|
||||
// unknown: load the head and compare its commit ID (full resolution is M7.5).
|
||||
// Canonical manifest v2 bytes plus the decision block they digest (ADR-0063). The
|
||||
// database parses identity, lease and chain, locks the lease then the head, requires
|
||||
// the exact committed predecessor and records decisions and outbox rows in the same
|
||||
// statement. An exact retry of a committed ID returns its original receipt; changed
|
||||
// bytes are commit_conflict. The payload and trace must already be verified objects.
|
||||
[[nodiscard]] std::expected<RegionHead, LeaseStoreFailure>
|
||||
publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions,
|
||||
std::chrono::milliseconds timeout);
|
||||
|
||||
enum class Resolution : std::uint8_t {
|
||||
committed, // Exact bytes are durable; receipt holds the head. Grants no authority.
|
||||
absent, // Not committed now; proves nothing about a request still in flight.
|
||||
conflict, // The ID holds other bytes or belongs to another region: fail closed.
|
||||
};
|
||||
struct ResolvedCommit {
|
||||
Resolution status{};
|
||||
RegionHead receipt{}; // Only for committed.
|
||||
};
|
||||
[[nodiscard]] std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions, std::chrono::milliseconds timeout);
|
||||
|
||||
struct CommittedDecision {
|
||||
std::uint64_t commit_sequence{};
|
||||
BatchDecision decision{};
|
||||
friend bool operator==(const CommittedDecision&, const CommittedDecision&) = default;
|
||||
};
|
||||
// Every committed permanent outcome of the region, ordered by (tick, actor).
|
||||
[[nodiscard]] std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
load_region_decisions(Postgres& db, RegionKey key, std::chrono::milliseconds timeout);
|
||||
// Undelivered outbox rows (1..64), ordered by (commit, tick, actor). Delivery is
|
||||
// at-least-once; consumers deduplicate by (region, outcome, actor, tick).
|
||||
[[nodiscard]] std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
pending_region_outbox(Postgres& db, RegionKey key, std::uint32_t limit,
|
||||
std::chrono::milliseconds timeout);
|
||||
// True on first acknowledgement, false when already delivered.
|
||||
[[nodiscard]] std::expected<bool, LeaseStoreFailure>
|
||||
acknowledge_region_outbox(Postgres& db, RegionKey key, const BatchDecision& decision,
|
||||
std::chrono::milliseconds timeout);
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -0,0 +1,396 @@
|
||||
-- M7.5 durable batch outcomes (ADR-0063). Manifest v2 binds the validation trace
|
||||
-- object and the decision block by SHA-256. Publication records the checkpoint,
|
||||
-- its permanent decisions and their outbox rows in one statement; an exact retry
|
||||
-- of a committed ID returns its receipt, changed bytes conflict. Committed history
|
||||
-- is immutable to every caller that does not first disable these triggers.
|
||||
|
||||
ALTER TABLE evolution.region_commits DROP CONSTRAINT region_commits_manifest_check;
|
||||
ALTER TABLE evolution.region_commits
|
||||
ADD CONSTRAINT region_commits_manifest_check CHECK (octet_length(manifest) IN (280, 360)),
|
||||
ADD COLUMN decisions bytea NOT NULL DEFAULT '\x'::bytea
|
||||
CHECK (octet_length(decisions) % 24 = 0 AND octet_length(decisions) <= 192);
|
||||
|
||||
ALTER TYPE evolution.checkpoint_manifest
|
||||
ADD ATTRIBUTE profile integer,
|
||||
ADD ATTRIBUTE trace_bytes evolution.u64,
|
||||
ADD ATTRIBUTE trace_sha256 bytea,
|
||||
ADD ATTRIBUTE decision_count integer,
|
||||
ADD ATTRIBUTE decisions_sha256 bytea;
|
||||
|
||||
-- Permanent outcomes. A death is terminal: one per actor per region, ever.
|
||||
CREATE TABLE evolution.region_decisions (
|
||||
world_id evolution.u64 NOT NULL,
|
||||
region_id evolution.u64 NOT NULL,
|
||||
outcome smallint NOT NULL CHECK (outcome = 1),
|
||||
actor smallint NOT NULL CHECK (actor IN (0, 1)),
|
||||
tick evolution.u64 NOT NULL CHECK (tick > 0),
|
||||
organism evolution.u64 NOT NULL,
|
||||
commit_sequence evolution.u64 NOT NULL,
|
||||
PRIMARY KEY (world_id, region_id, outcome, actor, tick),
|
||||
FOREIGN KEY (world_id, region_id, commit_sequence) REFERENCES evolution.region_commits
|
||||
);
|
||||
CREATE UNIQUE INDEX region_decisions_terminal
|
||||
ON evolution.region_decisions (world_id, region_id, outcome, actor) WHERE outcome = 1;
|
||||
-- Delivery queue derived from decisions; delivery never mutates simulation state.
|
||||
CREATE TABLE evolution.region_outbox (
|
||||
world_id evolution.u64 NOT NULL,
|
||||
region_id evolution.u64 NOT NULL,
|
||||
outcome smallint NOT NULL,
|
||||
actor smallint NOT NULL,
|
||||
tick evolution.u64 NOT NULL,
|
||||
commit_sequence evolution.u64 NOT NULL,
|
||||
delivered_at timestamptz,
|
||||
PRIMARY KEY (world_id, region_id, outcome, actor, tick),
|
||||
FOREIGN KEY (world_id, region_id, outcome, actor, tick) REFERENCES evolution.region_decisions
|
||||
);
|
||||
REVOKE ALL ON evolution.region_decisions, evolution.region_outbox FROM PUBLIC;
|
||||
|
||||
-- No GC in initial M7: committed history and outcomes are never rewritten or removed.
|
||||
CREATE FUNCTION evolution.refuse_history_rewrite() RETURNS trigger LANGUAGE plpgsql
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
RAISE EXCEPTION 'committed history is immutable' USING ERRCODE = 'EVL09';
|
||||
END $$;
|
||||
CREATE TRIGGER region_commits_immutable BEFORE UPDATE OR DELETE ON evolution.region_commits
|
||||
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
CREATE TRIGGER region_commits_no_truncate BEFORE TRUNCATE ON evolution.region_commits
|
||||
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
CREATE TRIGGER region_decisions_immutable BEFORE UPDATE OR DELETE ON evolution.region_decisions
|
||||
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
CREATE TRIGGER region_decisions_no_truncate BEFORE TRUNCATE ON evolution.region_decisions
|
||||
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
-- Outbox rows may only record delivery; their identity and existence are permanent.
|
||||
CREATE FUNCTION evolution.refuse_outbox_rewrite() RETURNS trigger LANGUAGE plpgsql
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF TG_OP = 'UPDATE' AND (NEW.world_id, NEW.region_id, NEW.outcome, NEW.actor, NEW.tick,
|
||||
NEW.commit_sequence)
|
||||
IS NOT DISTINCT FROM
|
||||
(OLD.world_id, OLD.region_id, OLD.outcome, OLD.actor, OLD.tick,
|
||||
OLD.commit_sequence)
|
||||
AND OLD.delivered_at IS NULL AND NEW.delivered_at IS NOT NULL THEN
|
||||
RETURN NEW;
|
||||
END IF;
|
||||
RAISE EXCEPTION 'outbox rows only record first delivery' USING ERRCODE = 'EVL09';
|
||||
END $$;
|
||||
CREATE TRIGGER region_outbox_immutable BEFORE UPDATE OR DELETE ON evolution.region_outbox
|
||||
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_outbox_rewrite();
|
||||
CREATE TRIGGER region_outbox_no_truncate BEFORE TRUNCATE ON evolution.region_outbox
|
||||
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
|
||||
|
||||
-- Manifest v2 only for new records (v1 records stay readable). Mirrors
|
||||
-- decode_manifest + supported() in src/persistence/region_checkpoint.cpp.
|
||||
CREATE OR REPLACE FUNCTION evolution.parse_checkpoint_manifest(m bytea)
|
||||
RETURNS evolution.checkpoint_manifest LANGUAGE plpgsql IMMUTABLE
|
||||
SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
r evolution.checkpoint_manifest;
|
||||
zero16 constant bytea := '\x00000000000000000000000000000000';
|
||||
zero32 constant bytea := decode(repeat('00', 32), 'hex');
|
||||
history numeric;
|
||||
cap numeric;
|
||||
BEGIN
|
||||
IF m IS NULL OR octet_length(m) < 12 OR substring(m FROM 1 FOR 8) <> '\x4145564f52434d31' THEN
|
||||
RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF evolution.manifest_uint(m, 8, 4) <> 2 THEN
|
||||
RAISE EXCEPTION 'unsupported checkpoint manifest' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
IF octet_length(m) <> 360 THEN
|
||||
RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
r.profile := evolution.manifest_uint(m, 12, 4);
|
||||
history := evolution.manifest_uint(m, 232, 4);
|
||||
IF NOT ((r.profile = 1 AND history = 0) OR (r.profile = 2 AND history = 1))
|
||||
OR substring(m FROM 161 FOR 36) <> '\x020000000200000001000000010000000100000001000000010000000100000001000000'
|
||||
OR evolution.manifest_uint(m, 228, 4) <> 0 OR evolution.manifest_uint(m, 236, 4) <> 1 THEN
|
||||
RAISE EXCEPTION 'unsupported checkpoint manifest' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
-- Payload caps: 1 MiB envelope; 72 + 1 MiB + lineage archive cap (3219496) container.
|
||||
cap := CASE r.profile WHEN 1 THEN 1048576 ELSE 4268144 END;
|
||||
r.world_id := evolution.manifest_uint(m, 16, 8);
|
||||
r.region_id := evolution.manifest_uint(m, 24, 8);
|
||||
r.frame_id := evolution.manifest_uint(m, 32, 8);
|
||||
r.commit_id := encode(substring(m FROM 41 FOR 16), 'hex')::uuid;
|
||||
r.commit_sequence := evolution.manifest_uint(m, 56, 8);
|
||||
r.predecessor_commit := encode(substring(m FROM 65 FOR 16), 'hex')::uuid;
|
||||
r.predecessor_digest := substring(m FROM 81 FOR 32);
|
||||
r.begin_tick := evolution.manifest_uint(m, 112, 8);
|
||||
r.end_tick := evolution.manifest_uint(m, 120, 8);
|
||||
r.holder := encode(substring(m FROM 129 FOR 16), 'hex')::uuid;
|
||||
r.epoch := evolution.manifest_uint(m, 144, 8);
|
||||
r.tick_microseconds := evolution.manifest_uint(m, 152, 8);
|
||||
r.versions := substring(m FROM 161 FOR 36);
|
||||
r.build_identity := substring(m FROM 197 FOR 32);
|
||||
r.payload_bytes := evolution.manifest_uint(m, 240, 8);
|
||||
r.payload_sha256 := substring(m FROM 249 FOR 32);
|
||||
r.trace_bytes := evolution.manifest_uint(m, 280, 8);
|
||||
r.trace_sha256 := substring(m FROM 289 FOR 32);
|
||||
r.decision_count := evolution.manifest_uint(m, 320, 4);
|
||||
r.decisions_sha256 := substring(m FROM 329 FOR 32);
|
||||
IF r.world_id = 0 OR r.region_id = 0 OR r.frame_id = 0
|
||||
OR substring(m FROM 41 FOR 16) = zero16 OR r.tick_microseconds = 0
|
||||
OR r.build_identity = zero32 OR r.payload_bytes < 1 OR r.payload_bytes > cap
|
||||
OR r.payload_sha256 = zero32 OR evolution.manifest_uint(m, 324, 4) <> 0 THEN
|
||||
RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
IF r.commit_sequence = 0 THEN
|
||||
IF substring(m FROM 65 FOR 16) <> zero16 OR r.predecessor_digest <> zero32
|
||||
OR r.begin_tick <> 0 OR r.end_tick <> 0 OR substring(m FROM 129 FOR 16) <> zero16
|
||||
OR r.epoch <> 0 OR r.trace_bytes <> 0 OR r.trace_sha256 <> zero32
|
||||
OR r.decision_count <> 0 OR r.decisions_sha256 <> zero32 THEN
|
||||
RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
r.predecessor_commit := NULL;
|
||||
r.holder := NULL;
|
||||
ELSIF substring(m FROM 65 FOR 16) = zero16 OR r.predecessor_commit = r.commit_id
|
||||
OR r.predecessor_digest = zero32 OR substring(m FROM 129 FOR 16) = zero16
|
||||
OR r.epoch = 0 OR r.end_tick <= r.begin_tick OR r.end_tick - r.begin_tick > 64
|
||||
-- One 304-byte trace step per tick after a 96-byte header.
|
||||
OR r.trace_bytes <> 96 + 304 * (r.end_tick - r.begin_tick) OR r.trace_sha256 = zero32
|
||||
OR r.decision_count > 8 OR (r.decision_count = 0) <> (r.decisions_sha256 = zero32) THEN
|
||||
RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
RETURN r;
|
||||
END $$;
|
||||
|
||||
-- Owner-only genesis, now manifest v2 (no batch evidence at sequence 0).
|
||||
CREATE OR REPLACE FUNCTION evolution.provision_region_checkpoint(p_manifest bytea)
|
||||
RETURNS void LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
g evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
|
||||
d bytea := sha256(p_manifest);
|
||||
BEGIN
|
||||
IF g.commit_sequence <> 0 THEN
|
||||
RAISE EXCEPTION 'genesis manifest required' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
INSERT INTO evolution.region_leases(world_id, region_id, frame_id)
|
||||
VALUES (g.world_id, g.region_id, g.frame_id);
|
||||
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
|
||||
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
|
||||
versions, build_identity, payload_bytes, payload_sha256)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, p_manifest, 0, 0, 0,
|
||||
g.tick_microseconds, g.versions, g.build_identity, g.payload_bytes,
|
||||
g.payload_sha256);
|
||||
INSERT INTO evolution.region_heads(world_id, region_id, commit_sequence, commit_id, digest,
|
||||
end_tick, published_epoch)
|
||||
VALUES (g.world_id, g.region_id, 0, g.commit_id, d, 0, 0);
|
||||
END $$;
|
||||
|
||||
DROP FUNCTION evolution.publish_region_checkpoint(bytea);
|
||||
CREATE FUNCTION evolution.publish_region_checkpoint(p_manifest bytea, p_decisions bytea)
|
||||
RETURNS TABLE (region_frame evolution.u64, head_sequence evolution.u64, head_commit uuid,
|
||||
head_digest text, head_tick evolution.u64)
|
||||
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
c evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(p_manifest);
|
||||
h evolution.region_heads%ROWTYPE;
|
||||
p evolution.region_commits%ROWTYPE;
|
||||
e evolution.region_commits%ROWTYPE;
|
||||
frame evolution.u64;
|
||||
d bytea := sha256(p_manifest);
|
||||
zero32 constant bytea := decode(repeat('00', 32), 'hex');
|
||||
n integer;
|
||||
tick numeric;
|
||||
actor numeric;
|
||||
outcome numeric;
|
||||
organism numeric;
|
||||
last_tick numeric := -1;
|
||||
last_actor numeric := -1;
|
||||
actors integer[] := '{}';
|
||||
BEGIN
|
||||
IF c.commit_sequence = 0 THEN
|
||||
RAISE EXCEPTION 'genesis is provisioned, not published' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
-- The decision block is exactly what the manifest digests (ADR-0063 order/rules).
|
||||
n := coalesce(octet_length(p_decisions), -1);
|
||||
IF n <> 24 * c.decision_count
|
||||
OR (c.decision_count = 0 AND c.decisions_sha256 <> zero32)
|
||||
OR (c.decision_count > 0 AND sha256(p_decisions) <> c.decisions_sha256) THEN
|
||||
RAISE EXCEPTION 'decisions do not match manifest' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
FOR i IN 0 .. c.decision_count - 1 LOOP
|
||||
tick := evolution.manifest_uint(p_decisions, 24 * i, 8);
|
||||
actor := evolution.manifest_uint(p_decisions, 24 * i + 8, 4);
|
||||
outcome := evolution.manifest_uint(p_decisions, 24 * i + 12, 4);
|
||||
organism := evolution.manifest_uint(p_decisions, 24 * i + 16, 8);
|
||||
IF outcome <> 1 OR actor NOT IN (0, 1) OR tick <= c.begin_tick OR tick > c.end_tick
|
||||
OR tick < last_tick OR (tick = last_tick AND actor <= last_actor)
|
||||
OR actor::integer = ANY (actors)
|
||||
OR (c.profile = 1) <> (organism = 0) THEN
|
||||
RAISE EXCEPTION 'invalid decision' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
actors := actors || actor::integer;
|
||||
last_tick := tick;
|
||||
last_actor := actor;
|
||||
END LOOP;
|
||||
-- Exact retry of an already committed ID returns its immutable receipt without
|
||||
-- mutation or authority; any difference conflicts.
|
||||
SELECT * INTO e FROM evolution.region_commits r WHERE r.commit_id = c.commit_id;
|
||||
IF FOUND THEN
|
||||
IF e.world_id = c.world_id AND e.region_id = c.region_id AND e.manifest = p_manifest
|
||||
AND e.decisions = p_decisions THEN
|
||||
RETURN QUERY SELECT l.frame_id, e.commit_sequence, e.commit_id,
|
||||
encode(e.commit_digest, 'hex'), e.end_tick
|
||||
FROM evolution.region_leases l
|
||||
WHERE l.world_id = e.world_id AND l.region_id = e.region_id;
|
||||
RETURN;
|
||||
END IF;
|
||||
RAISE EXCEPTION 'commit identity reused' USING ERRCODE = 'EVL07';
|
||||
END IF;
|
||||
PERFORM evolution.check_region_lease(c.world_id, c.region_id, c.holder, c.epoch);
|
||||
SELECT * INTO h FROM evolution.region_heads r
|
||||
WHERE r.world_id = c.world_id AND r.region_id = c.region_id FOR UPDATE;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'region not provisioned' USING ERRCODE = 'EVL04';
|
||||
END IF;
|
||||
IF h.commit_sequence = 18446744073709551615 THEN
|
||||
RAISE EXCEPTION 'commit sequences exhausted' USING ERRCODE = 'EVL05';
|
||||
END IF;
|
||||
IF c.commit_sequence <> h.commit_sequence + 1 OR c.predecessor_commit <> h.commit_id
|
||||
OR c.predecessor_digest <> h.digest OR c.begin_tick <> h.end_tick THEN
|
||||
RAISE EXCEPTION 'region head moved' USING ERRCODE = 'EVL03';
|
||||
END IF;
|
||||
SELECT l.frame_id INTO frame FROM evolution.region_leases l
|
||||
WHERE l.world_id = c.world_id AND l.region_id = c.region_id;
|
||||
IF c.frame_id <> frame THEN
|
||||
RAISE EXCEPTION 'foreign frame' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
SELECT * INTO p FROM evolution.region_commits r
|
||||
WHERE r.world_id = c.world_id AND r.region_id = c.region_id
|
||||
AND r.commit_sequence = h.commit_sequence;
|
||||
IF c.versions <> p.versions OR c.tick_microseconds <> p.tick_microseconds
|
||||
OR substring(p_manifest FROM 13 FOR 4) <> substring(p.manifest FROM 13 FOR 4)
|
||||
OR substring(p_manifest FROM 229 FOR 12) <> substring(p.manifest FROM 229 FOR 12)
|
||||
OR NOT EXISTS (SELECT 1 FROM evolution.retained_builds b
|
||||
WHERE b.build_identity = c.build_identity) THEN
|
||||
RAISE EXCEPTION 'unsupported checkpoint' USING ERRCODE = 'EVL06';
|
||||
END IF;
|
||||
-- A final outcome is never repeated: the committed dead stay dead.
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_decisions x
|
||||
WHERE x.world_id = c.world_id AND x.region_id = c.region_id AND x.outcome = 1
|
||||
AND x.actor = ANY (actors)) THEN
|
||||
RAISE EXCEPTION 'outcome already final' USING ERRCODE = 'EVL08';
|
||||
END IF;
|
||||
INSERT INTO evolution.region_commits(world_id, region_id, commit_sequence, commit_id,
|
||||
commit_digest, manifest, begin_tick, end_tick, published_epoch, tick_microseconds,
|
||||
versions, build_identity, payload_bytes, payload_sha256, decisions)
|
||||
VALUES (c.world_id, c.region_id, c.commit_sequence, c.commit_id, d, p_manifest,
|
||||
c.begin_tick, c.end_tick, c.epoch, c.tick_microseconds, c.versions,
|
||||
c.build_identity, c.payload_bytes, c.payload_sha256, p_decisions);
|
||||
UPDATE evolution.region_heads r
|
||||
SET commit_sequence = c.commit_sequence, commit_id = c.commit_id, digest = d,
|
||||
end_tick = c.end_tick, published_epoch = c.epoch
|
||||
WHERE r.world_id = c.world_id AND r.region_id = c.region_id;
|
||||
FOR i IN 0 .. c.decision_count - 1 LOOP
|
||||
INSERT INTO evolution.region_decisions(world_id, region_id, outcome, actor, tick, organism,
|
||||
commit_sequence)
|
||||
VALUES (c.world_id, c.region_id, 1, evolution.manifest_uint(p_decisions, 24 * i + 8, 4),
|
||||
evolution.manifest_uint(p_decisions, 24 * i, 8),
|
||||
evolution.manifest_uint(p_decisions, 24 * i + 16, 8), c.commit_sequence);
|
||||
INSERT INTO evolution.region_outbox(world_id, region_id, outcome, actor, tick,
|
||||
commit_sequence)
|
||||
VALUES (c.world_id, c.region_id, 1, evolution.manifest_uint(p_decisions, 24 * i + 8, 4),
|
||||
evolution.manifest_uint(p_decisions, 24 * i, 8), c.commit_sequence);
|
||||
END LOOP;
|
||||
RETURN QUERY SELECT frame, c.commit_sequence, c.commit_id, encode(d, 'hex'), c.end_tick;
|
||||
END $$;
|
||||
|
||||
-- Read-only resolution after an ambiguous outcome. 'committed' carries the original
|
||||
-- receipt (under any later epoch; it grants no authority), 'conflict' means the ID
|
||||
-- holds other bytes or another region, 'absent' proves nothing about the future.
|
||||
CREATE FUNCTION evolution.resolve_region_commit(p_world evolution.u64, p_region evolution.u64,
|
||||
p_commit uuid, p_manifest bytea, p_decisions bytea)
|
||||
RETURNS TABLE (status text, region_frame evolution.u64, head_sequence evolution.u64,
|
||||
head_commit uuid, head_digest text, head_tick evolution.u64)
|
||||
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
DECLARE
|
||||
e evolution.region_commits%ROWTYPE;
|
||||
BEGIN
|
||||
IF p_world IS NULL OR p_region IS NULL OR p_commit IS NULL OR p_manifest IS NULL
|
||||
OR p_decisions IS NULL THEN
|
||||
RAISE EXCEPTION 'invalid resolution request' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
SELECT * INTO e FROM evolution.region_commits r WHERE r.commit_id = p_commit;
|
||||
IF NOT FOUND THEN
|
||||
RETURN QUERY SELECT 'absent'::text, NULL::evolution.u64, NULL::evolution.u64, NULL::uuid,
|
||||
NULL::text, NULL::evolution.u64;
|
||||
ELSIF e.world_id <> p_world OR e.region_id <> p_region OR e.manifest <> p_manifest
|
||||
OR e.decisions <> p_decisions THEN
|
||||
RETURN QUERY SELECT 'conflict'::text, NULL::evolution.u64, NULL::evolution.u64, NULL::uuid,
|
||||
NULL::text, NULL::evolution.u64;
|
||||
ELSE
|
||||
RETURN QUERY SELECT 'committed'::text, l.frame_id, e.commit_sequence, e.commit_id,
|
||||
encode(e.commit_digest, 'hex'), e.end_tick
|
||||
FROM evolution.region_leases l
|
||||
WHERE l.world_id = e.world_id AND l.region_id = e.region_id;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
-- Every committed permanent outcome of a region, for recovery reconciliation.
|
||||
CREATE FUNCTION evolution.load_region_decisions(p_world evolution.u64, p_region evolution.u64)
|
||||
RETURNS TABLE (commit_sequence evolution.u64, tick evolution.u64, actor smallint,
|
||||
outcome smallint, organism evolution.u64)
|
||||
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
SELECT x.commit_sequence, x.tick, x.actor, x.outcome, x.organism
|
||||
FROM evolution.region_decisions x
|
||||
WHERE x.world_id = p_world AND x.region_id = p_region
|
||||
ORDER BY x.tick, x.actor
|
||||
LIMIT 64;
|
||||
$$;
|
||||
|
||||
CREATE FUNCTION evolution.pending_region_outbox(p_world evolution.u64, p_region evolution.u64,
|
||||
p_limit integer)
|
||||
RETURNS TABLE (commit_sequence evolution.u64, tick evolution.u64, actor smallint,
|
||||
outcome smallint, organism evolution.u64)
|
||||
LANGUAGE plpgsql STABLE SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
IF p_limit IS NULL OR p_limit NOT BETWEEN 1 AND 64 THEN
|
||||
RAISE EXCEPTION 'invalid outbox limit' USING ERRCODE = '22023';
|
||||
END IF;
|
||||
RETURN QUERY SELECT o.commit_sequence, o.tick, o.actor, o.outcome, x.organism
|
||||
FROM evolution.region_outbox o
|
||||
JOIN evolution.region_decisions x USING (world_id, region_id, outcome, actor, tick)
|
||||
WHERE o.world_id = p_world AND o.region_id = p_region AND o.delivered_at IS NULL
|
||||
ORDER BY o.commit_sequence, o.tick, o.actor
|
||||
LIMIT p_limit;
|
||||
END $$;
|
||||
|
||||
-- Idempotent: true the first time a row is marked delivered, false afterwards.
|
||||
CREATE FUNCTION evolution.acknowledge_region_outbox(p_world evolution.u64,
|
||||
p_region evolution.u64, p_outcome integer, p_actor integer, p_tick evolution.u64)
|
||||
RETURNS boolean LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
|
||||
BEGIN
|
||||
UPDATE evolution.region_outbox o SET delivered_at = clock_timestamp()
|
||||
WHERE o.world_id = p_world AND o.region_id = p_region AND o.outcome = p_outcome
|
||||
AND o.actor = p_actor AND o.tick = p_tick AND o.delivered_at IS NULL;
|
||||
IF FOUND THEN
|
||||
RETURN true;
|
||||
END IF;
|
||||
IF EXISTS (SELECT 1 FROM evolution.region_outbox o
|
||||
WHERE o.world_id = p_world AND o.region_id = p_region AND o.outcome = p_outcome
|
||||
AND o.actor = p_actor AND o.tick = p_tick) THEN
|
||||
RETURN false;
|
||||
END IF;
|
||||
RAISE EXCEPTION 'unknown outbox row' USING ERRCODE = '22023';
|
||||
END $$;
|
||||
|
||||
REVOKE ALL ON FUNCTION
|
||||
evolution.refuse_history_rewrite(),
|
||||
evolution.refuse_outbox_rewrite(),
|
||||
evolution.publish_region_checkpoint(bytea, bytea),
|
||||
evolution.resolve_region_commit(evolution.u64, evolution.u64, uuid, bytea, bytea),
|
||||
evolution.load_region_decisions(evolution.u64, evolution.u64),
|
||||
evolution.pending_region_outbox(evolution.u64, evolution.u64, integer),
|
||||
evolution.acknowledge_region_outbox(evolution.u64, evolution.u64, integer, integer,
|
||||
evolution.u64)
|
||||
FROM PUBLIC;
|
||||
GRANT EXECUTE ON FUNCTION
|
||||
evolution.publish_region_checkpoint(bytea, bytea),
|
||||
evolution.resolve_region_commit(evolution.u64, evolution.u64, uuid, bytea, bytea),
|
||||
evolution.load_region_decisions(evolution.u64, evolution.u64),
|
||||
evolution.pending_region_outbox(evolution.u64, evolution.u64, integer),
|
||||
evolution.acknowledge_region_outbox(evolution.u64, evolution.u64, integer, integer,
|
||||
evolution.u64)
|
||||
TO :"runtime_role";
|
||||
INSERT INTO evolution.schema_migrations(version, sha256) VALUES (5, :'migration_sha_5');
|
||||
+18
-5
@@ -21,17 +21,25 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' AS at_v3,
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' AS at_v4 \gset
|
||||
\if :at_v4
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' AS at_v4,
|
||||
:'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2'
|
||||
|| ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4'
|
||||
|| ',5:' || :'migration_sha_5' AS at_v5 \gset
|
||||
\if :at_v5
|
||||
\elif :at_v4
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v3
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v2
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\elif :at_v1
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\else
|
||||
\echo 'Unknown or changed migration history'
|
||||
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
|
||||
@@ -41,6 +49,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
|
||||
\ir 0002_region_fencing.sql
|
||||
\ir 0003_region_checkpoints.sql
|
||||
\ir 0004_lineage_profile.sql
|
||||
\ir 0005_batch_decisions.sql
|
||||
\endif
|
||||
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
|
||||
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
|
||||
@@ -65,8 +74,12 @@ BEGIN
|
||||
('evolution.renew_region_lease(evolution.u64,evolution.u64,uuid,evolution.u64,integer)'::regprocedure::oid),
|
||||
('evolution.release_region_lease(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.load_region_head(evolution.u64,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.publish_region_checkpoint(bytea)'::regprocedure::oid),
|
||||
('evolution.load_region_checkpoint(evolution.u64,evolution.u64)'::regprocedure::oid)
|
||||
('evolution.publish_region_checkpoint(bytea,bytea)'::regprocedure::oid),
|
||||
('evolution.load_region_checkpoint(evolution.u64,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.resolve_region_commit(evolution.u64,evolution.u64,uuid,bytea,bytea)'::regprocedure::oid),
|
||||
('evolution.load_region_decisions(evolution.u64,evolution.u64)'::regprocedure::oid),
|
||||
('evolution.pending_region_outbox(evolution.u64,evolution.u64,integer)'::regprocedure::oid),
|
||||
('evolution.acknowledge_region_outbox(evolution.u64,evolution.u64,integer,integer,evolution.u64)'::regprocedure::oid)
|
||||
), violations(role_oid, reason) AS (
|
||||
SELECT r.oid, 'privileged attribute or owner identity' FROM pg_catalog.pg_roles r
|
||||
JOIN closure c ON c.oid = r.oid
|
||||
@@ -90,7 +103,7 @@ BEGIN
|
||||
OR EXISTS (SELECT 1 FROM pg_catalog.pg_namespace n WHERE n.nspname = 'public'
|
||||
AND pg_catalog.has_schema_privilege(c.oid, n.oid, 'CREATE'))
|
||||
UNION ALL
|
||||
-- Only the six guarded functions are executable; provisioning/internal are not.
|
||||
-- Only the ten guarded functions are executable; provisioning/internal are not.
|
||||
SELECT c.oid, 'EXECUTE outside the fenced function set' FROM closure c
|
||||
CROSS JOIN pg_catalog.pg_proc p
|
||||
WHERE p.pronamespace = 'evolution'::regnamespace
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
# Region batch evidence v1 (M7.5)
|
||||
|
||||
Authority: ADR-0063. The codecs are in `src/persistence/region_batch.cpp`. All
|
||||
integers are little-endian.
|
||||
|
||||
## Validation trace object
|
||||
|
||||
| Offset | Size | Field |
|
||||
|---|---|---|
|
||||
| 0 | 8 | magic `AEVOTRC1` |
|
||||
| 8 | 4 | trace format = 1 |
|
||||
| 12 | 4 | checkpoint profile (1 or 2) |
|
||||
| 16 | 16 | world, region |
|
||||
| 32 | 16 | predecessor commit ID |
|
||||
| 48 | 32 | predecessor manifest SHA-256 |
|
||||
| 80 | 16 | begin_tick, end_tick (1..64 ticks) |
|
||||
| 96 | 304·(end−begin) | one step per tick, in tick order |
|
||||
|
||||
Each 304-byte step is 38 little-endian u64 values in this order:
|
||||
- **Input:** temperature_mK, source_enabled, bite[5], terminal[2].
|
||||
- **Report:** tick, captured, retained, capture_heat, grown, unmet, source_status,
|
||||
assimilated[5], waste[5], terminal_transition, consumer_transition,
|
||||
decomposed[10].
|
||||
|
||||
Report ticks must be exactly begin+1 through end. The object is content-addressed
|
||||
beside the payload. Manifest v2 carries its length (96 + 304·ticks, which must
|
||||
match the tick range) and SHA-256. There is no trailing checksum: the manifest
|
||||
digest is the integrity check.
|
||||
|
||||
## Decision block
|
||||
|
||||
A decision block is 0 to 8 records, 24 bytes each: tick u64, actor u32
|
||||
(0 producer, 1 consumer), outcome u32 (1 = death), organism u64. The rules are:
|
||||
- Records are strictly ordered by (tick, actor), with at most one per actor.
|
||||
- Each tick lies in (begin_tick, end_tick].
|
||||
- organism is 0 for profile 1, and the bound archived organism (nonzero) for
|
||||
profile 2.
|
||||
- Unknown actor or outcome values are refused, never truncated.
|
||||
|
||||
Manifest v2 carries the count and the block's SHA-256; the digest is all zero
|
||||
exactly when the block is empty. Decisions are derived from the trace: every
|
||||
terminal transition yields exactly one decision. A durable decision's identity is
|
||||
(world, region, outcome, actor, tick).
|
||||
@@ -1,6 +1,6 @@
|
||||
# Region commit manifest contract v1 (M7.1 design, M7.4a encoding)
|
||||
|
||||
## Canonical manifest encoding v1 (ADR-0061)
|
||||
## Canonical manifest encoding v1/v2 (ADR-0061, ADR-0063)
|
||||
|
||||
The encoder, decoder and `supported()` check live in
|
||||
`src/persistence/region_checkpoint.cpp`; `evolution.parse_checkpoint_manifest`
|
||||
@@ -29,6 +29,11 @@ record, and each commit row stores it.
|
||||
| 236 | 4 | durability_policy = 1 (single-volume development) |
|
||||
| 240 | 8 | payload_bytes: profile 1 is 1..1 MiB; profile 2 is 1..4,268,144 |
|
||||
| 248 | 32 | payload SHA-256 (nonzero) |
|
||||
| 280 | 8 | v2: trace bytes, exactly 96 + 304·(end_tick − begin_tick); 0 at genesis |
|
||||
| 288 | 32 | v2: trace SHA-256 (nonzero except at genesis) |
|
||||
| 320 | 4 | v2: decision count 0..8 |
|
||||
| 324 | 4 | v2: reserved, must be 0 |
|
||||
| 328 | 32 | v2: decision block SHA-256, all zero exactly when the count is 0 |
|
||||
|
||||
A structurally invalid record is malformed (SQL 22023). A record that parses but
|
||||
has an unsupported version, profile, RNG, history or durability declaration is
|
||||
@@ -40,6 +45,14 @@ environmental envelope format 2; its object key is the
|
||||
hex payload SHA-256 in the region's namespace. Decisions, trace and outbox are not
|
||||
encoded here: M7.5 adds them under a new manifest version.
|
||||
|
||||
Manifest v2 (M7.5) is 360 bytes. Offset 8 holds 2, and the v1 fields keep their
|
||||
offsets. v1 records from M7.4 remain decodable but are never written again, and
|
||||
schema v5 admits only v2 for new commits and genesis. The trace object and
|
||||
decision block are defined in schemas/region-batch.md. Publication sends the
|
||||
manifest and the exact decision block together. The database records the
|
||||
decisions and their outbox rows in the same statement as the head. An
|
||||
identical retry of a committed ID returns the original receipt.
|
||||
|
||||
Authority: ADR-0058 and docs/architecture/REGION_DURABILITY.md. Logical schema,
|
||||
not an implemented wire codec, SQL migration or permission to serialize structs.
|
||||
M7.2 chooses concrete storage types preserving these domains; M7.4 specifies
|
||||
|
||||
@@ -8,7 +8,30 @@ namespace evo::persistence {
|
||||
namespace {
|
||||
using std::chrono::milliseconds;
|
||||
|
||||
// Per-profile payload operations; the publication and recovery paths are shared.
|
||||
struct Organisms {
|
||||
std::uint64_t producer{}, consumer{};
|
||||
};
|
||||
// Lifecycle agreement shared by both profiles: exactly one durable death per
|
||||
// terminal actor, none for an active one, never after the committed tick.
|
||||
bool lifecycle_agrees(const processes::EnvironmentalProgress& progress, Organisms organisms,
|
||||
std::span<const CommittedDecision> decisions, const RegionHead& head) {
|
||||
for (const auto actor : {Actor::producer, Actor::consumer}) {
|
||||
const auto terminal =
|
||||
actor == Actor::producer ? progress.producer_terminal : progress.consumer_terminal;
|
||||
const auto count = std::ranges::count_if(
|
||||
decisions, [&](const CommittedDecision& d) { return d.decision.actor == actor; });
|
||||
if (count > 1 || (terminal == 1) != (count == 1))
|
||||
return false;
|
||||
}
|
||||
return std::ranges::all_of(decisions, [&](const CommittedDecision& d) {
|
||||
const auto organism =
|
||||
d.decision.actor == Actor::producer ? organisms.producer : organisms.consumer;
|
||||
return d.decision.tick >= 1 && d.decision.tick <= head.end_tick &&
|
||||
d.commit_sequence <= head.sequence && d.decision.organism == organism;
|
||||
});
|
||||
}
|
||||
|
||||
// Per-profile payload operations; publication and recovery paths are shared.
|
||||
struct EnvironmentalCodec {
|
||||
using State = processes::EnvironmentalCommunity;
|
||||
using Recovered = RecoveredCheckpoint;
|
||||
@@ -22,10 +45,17 @@ struct EnvironmentalCodec {
|
||||
static std::uint64_t quantum(const State& s) {
|
||||
return s.tick_microseconds();
|
||||
}
|
||||
static Organisms organisms(const State&) {
|
||||
return {};
|
||||
}
|
||||
static auto restore(std::span<const std::uint8_t> payload, const CheckpointManifest& m,
|
||||
std::size_t budget) {
|
||||
return restore_environmental_payload(payload, m, budget);
|
||||
}
|
||||
static bool reconcile(const State& s, std::span<const CommittedDecision> d,
|
||||
const RegionHead& head) {
|
||||
return lifecycle_agrees(s.progress(), organisms(s), d, head);
|
||||
}
|
||||
};
|
||||
struct LineageCodec {
|
||||
using State = processes::LineageRegion;
|
||||
@@ -40,44 +70,61 @@ struct LineageCodec {
|
||||
static std::uint64_t quantum(const State& s) {
|
||||
return s.community().tick_microseconds();
|
||||
}
|
||||
static Organisms organisms(const State& s) {
|
||||
return {s.binding().producer, s.binding().consumer};
|
||||
}
|
||||
static auto restore(std::span<const std::uint8_t> payload, const CheckpointManifest& m,
|
||||
std::size_t budget) {
|
||||
return restore_lineage_payload(payload, m, budget);
|
||||
}
|
||||
// Plus: each durable death is the archive's death, at the same tick.
|
||||
static bool reconcile(const State& s, std::span<const CommittedDecision> d,
|
||||
const RegionHead& head) {
|
||||
if (!lifecycle_agrees(s.community().progress(), organisms(s), d, head))
|
||||
return false;
|
||||
return std::ranges::all_of(d, [&](const CommittedDecision& x) {
|
||||
const auto record = s.archive().organism(x.decision.organism);
|
||||
return record && !record->alive && record->died.value == x.decision.tick;
|
||||
});
|
||||
}
|
||||
};
|
||||
|
||||
PublicationFailure fail(PublicationError kind) {
|
||||
return {kind, {}, {}};
|
||||
PublicationFailure fail(PublicationError kind, ObjectError object = {}) {
|
||||
return {kind, {}, object, {}};
|
||||
}
|
||||
// Validates the exact payload bytes as recovery would, uploads and reads back.
|
||||
// Returns the manifest bytes ready for the database.
|
||||
template <typename Codec>
|
||||
std::expected<std::vector<std::uint8_t>, PublicationFailure>
|
||||
stage(const FilesystemObjectStore& objects, const std::vector<std::uint8_t>& payload,
|
||||
const CheckpointManifest& manifest, std::size_t budget) {
|
||||
auto bytes = encode_manifest(manifest);
|
||||
if (!bytes || manifest.profile != Codec::profile || !Codec::restore(payload, manifest, budget))
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
auto stored = objects.put(payload);
|
||||
std::string hex_parameter(std::span<const std::uint8_t> bytes) {
|
||||
constexpr char digits[] = "0123456789abcdef";
|
||||
std::string out = "\\x";
|
||||
for (const auto b : bytes) {
|
||||
out.push_back(digits[b >> 4U]);
|
||||
out.push_back(digits[b & 0x0FU]);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
std::expected<void, PublicationFailure> put_verified(const FilesystemObjectStore& objects,
|
||||
const std::vector<std::uint8_t>& bytes,
|
||||
const VerifiedObject& expected) {
|
||||
auto stored = objects.put(bytes);
|
||||
if (!stored)
|
||||
return std::unexpected{PublicationFailure{PublicationError::upload, {}, stored.error()}};
|
||||
return std::unexpected{fail(PublicationError::upload, stored.error())};
|
||||
auto back = objects.get(*stored);
|
||||
if (!back || *stored != VerifiedObject{manifest.payload_sha256, manifest.payload_bytes} ||
|
||||
!std::ranges::equal(*back, payload))
|
||||
return std::unexpected{PublicationFailure{
|
||||
PublicationError::verify, {}, back ? ObjectError::corrupt : back.error()}};
|
||||
return std::move(*bytes);
|
||||
if (!back || *stored != expected || !std::ranges::equal(*back, bytes))
|
||||
return std::unexpected{
|
||||
fail(PublicationError::verify, back ? ObjectError::corrupt : back.error())};
|
||||
return {};
|
||||
}
|
||||
// The receipt must agree with the local manifest the database digested.
|
||||
std::expected<CommittedCheckpoint, PublicationFailure>
|
||||
confirm(const std::expected<RegionHead, LeaseStoreFailure>& published,
|
||||
std::vector<std::uint8_t> manifest_bytes, const CheckpointManifest& manifest) {
|
||||
std::vector<std::uint8_t> manifest_bytes, const CheckpointManifest& manifest,
|
||||
PreparedCommit prepared) {
|
||||
if (!published) {
|
||||
const auto kind = published.error().kind == LeaseStoreError::unknown
|
||||
? PublicationError::unknown
|
||||
: PublicationError::rejected;
|
||||
return std::unexpected{PublicationFailure{kind, published.error(), {}}};
|
||||
return std::unexpected{
|
||||
PublicationFailure{kind, published.error(), {}, std::move(prepared)}};
|
||||
}
|
||||
// The database digested the bytes independently; the receipt must agree.
|
||||
const auto digest = sha256(manifest_bytes);
|
||||
if (!digest || published->digest != *digest || published->sequence != manifest.sequence ||
|
||||
published->commit != manifest.commit || published->end_tick != manifest.end_tick ||
|
||||
@@ -99,67 +146,111 @@ provision(Postgres& owner, const FilesystemObjectStore& objects, RegionKey key,
|
||||
return std::unexpected{fail(PublicationError::digest)};
|
||||
auto manifest = genesis_manifest(key, frame, commit, Codec::quantum(opening), build,
|
||||
payload->size(), *digest, Codec::profile);
|
||||
if (!manifest)
|
||||
auto bytes = manifest ? encode_manifest(*manifest)
|
||||
: std::expected<std::vector<std::uint8_t>, CheckpointError>{
|
||||
std::unexpected{CheckpointError::format}};
|
||||
if (!bytes || !Codec::restore(*payload, *manifest, budget))
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
auto bytes = stage<Codec>(objects, *payload, *manifest, budget);
|
||||
if (!bytes)
|
||||
return std::unexpected{bytes.error()};
|
||||
constexpr char digits[] = "0123456789abcdef";
|
||||
std::string parameter = "\\x";
|
||||
for (const auto b : *bytes) {
|
||||
parameter.push_back(digits[b >> 4U]);
|
||||
parameter.push_back(digits[b & 0x0FU]);
|
||||
}
|
||||
const std::array<std::string, 1> p{parameter};
|
||||
if (auto stored = put_verified(objects, *payload, {*digest, payload->size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
const std::array<std::string, 1> p{hex_parameter(*bytes)};
|
||||
auto done = owner.query("SELECT evolution.provision_region_checkpoint($1::bytea)", p, timeout);
|
||||
if (!done) {
|
||||
const bool unknown = done.error().kind == PgError::unknown;
|
||||
return std::unexpected{PublicationFailure{
|
||||
unknown ? PublicationError::unknown : PublicationError::rejected,
|
||||
{unknown ? LeaseStoreError::unknown : LeaseStoreError::store, done.error()},
|
||||
{},
|
||||
{}}};
|
||||
}
|
||||
auto committed = load_region_checkpoint(owner, key, timeout);
|
||||
if (!committed)
|
||||
return std::unexpected{
|
||||
PublicationFailure{PublicationError::unknown, committed.error(), {}}};
|
||||
return confirm(committed->head, std::move(*bytes), *manifest);
|
||||
PublicationFailure{PublicationError::unknown, committed.error(), {}, {}}};
|
||||
return confirm(committed->head, std::move(*bytes), *manifest, {});
|
||||
}
|
||||
|
||||
template <typename Codec>
|
||||
std::expected<CommittedCheckpoint, PublicationFailure>
|
||||
publish(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const typename Codec::State& state, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t budget, milliseconds timeout) {
|
||||
std::expected<PublishedBatch, PublicationFailure>
|
||||
publish_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<typename Codec::State>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t budget, milliseconds timeout) {
|
||||
const auto previous = decode_manifest(predecessor.manifest);
|
||||
auto payload = Codec::snapshot(state);
|
||||
if (!previous || !payload || previous->profile != Codec::profile)
|
||||
if (!previous || previous->profile != Codec::profile || !batch.sealed() ||
|
||||
batch.steps().empty() || batch.begin_tick() != previous->end_tick)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const auto digest = sha256(*payload);
|
||||
if (!digest)
|
||||
// Replay the recorded inputs from the committed predecessor's verified bytes.
|
||||
auto base = objects.get({previous->payload_sha256, previous->payload_bytes});
|
||||
if (!base)
|
||||
return std::unexpected{fail(PublicationError::predecessor, base.error())};
|
||||
auto replay = Codec::restore(*base, *previous, budget);
|
||||
if (!replay)
|
||||
return std::unexpected{fail(PublicationError::predecessor)};
|
||||
for (const auto& step : batch.steps()) {
|
||||
auto report = (*replay)->advance(step.report.tick, step.input);
|
||||
if (!report || *report != step.report)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
}
|
||||
auto payload = Codec::snapshot(batch.state());
|
||||
const auto replayed = Codec::snapshot(**replay);
|
||||
if (!payload || !replayed || *payload != *replayed)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
|
||||
const auto organisms = Codec::organisms(batch.state());
|
||||
const auto decisions = derive_decisions(batch.steps(), organisms.producer, organisms.consumer);
|
||||
auto decision_bytes = encode_decisions(decisions, batch.begin_tick(), batch.end_tick());
|
||||
const TraceHeader header{static_cast<std::uint32_t>(Codec::profile),
|
||||
previous->key,
|
||||
previous->commit,
|
||||
predecessor.head.digest,
|
||||
batch.begin_tick(),
|
||||
batch.end_tick()};
|
||||
auto trace = encode_trace(header, batch.steps());
|
||||
if (!decision_bytes || !trace)
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
const auto payload_digest = sha256(*payload);
|
||||
const auto trace_digest = sha256(*trace);
|
||||
const auto decisions_digest = sha256(*decision_bytes);
|
||||
if (!payload_digest || !trace_digest || !decisions_digest)
|
||||
return std::unexpected{fail(PublicationError::digest)};
|
||||
auto manifest = successor_manifest(*previous, predecessor.head.digest, commit, lease,
|
||||
Codec::tick(state), build, payload->size(), *digest);
|
||||
if (!manifest)
|
||||
const BatchDigests digests{trace->size(), *trace_digest,
|
||||
static_cast<std::uint32_t>(decisions.size()),
|
||||
decisions.empty() ? Sha256Digest{} : *decisions_digest};
|
||||
auto manifest =
|
||||
successor_manifest(*previous, predecessor.head.digest, commit, lease, batch.end_tick(),
|
||||
build, payload->size(), *payload_digest, digests);
|
||||
auto bytes = manifest ? encode_manifest(*manifest)
|
||||
: std::expected<std::vector<std::uint8_t>, CheckpointError>{
|
||||
std::unexpected{CheckpointError::format}};
|
||||
if (!bytes || !Codec::restore(*payload, *manifest, budget))
|
||||
return std::unexpected{fail(PublicationError::candidate)};
|
||||
auto bytes = stage<Codec>(objects, *payload, *manifest, budget);
|
||||
if (!bytes)
|
||||
return std::unexpected{bytes.error()};
|
||||
auto published = publish_region_checkpoint(db, *bytes, timeout);
|
||||
return confirm(published, std::move(*bytes), *manifest);
|
||||
if (auto stored = put_verified(objects, *payload, {*payload_digest, payload->size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
if (auto stored = put_verified(objects, *trace, {*trace_digest, trace->size()}); !stored)
|
||||
return std::unexpected{stored.error()};
|
||||
|
||||
PreparedCommit prepared{previous->key, commit, *bytes, *decision_bytes};
|
||||
auto published = publish_region_checkpoint(db, *bytes, *decision_bytes, timeout);
|
||||
auto committed = confirm(published, std::move(*bytes), *manifest, std::move(prepared));
|
||||
if (!committed)
|
||||
return std::unexpected{committed.error()};
|
||||
return PublishedBatch{std::move(*committed), decisions};
|
||||
}
|
||||
|
||||
template <typename Codec>
|
||||
std::expected<typename Codec::Recovered, RecoveryFailure>
|
||||
recover(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
std::span<const Sha256Digest> retained_builds, std::size_t budget, milliseconds timeout) {
|
||||
const auto store_failure = [](const LeaseStoreFailure& f) {
|
||||
return std::unexpected{RecoveryFailure{f.kind == LeaseStoreError::not_provisioned
|
||||
? RecoveryError::not_provisioned
|
||||
: RecoveryError::store,
|
||||
f}};
|
||||
};
|
||||
auto committed = load_region_checkpoint(db, key, timeout);
|
||||
if (!committed)
|
||||
return std::unexpected{
|
||||
RecoveryFailure{committed.error().kind == LeaseStoreError::not_provisioned
|
||||
? RecoveryError::not_provisioned
|
||||
: RecoveryError::store,
|
||||
committed.error()}};
|
||||
return store_failure(committed.error());
|
||||
const auto fail_with = [](RecoveryError kind) {
|
||||
return std::unexpected{RecoveryFailure{kind, {}}};
|
||||
};
|
||||
@@ -187,7 +278,14 @@ recover(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
auto state = Codec::restore(*payload, *manifest, budget);
|
||||
if (!state)
|
||||
return fail_with(RecoveryError::payload);
|
||||
return typename Codec::Recovered{std::move(*committed), *manifest, std::move(*state)};
|
||||
// Durable permanent outcomes must agree with the restored state they finalized.
|
||||
auto decisions = load_region_decisions(db, key, timeout);
|
||||
if (!decisions)
|
||||
return store_failure(decisions.error());
|
||||
if (!Codec::reconcile(**state, *decisions, head))
|
||||
return fail_with(RecoveryError::history);
|
||||
return typename Codec::Recovered{std::move(*committed), *manifest, std::move(*state),
|
||||
std::move(*decisions)};
|
||||
}
|
||||
} // namespace
|
||||
|
||||
@@ -198,12 +296,14 @@ std::expected<CommittedCheckpoint, PublicationFailure> provision_environmental_r
|
||||
return provision<EnvironmentalCodec>(owner, objects, key, frame, opening, commit, build,
|
||||
restore_budget, timeout);
|
||||
}
|
||||
std::expected<CommittedCheckpoint, PublicationFailure> publish_environmental_checkpoint(
|
||||
Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const processes::EnvironmentalCommunity& state,
|
||||
Uuid commit, const Sha256Digest& build, std::size_t restore_budget, milliseconds timeout) {
|
||||
return publish<EnvironmentalCodec>(db, objects, lease, predecessor, state, commit, build,
|
||||
restore_budget, timeout);
|
||||
std::expected<PublishedBatch, PublicationFailure>
|
||||
publish_environmental_batch(Postgres& db, const FilesystemObjectStore& objects,
|
||||
const LeaseGrant& lease, const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::EnvironmentalCommunity>& batch,
|
||||
Uuid commit, const Sha256Digest& build, std::size_t restore_budget,
|
||||
milliseconds timeout) {
|
||||
return publish_batch<EnvironmentalCodec>(db, objects, lease, predecessor, batch, commit, build,
|
||||
restore_budget, timeout);
|
||||
}
|
||||
std::expected<RecoveredCheckpoint, RecoveryFailure>
|
||||
recover_environmental_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
@@ -220,12 +320,13 @@ provision_lineage_region(Postgres& owner, const FilesystemObjectStore& objects,
|
||||
return provision<LineageCodec>(owner, objects, key, frame, opening, commit, build,
|
||||
restore_budget, timeout);
|
||||
}
|
||||
std::expected<CommittedCheckpoint, PublicationFailure> publish_lineage_checkpoint(
|
||||
Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor, const processes::LineageRegion& state, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget, milliseconds timeout) {
|
||||
return publish<LineageCodec>(db, objects, lease, predecessor, state, commit, build,
|
||||
restore_budget, timeout);
|
||||
std::expected<PublishedBatch, PublicationFailure>
|
||||
publish_lineage_batch(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant& lease,
|
||||
const CommittedCheckpoint& predecessor,
|
||||
const RegionBatch<processes::LineageRegion>& batch, Uuid commit,
|
||||
const Sha256Digest& build, std::size_t restore_budget, milliseconds timeout) {
|
||||
return publish_batch<LineageCodec>(db, objects, lease, predecessor, batch, commit, build,
|
||||
restore_budget, timeout);
|
||||
}
|
||||
std::expected<RecoveredLineageCheckpoint, RecoveryFailure>
|
||||
recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, RegionKey key,
|
||||
@@ -233,4 +334,14 @@ recover_lineage_checkpoint(Postgres& db, const FilesystemObjectStore& objects, R
|
||||
std::size_t restore_budget, milliseconds timeout) {
|
||||
return recover<LineageCodec>(db, objects, key, retained_builds, restore_budget, timeout);
|
||||
}
|
||||
|
||||
std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_prepared(Postgres& db, const PreparedCommit& prepared, milliseconds timeout) {
|
||||
return resolve_region_commit(db, prepared.key, prepared.commit, prepared.manifest,
|
||||
prepared.decisions, timeout);
|
||||
}
|
||||
std::expected<RegionHead, LeaseStoreFailure>
|
||||
retry_prepared(Postgres& db, const PreparedCommit& prepared, milliseconds timeout) {
|
||||
return publish_region_checkpoint(db, prepared.manifest, prepared.decisions, timeout);
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include <algorithm>
|
||||
|
||||
namespace evo::persistence {
|
||||
namespace {
|
||||
constexpr std::uint64_t trace_magic = 0x314352544F564541ULL; // "AEVOTRC1"
|
||||
constexpr std::uint32_t trace_format = 1;
|
||||
constexpr std::uint64_t max_trace_ticks = 64;
|
||||
|
||||
template <std::size_t N> bool zero(const std::array<std::uint8_t, N>& bytes) {
|
||||
return std::ranges::all_of(bytes, [](std::uint8_t b) { return b == 0; });
|
||||
}
|
||||
// Fills an exactly sized buffer; callers check the final cursor.
|
||||
class Writer {
|
||||
public:
|
||||
explicit Writer(std::vector<std::uint8_t>& out) : out_(out) {}
|
||||
void u(std::uint64_t value, unsigned width) {
|
||||
for (unsigned i = 0; i < width; ++i, value >>= 8U)
|
||||
out_[at_++] = static_cast<std::uint8_t>(value & 0xFFU);
|
||||
}
|
||||
template <std::size_t N> void bytes(const std::array<std::uint8_t, N>& value) {
|
||||
std::ranges::copy(value, out_.begin() + static_cast<std::ptrdiff_t>(at_));
|
||||
at_ += N;
|
||||
}
|
||||
template <std::size_t N> void all(const std::array<std::uint64_t, N>& values) {
|
||||
for (const auto v : values)
|
||||
u(v, 8);
|
||||
}
|
||||
[[nodiscard]] std::size_t at() const noexcept {
|
||||
return at_;
|
||||
}
|
||||
|
||||
private:
|
||||
std::vector<std::uint8_t>& out_;
|
||||
std::size_t at_{};
|
||||
};
|
||||
class Reader {
|
||||
public:
|
||||
explicit Reader(std::span<const std::uint8_t> in) : in_(in) {}
|
||||
std::uint64_t u(unsigned width) {
|
||||
std::uint64_t value{};
|
||||
for (unsigned i = 0; i < width; ++i)
|
||||
value |= static_cast<std::uint64_t>(in_[at_ + i]) << (8U * i);
|
||||
at_ += width;
|
||||
return value;
|
||||
}
|
||||
template <std::size_t N> std::array<std::uint8_t, N> bytes() {
|
||||
std::array<std::uint8_t, N> value{};
|
||||
std::copy_n(in_.begin() + static_cast<std::ptrdiff_t>(at_), N, value.begin());
|
||||
at_ += N;
|
||||
return value;
|
||||
}
|
||||
template <std::size_t N> void all(std::array<std::uint64_t, N>& values) {
|
||||
for (auto& v : values)
|
||||
v = u(8);
|
||||
}
|
||||
|
||||
private:
|
||||
std::span<const std::uint8_t> in_;
|
||||
std::size_t at_{};
|
||||
};
|
||||
|
||||
bool valid(const TraceHeader& h) noexcept {
|
||||
return h.profile != 0 && h.key.world != 0 && h.key.region != 0 && !zero(h.predecessor_commit) &&
|
||||
!zero(h.predecessor_digest) && h.end_tick > h.begin_tick &&
|
||||
h.end_tick - h.begin_tick <= max_trace_ticks;
|
||||
}
|
||||
bool contiguous(const TraceHeader& h, std::span<const TraceStep> steps) noexcept {
|
||||
if (steps.size() != h.end_tick - h.begin_tick)
|
||||
return false;
|
||||
for (std::size_t i = 0; i < steps.size(); ++i)
|
||||
if (steps[i].report.tick != h.begin_tick + i + 1)
|
||||
return false;
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_trace(const TraceHeader& h, std::span<const TraceStep> steps) {
|
||||
if (!valid(h) || !contiguous(h, steps))
|
||||
return std::unexpected{BatchError::format};
|
||||
std::vector<std::uint8_t> out(trace_bytes_for(steps.size()));
|
||||
Writer w{out};
|
||||
w.u(trace_magic, 8);
|
||||
w.u(trace_format, 4);
|
||||
w.u(h.profile, 4);
|
||||
w.u(h.key.world, 8);
|
||||
w.u(h.key.region, 8);
|
||||
w.bytes(h.predecessor_commit);
|
||||
w.bytes(h.predecessor_digest);
|
||||
w.u(h.begin_tick, 8);
|
||||
w.u(h.end_tick, 8);
|
||||
for (const auto& s : steps) {
|
||||
const auto& in = s.input;
|
||||
const auto& r = s.report;
|
||||
w.u(in.temperature_mK, 8);
|
||||
w.u(in.source_enabled, 8);
|
||||
w.all(in.bite);
|
||||
w.all(in.terminal);
|
||||
for (const auto v :
|
||||
{r.tick, r.captured, r.retained, r.capture_heat, r.grown, r.unmet, r.source_status})
|
||||
w.u(v, 8);
|
||||
w.all(r.assimilated);
|
||||
w.all(r.waste);
|
||||
w.u(r.terminal_transition, 8);
|
||||
w.u(r.consumer_transition, 8);
|
||||
w.all(r.decomposed);
|
||||
}
|
||||
if (w.at() != out.size())
|
||||
return std::unexpected{BatchError::format};
|
||||
return out;
|
||||
}
|
||||
|
||||
std::expected<DecodedTrace, BatchError> decode_trace(std::span<const std::uint8_t> bytes) {
|
||||
if (bytes.size() < trace_header_bytes)
|
||||
return std::unexpected{BatchError::format};
|
||||
Reader r{bytes};
|
||||
if (r.u(8) != trace_magic || r.u(4) != trace_format)
|
||||
return std::unexpected{BatchError::format};
|
||||
DecodedTrace t;
|
||||
auto& h = t.header;
|
||||
h.profile = static_cast<std::uint32_t>(r.u(4));
|
||||
h.key.world = r.u(8);
|
||||
h.key.region = r.u(8);
|
||||
h.predecessor_commit = r.bytes<16>();
|
||||
h.predecessor_digest = r.bytes<32>();
|
||||
h.begin_tick = r.u(8);
|
||||
h.end_tick = r.u(8);
|
||||
// Size is fixed by the header before any step is allocated.
|
||||
if (!valid(h) || bytes.size() != trace_bytes_for(h.end_tick - h.begin_tick))
|
||||
return std::unexpected{BatchError::format};
|
||||
t.steps.resize(static_cast<std::size_t>(h.end_tick - h.begin_tick));
|
||||
for (auto& s : t.steps) {
|
||||
auto& in = s.input;
|
||||
auto& rep = s.report;
|
||||
in.temperature_mK = r.u(8);
|
||||
in.source_enabled = r.u(8);
|
||||
r.all(in.bite);
|
||||
r.all(in.terminal);
|
||||
for (auto* v : {&rep.tick, &rep.captured, &rep.retained, &rep.capture_heat, &rep.grown,
|
||||
&rep.unmet, &rep.source_status})
|
||||
*v = r.u(8);
|
||||
r.all(rep.assimilated);
|
||||
r.all(rep.waste);
|
||||
rep.terminal_transition = r.u(8);
|
||||
rep.consumer_transition = r.u(8);
|
||||
r.all(rep.decomposed);
|
||||
}
|
||||
if (!contiguous(h, t.steps))
|
||||
return std::unexpected{BatchError::format};
|
||||
return t;
|
||||
}
|
||||
|
||||
namespace {
|
||||
bool valid(std::span<const BatchDecision> d, std::uint64_t begin, std::uint64_t end) noexcept {
|
||||
if (d.size() > max_batch_decisions)
|
||||
return false;
|
||||
for (std::size_t i = 0; i < d.size(); ++i) {
|
||||
const auto& x = d[i];
|
||||
if (x.outcome != Outcome::death ||
|
||||
(x.actor != Actor::producer && x.actor != Actor::consumer) || x.tick <= begin ||
|
||||
x.tick > end)
|
||||
return false;
|
||||
for (std::size_t j = 0; j < i; ++j)
|
||||
if (d[j].actor == x.actor)
|
||||
return false; // Terminal: at most one death per actor.
|
||||
if (i > 0 &&
|
||||
(d[i - 1].tick > x.tick || (d[i - 1].tick == x.tick && d[i - 1].actor >= x.actor)))
|
||||
return false; // Canonical order (tick, actor).
|
||||
}
|
||||
return true;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<std::vector<std::uint8_t>, BatchError>
|
||||
encode_decisions(std::span<const BatchDecision> decisions, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick) {
|
||||
if (!valid(decisions, begin_tick, end_tick))
|
||||
return std::unexpected{BatchError::format};
|
||||
std::vector<std::uint8_t> out(decisions.size() * decision_bytes);
|
||||
Writer w{out};
|
||||
for (const auto& d : decisions) {
|
||||
w.u(d.tick, 8);
|
||||
w.u(static_cast<std::uint32_t>(d.actor), 4);
|
||||
w.u(static_cast<std::uint32_t>(d.outcome), 4);
|
||||
w.u(d.organism, 8);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
std::expected<std::vector<BatchDecision>, BatchError>
|
||||
decode_decisions(std::span<const std::uint8_t> bytes, std::uint64_t begin_tick,
|
||||
std::uint64_t end_tick) {
|
||||
if (bytes.size() % decision_bytes != 0 || bytes.size() > max_batch_decisions * decision_bytes)
|
||||
return std::unexpected{BatchError::format};
|
||||
std::vector<BatchDecision> out(bytes.size() / decision_bytes);
|
||||
Reader r{bytes};
|
||||
for (auto& d : out) {
|
||||
d.tick = r.u(8);
|
||||
const auto actor = r.u(4);
|
||||
const auto outcome = r.u(4);
|
||||
// Unknown values are refused, never truncated onto known ones.
|
||||
if (actor > 1 || outcome != 1)
|
||||
return std::unexpected{BatchError::format};
|
||||
d.actor = static_cast<Actor>(actor);
|
||||
d.outcome = Outcome::death;
|
||||
d.organism = r.u(8);
|
||||
}
|
||||
if (!valid(out, begin_tick, end_tick))
|
||||
return std::unexpected{BatchError::format};
|
||||
return out;
|
||||
}
|
||||
|
||||
std::vector<BatchDecision> derive_decisions(std::span<const TraceStep> steps,
|
||||
std::uint64_t producer_organism,
|
||||
std::uint64_t consumer_organism) {
|
||||
std::vector<BatchDecision> out;
|
||||
for (const auto& s : steps) {
|
||||
if (s.report.terminal_transition == 1)
|
||||
out.push_back({s.report.tick, Actor::producer, Outcome::death, producer_organism});
|
||||
if (s.report.consumer_transition == 1)
|
||||
out.push_back({s.report.tick, Actor::consumer, Outcome::death, consumer_organism});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
@@ -5,7 +5,6 @@
|
||||
namespace evo::persistence {
|
||||
namespace {
|
||||
constexpr std::uint64_t manifest_magic = 0x314D43524F564541ULL; // "AEVORCM1"
|
||||
constexpr std::uint32_t manifest_version = 1;
|
||||
constexpr std::uint64_t maximum = ~std::uint64_t{0};
|
||||
|
||||
template <std::size_t N> bool zero(const std::array<std::uint8_t, N>& bytes) {
|
||||
@@ -58,12 +57,23 @@ bool structural(const CheckpointManifest& m) noexcept {
|
||||
m.tick_microseconds == 0 || zero(m.build) || m.payload_bytes == 0 ||
|
||||
m.payload_bytes > limit || zero(m.payload_sha256))
|
||||
return false;
|
||||
const auto& b = m.batch;
|
||||
if (m.manifest_version != 1 && m.manifest_version != 2)
|
||||
return false;
|
||||
const bool no_batch = b == BatchDigests{};
|
||||
if (m.sequence == 0)
|
||||
return zero(m.predecessor_commit) && zero(m.predecessor_digest) && m.begin_tick == 0 &&
|
||||
m.end_tick == 0 && zero(m.holder) && m.epoch == 0;
|
||||
return !zero(m.predecessor_commit) && m.predecessor_commit != m.commit &&
|
||||
!zero(m.predecessor_digest) && !zero(m.holder) && m.epoch != 0 &&
|
||||
m.end_tick > m.begin_tick && m.end_tick - m.begin_tick <= max_checkpoint_ticks;
|
||||
m.end_tick == 0 && zero(m.holder) && m.epoch == 0 && no_batch;
|
||||
const bool link = !zero(m.predecessor_commit) && m.predecessor_commit != m.commit &&
|
||||
!zero(m.predecessor_digest) && !zero(m.holder) && m.epoch != 0 &&
|
||||
m.end_tick > m.begin_tick &&
|
||||
m.end_tick - m.begin_tick <= max_checkpoint_ticks;
|
||||
if (m.manifest_version == 1)
|
||||
return link && no_batch;
|
||||
// v2: the trace has exactly one step per tick; decisions are bounded.
|
||||
return link && b.trace_bytes == trace_bytes_for(m.end_tick - m.begin_tick) &&
|
||||
!zero(b.trace_sha256) && b.decision_count <= max_batch_decisions &&
|
||||
(b.decision_count == 0) == zero(b.decisions_sha256);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
@@ -78,13 +88,13 @@ bool supported(const CheckpointManifest& m) noexcept {
|
||||
|
||||
std::expected<std::vector<std::uint8_t>, CheckpointError>
|
||||
encode_manifest(const CheckpointManifest& m) {
|
||||
if (!structural(m))
|
||||
if (!structural(m) || m.manifest_version != 2)
|
||||
return std::unexpected{CheckpointError::format};
|
||||
std::vector<std::uint8_t> out;
|
||||
out.reserve(checkpoint_manifest_bytes);
|
||||
Writer w{out};
|
||||
w.u(manifest_magic, 8);
|
||||
w.u(manifest_version, 4);
|
||||
w.u(m.manifest_version, 4);
|
||||
w.u(static_cast<std::uint8_t>(m.profile), 4);
|
||||
w.u(m.key.world, 8);
|
||||
w.u(m.key.region, 8);
|
||||
@@ -108,19 +118,28 @@ encode_manifest(const CheckpointManifest& m) {
|
||||
w.u(static_cast<std::uint8_t>(m.durability), 4);
|
||||
w.u(m.payload_bytes, 8);
|
||||
w.bytes(m.payload_sha256);
|
||||
w.u(m.batch.trace_bytes, 8);
|
||||
w.bytes(m.batch.trace_sha256);
|
||||
w.u(m.batch.decision_count, 4);
|
||||
w.u(0, 4);
|
||||
w.bytes(m.batch.decisions_sha256);
|
||||
return out;
|
||||
}
|
||||
|
||||
std::expected<CheckpointManifest, CheckpointError>
|
||||
decode_manifest(std::span<const std::uint8_t> bytes) {
|
||||
if (bytes.size() != checkpoint_manifest_bytes)
|
||||
if (bytes.size() < 12)
|
||||
return std::unexpected{CheckpointError::format};
|
||||
Reader r{bytes};
|
||||
if (r.u(8) != manifest_magic)
|
||||
return std::unexpected{CheckpointError::format};
|
||||
if (r.u(4) != manifest_version)
|
||||
return std::unexpected{CheckpointError::unsupported};
|
||||
CheckpointManifest m;
|
||||
m.manifest_version = static_cast<std::uint32_t>(r.u(4));
|
||||
if (m.manifest_version != 1 && m.manifest_version != 2)
|
||||
return std::unexpected{CheckpointError::unsupported};
|
||||
if (bytes.size() !=
|
||||
(m.manifest_version == 1 ? checkpoint_manifest_v1_bytes : checkpoint_manifest_bytes))
|
||||
return std::unexpected{CheckpointError::format};
|
||||
const auto profile = r.u(4);
|
||||
m.profile =
|
||||
profile <= 0xFFU ? static_cast<CheckpointProfile>(profile) : CheckpointProfile::unknown;
|
||||
@@ -148,6 +167,14 @@ decode_manifest(std::span<const std::uint8_t> bytes) {
|
||||
durability <= 0xFFU ? static_cast<DurabilityPolicy>(durability) : DurabilityPolicy::unknown;
|
||||
m.payload_bytes = r.u(8);
|
||||
m.payload_sha256 = r.bytes<32>();
|
||||
if (m.manifest_version == 2) {
|
||||
m.batch.trace_bytes = r.u(8);
|
||||
m.batch.trace_sha256 = r.bytes<32>();
|
||||
m.batch.decision_count = static_cast<std::uint32_t>(r.u(4));
|
||||
if (r.u(4) != 0)
|
||||
return std::unexpected{CheckpointError::format}; // Reserved.
|
||||
m.batch.decisions_sha256 = r.bytes<32>();
|
||||
}
|
||||
if (!structural(m))
|
||||
return std::unexpected{CheckpointError::format};
|
||||
return m;
|
||||
@@ -176,7 +203,8 @@ genesis_manifest(RegionKey key, std::uint64_t frame, Uuid commit, std::uint64_t
|
||||
std::expected<CheckpointManifest, CheckpointError>
|
||||
successor_manifest(const CheckpointManifest& p, const Sha256Digest& predecessor_digest, Uuid commit,
|
||||
const LeaseGrant& lease, std::uint64_t end_tick, const Sha256Digest& build,
|
||||
std::uint64_t payload_bytes, const Sha256Digest& payload_sha256) {
|
||||
std::uint64_t payload_bytes, const Sha256Digest& payload_sha256,
|
||||
const BatchDigests& batch) {
|
||||
// This build writes only its own versions; a change requires a migration (M7.9).
|
||||
if (!supported(p))
|
||||
return std::unexpected{CheckpointError::unsupported};
|
||||
@@ -184,6 +212,8 @@ successor_manifest(const CheckpointManifest& p, const Sha256Digest& predecessor_
|
||||
end_tick <= p.end_tick || end_tick - p.end_tick > max_checkpoint_ticks)
|
||||
return std::unexpected{CheckpointError::chain};
|
||||
CheckpointManifest m = p;
|
||||
m.manifest_version = 2;
|
||||
m.batch = batch;
|
||||
m.commit = commit;
|
||||
m.sequence = p.sequence + 1;
|
||||
m.predecessor_commit = p.commit;
|
||||
|
||||
@@ -32,6 +32,10 @@ LeaseStoreFailure classify(const PgFailure& f) {
|
||||
kind = LeaseStoreError::unsupported;
|
||||
else if (state == "EVL07")
|
||||
kind = LeaseStoreError::commit_conflict;
|
||||
else if (state == "EVL08")
|
||||
kind = LeaseStoreError::outcome_final;
|
||||
else if (state == "EVL09")
|
||||
kind = LeaseStoreError::immutable;
|
||||
return {kind, f};
|
||||
}
|
||||
LeaseStoreFailure malformed(bool mutation) {
|
||||
@@ -214,10 +218,10 @@ load_region_checkpoint(Postgres& db, RegionKey key, milliseconds timeout) {
|
||||
|
||||
std::expected<RegionHead, LeaseStoreFailure>
|
||||
publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
milliseconds timeout) {
|
||||
const std::array<std::string, 1> p{bytea(manifest)};
|
||||
auto rows =
|
||||
db.query("SELECT * FROM evolution.publish_region_checkpoint($1::bytea)", p, timeout);
|
||||
std::span<const std::uint8_t> decisions, milliseconds timeout) {
|
||||
const std::array<std::string, 2> p{bytea(manifest), bytea(decisions)};
|
||||
auto rows = db.query("SELECT * FROM evolution.publish_region_checkpoint($1::bytea, $2::bytea)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
auto head = parse_head(*rows);
|
||||
@@ -225,4 +229,97 @@ publish_region_checkpoint(Postgres& db, std::span<const std::uint8_t> manifest,
|
||||
return std::unexpected{malformed(true)};
|
||||
return *head;
|
||||
}
|
||||
|
||||
std::expected<ResolvedCommit, LeaseStoreFailure>
|
||||
resolve_region_commit(Postgres& db, RegionKey key, const Uuid& commit,
|
||||
std::span<const std::uint8_t> manifest,
|
||||
std::span<const std::uint8_t> decisions, milliseconds timeout) {
|
||||
const std::array<std::string, 5> p{std::to_string(key.world), std::to_string(key.region),
|
||||
uuid_text(commit), bytea(manifest), bytea(decisions)};
|
||||
auto rows = db.query("SELECT * FROM evolution.resolve_region_commit($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::uuid, $4::bytea, $5::bytea)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
if (rows->size() != 1 || (*rows)[0].size() != 6)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto& status_text = (*rows)[0][0];
|
||||
if (!status_text)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto status = *status_text;
|
||||
if (status == "absent")
|
||||
return ResolvedCommit{Resolution::absent, {}};
|
||||
if (status == "conflict")
|
||||
return ResolvedCommit{Resolution::conflict, {}};
|
||||
const PgRows receipt{
|
||||
std::vector<std::optional<std::string>>((*rows)[0].begin() + 1, (*rows)[0].end())};
|
||||
auto head = status == "committed" ? parse_head(receipt) : std::nullopt;
|
||||
if (!head)
|
||||
return std::unexpected{malformed(false)};
|
||||
return ResolvedCommit{Resolution::committed, *head};
|
||||
}
|
||||
|
||||
namespace {
|
||||
std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
decisions_from(const PgRows& rows) {
|
||||
std::vector<CommittedDecision> out;
|
||||
out.reserve(rows.size());
|
||||
for (const auto& r : rows) {
|
||||
if (r.size() != 5)
|
||||
return std::unexpected{malformed(false)};
|
||||
const auto sequence = parse_integer<std::uint64_t>(r[0]);
|
||||
const auto tick = parse_integer<std::uint64_t>(r[1]);
|
||||
const auto actor = parse_integer<std::uint32_t>(r[2]);
|
||||
const auto outcome = parse_integer<std::uint32_t>(r[3]);
|
||||
const auto organism = parse_integer<std::uint64_t>(r[4]);
|
||||
if (!sequence || !tick || !actor || !outcome || !organism || *actor > 1 || *outcome != 1)
|
||||
return std::unexpected{malformed(false)};
|
||||
out.push_back({*sequence, {*tick, static_cast<Actor>(*actor), Outcome::death, *organism}});
|
||||
}
|
||||
return out;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
load_region_decisions(Postgres& db, RegionKey key, milliseconds timeout) {
|
||||
const std::array<std::string, 2> p{std::to_string(key.world), std::to_string(key.region)};
|
||||
auto rows = db.query(
|
||||
"SELECT * FROM evolution.load_region_decisions($1::evolution.u64, $2::evolution.u64)", p,
|
||||
timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
return decisions_from(*rows);
|
||||
}
|
||||
|
||||
std::expected<std::vector<CommittedDecision>, LeaseStoreFailure>
|
||||
pending_region_outbox(Postgres& db, RegionKey key, std::uint32_t limit, milliseconds timeout) {
|
||||
const std::array<std::string, 3> p{std::to_string(key.world), std::to_string(key.region),
|
||||
std::to_string(limit)};
|
||||
auto rows = db.query("SELECT * FROM evolution.pending_region_outbox($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::integer)",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
return decisions_from(*rows);
|
||||
}
|
||||
|
||||
std::expected<bool, LeaseStoreFailure> acknowledge_region_outbox(Postgres& db, RegionKey key,
|
||||
const BatchDecision& d,
|
||||
milliseconds timeout) {
|
||||
const std::array<std::string, 5> p{std::to_string(key.world), std::to_string(key.region),
|
||||
std::to_string(static_cast<std::uint32_t>(d.outcome)),
|
||||
std::to_string(static_cast<std::uint32_t>(d.actor)),
|
||||
std::to_string(d.tick)};
|
||||
auto rows = db.query("SELECT evolution.acknowledge_region_outbox($1::evolution.u64, "
|
||||
"$2::evolution.u64, $3::integer, $4::integer, $5::evolution.u64)::text",
|
||||
p, timeout);
|
||||
if (!rows)
|
||||
return std::unexpected{classify(rows.error())};
|
||||
if (rows->size() != 1 || (*rows)[0].size() != 1)
|
||||
return std::unexpected{malformed(true)};
|
||||
const auto& marked = (*rows)[0][0];
|
||||
if (!marked)
|
||||
return std::unexpected{malformed(true)};
|
||||
return *marked == "true";
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
|
||||
@@ -0,0 +1,153 @@
|
||||
#include "evo/persistence/region_batch.hpp"
|
||||
#include "fixtures/environmental_region.hpp"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdio>
|
||||
#include <source_location>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
|
||||
namespace {
|
||||
using namespace evo::persistence;
|
||||
using evo::processes::EnvironmentalCommunity;
|
||||
using evo::processes::LineageRegion;
|
||||
using evo::test::region_input;
|
||||
void check(bool ok, std::source_location at = std::source_location::current()) {
|
||||
if (!ok)
|
||||
throw std::runtime_error("region batch line " + std::to_string(at.line()));
|
||||
}
|
||||
#define CHECK(...) check(static_cast<bool>(__VA_ARGS__))
|
||||
template <std::size_t N> std::array<std::uint8_t, N> filled(std::uint8_t value) {
|
||||
std::array<std::uint8_t, N> out{};
|
||||
out.fill(value);
|
||||
return out;
|
||||
}
|
||||
TraceHeader header(std::uint64_t begin, std::uint64_t end) {
|
||||
return {1, {7, 3}, filled<16>(1), filled<32>(2), begin, end};
|
||||
}
|
||||
|
||||
void recorder_seals_at_outcomes_and_capacity() {
|
||||
auto state = evo::test::region_community();
|
||||
RegionBatch<EnvironmentalCommunity> batch{*state};
|
||||
CHECK(batch.begin_tick() == 0 && !batch.sealed());
|
||||
CHECK(!batch.confirm(0)); // Nothing to confirm.
|
||||
for (std::uint64_t tick = 1; tick <= 19; ++tick)
|
||||
CHECK(batch.advance(region_input(tick, true)));
|
||||
CHECK(!batch.sealed() && batch.end_tick() == 19);
|
||||
auto death = batch.advance(region_input(20, true));
|
||||
CHECK(death && death->consumer_transition == 1);
|
||||
// A permanent outcome pauses the region until durable confirmation.
|
||||
CHECK(batch.sealed() && batch.advance(region_input(21)).error() == BatchError::sealed);
|
||||
CHECK(state->progress().tick == 20);
|
||||
const auto decisions = derive_decisions(batch.steps(), 0, 0);
|
||||
CHECK(decisions.size() == 1 && decisions[0] == BatchDecision{20, Actor::consumer});
|
||||
CHECK(!batch.confirm(19) && batch.sealed());
|
||||
CHECK(batch.confirm(20) && !batch.sealed() && batch.begin_tick() == 20);
|
||||
CHECK(batch.steps().empty());
|
||||
// 64 ticks without an outcome also seals; an early flush is explicit.
|
||||
for (std::uint64_t i = 0; i < 64; ++i)
|
||||
CHECK(batch.advance(region_input(21 + i)));
|
||||
CHECK(batch.sealed() && batch.steps().size() == 64);
|
||||
CHECK(batch.confirm(84));
|
||||
CHECK(batch.advance(region_input(85)));
|
||||
batch.seal();
|
||||
CHECK(batch.sealed() && batch.confirm(85));
|
||||
// A refused tick records nothing.
|
||||
evo::processes::EnvironmentalInput bad = region_input(86);
|
||||
bad.terminal[1] = 1; // The consumer is already dead.
|
||||
CHECK(batch.advance(bad).error() == BatchError::state && batch.steps().empty());
|
||||
}
|
||||
|
||||
void lineage_recorder_uses_archived_organisms() {
|
||||
auto region = evo::test::lineage_region();
|
||||
RegionBatch<LineageRegion> batch{*region};
|
||||
for (std::uint64_t tick = 1; tick <= 20; ++tick)
|
||||
CHECK(batch.advance(region_input(tick, true)));
|
||||
CHECK(batch.sealed());
|
||||
const auto d =
|
||||
derive_decisions(batch.steps(), region->binding().producer, region->binding().consumer);
|
||||
CHECK(d.size() == 1 && d[0].organism == 11 && d[0].tick == 20);
|
||||
CHECK(!region->archive().organism(11)->alive);
|
||||
}
|
||||
|
||||
void traces_round_trip_and_bind_their_place() {
|
||||
auto state = evo::test::region_community();
|
||||
RegionBatch<EnvironmentalCommunity> batch{*state};
|
||||
for (std::uint64_t tick = 1; tick <= 20; ++tick)
|
||||
CHECK(batch.advance(region_input(tick, true)));
|
||||
auto bytes = encode_trace(header(0, 20), batch.steps());
|
||||
CHECK(bytes && bytes->size() == trace_bytes_for(20) && bytes->size() == 96 + (20 * 304));
|
||||
auto decoded = decode_trace(*bytes);
|
||||
CHECK(decoded && decoded->header == header(0, 20));
|
||||
CHECK(std::ranges::equal(decoded->steps, batch.steps()));
|
||||
CHECK(trace_bytes_for(64) == 19552);
|
||||
|
||||
const auto refused = [&](TraceHeader h, std::size_t count) {
|
||||
return !encode_trace(h, batch.steps().first(count));
|
||||
};
|
||||
CHECK(refused(header(0, 19), 20)); // Count disagrees with the range.
|
||||
CHECK(refused(header(1, 21), 20)); // Reports are for ticks 1..20.
|
||||
auto h = header(0, 20);
|
||||
h.predecessor_digest = {};
|
||||
CHECK(refused(h, 20));
|
||||
h = header(0, 20);
|
||||
h.key.region = 0;
|
||||
CHECK(refused(h, 20));
|
||||
CHECK(!encode_trace(header(0, 0), {}));
|
||||
auto damaged = *bytes;
|
||||
damaged[0] ^= 1U;
|
||||
CHECK(!decode_trace(damaged));
|
||||
damaged = *bytes;
|
||||
damaged.pop_back();
|
||||
CHECK(!decode_trace(damaged));
|
||||
damaged = *bytes;
|
||||
damaged[96 + 72] = 7; // First report's tick: no longer contiguous.
|
||||
CHECK(!decode_trace(damaged));
|
||||
damaged = *bytes;
|
||||
damaged[88] = 21; // End tick 21 with 20 steps: size check.
|
||||
CHECK(!decode_trace(damaged));
|
||||
}
|
||||
|
||||
void decisions_are_canonical() {
|
||||
const BatchDecision producer{5, Actor::producer, Outcome::death, 10};
|
||||
const BatchDecision consumer{5, Actor::consumer, Outcome::death, 11};
|
||||
const std::array ok{producer, consumer};
|
||||
auto bytes = encode_decisions(ok, 0, 5);
|
||||
CHECK(bytes && bytes->size() == 2 * decision_bytes);
|
||||
auto back = decode_decisions(*bytes, 0, 5);
|
||||
CHECK(back && std::ranges::equal(*back, ok));
|
||||
CHECK(encode_decisions({}, 0, 5) && encode_decisions({}, 0, 5)->empty());
|
||||
const std::array reversed{consumer, producer};
|
||||
CHECK(!encode_decisions(reversed, 0, 5));
|
||||
const std::array twice{producer, BatchDecision{6, Actor::producer}};
|
||||
CHECK(!encode_decisions(twice, 0, 6)); // An actor dies once.
|
||||
CHECK(!encode_decisions(std::array{producer}, 5, 9)); // Tick not in (begin, end].
|
||||
CHECK(!encode_decisions(std::array{producer}, 0, 4));
|
||||
auto damaged = *bytes;
|
||||
damaged[8] = 2; // Unknown actor.
|
||||
CHECK(!decode_decisions(damaged, 0, 5));
|
||||
damaged = *bytes;
|
||||
damaged[12] = 2; // Unknown outcome.
|
||||
CHECK(!decode_decisions(damaged, 0, 5));
|
||||
damaged = *bytes;
|
||||
damaged[9] = 1; // Actor 256: never truncated onto a known actor.
|
||||
CHECK(!decode_decisions(damaged, 0, 5));
|
||||
CHECK(!decode_decisions(std::span{*bytes}.first(23), 0, 5));
|
||||
std::vector<std::uint8_t> nine(9 * decision_bytes);
|
||||
CHECK(!decode_decisions(nine, 0, 5));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
int main() {
|
||||
try {
|
||||
recorder_seals_at_outcomes_and_capacity();
|
||||
lineage_recorder_uses_archived_organisms();
|
||||
traces_round_trip_and_bind_their_place();
|
||||
decisions_are_canonical();
|
||||
std::puts("region batch contracts passed");
|
||||
return 0;
|
||||
} catch (const std::exception& e) {
|
||||
std::fprintf(stderr, "%s\n", e.what());
|
||||
return 1;
|
||||
}
|
||||
}
|
||||
@@ -7,10 +7,12 @@
|
||||
#include <cstdlib>
|
||||
#include <filesystem>
|
||||
#include <fstream>
|
||||
#include <optional>
|
||||
#include <source_location>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <sys/stat.h>
|
||||
#include <thread>
|
||||
#include <unistd.h>
|
||||
|
||||
void run_region_checkpoint_store_tests();
|
||||
@@ -172,6 +174,15 @@ std::size_t count_objects(const fs::path& dir) {
|
||||
return n;
|
||||
}
|
||||
|
||||
using EnvBatch = RegionBatch<evo::processes::EnvironmentalCommunity>;
|
||||
using LineageBatch = RegionBatch<evo::processes::LineageRegion>;
|
||||
// Runs up to `ticks` more ticks (stopping early if an outcome seals it), then flushes.
|
||||
template <typename Batch> void run_batch(Batch& b, std::uint64_t ticks, bool deaths = false) {
|
||||
for (std::uint64_t i = 0; i < ticks && !b.sealed(); ++i)
|
||||
CHECK(b.advance(evo::test::region_input(b.end_tick() + 1, deaths)));
|
||||
b.seal();
|
||||
}
|
||||
|
||||
void publish_recover_and_continue() {
|
||||
auto r = provisioned("publish", 3, 0xC0);
|
||||
const std::array<Sha256Digest, 1> retained{build};
|
||||
@@ -180,7 +191,7 @@ void publish_recover_and_continue() {
|
||||
// Genesis recovers to the exact opening state.
|
||||
auto recovered =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(recovered && recovered->state->progress().tick == 0);
|
||||
CHECK(recovered && recovered->state->progress().tick == 0 && recovered->decisions.empty());
|
||||
auto head = recovered->committed;
|
||||
|
||||
// Genesis cannot be re-provisioned or published over.
|
||||
@@ -189,31 +200,55 @@ void publish_recover_and_continue() {
|
||||
filled<16>(0xC9), build, budget, 5s));
|
||||
|
||||
auto original = evo::test::region_community();
|
||||
EnvBatch batch{*original};
|
||||
// An unsealed batch is still being simulated: refused before any upload.
|
||||
CHECK(batch.advance(evo::test::region_input(1)));
|
||||
const auto before = count_objects(r.objects->directory());
|
||||
auto open = publish_environmental_batch(*r.runtime, *r.objects, *lease, head, batch,
|
||||
filled<16>(0xCA), build, budget, 5s);
|
||||
CHECK(!open && open.error().kind == PublicationError::candidate);
|
||||
CHECK(count_objects(r.objects->directory()) == before);
|
||||
std::uint8_t seed = 0xC1;
|
||||
for (int batch = 0; batch < 3; ++batch) {
|
||||
evo::test::advance_region(*original, batch == 1 ? 64 : 16);
|
||||
for (int n = 0; n < 3; ++n) {
|
||||
run_batch(batch, n == 1 ? 64 : 15);
|
||||
const auto started = std::chrono::steady_clock::now();
|
||||
auto published = publish_environmental_checkpoint(
|
||||
*r.runtime, *r.objects, *lease, head, *original, filled<16>(seed++), build, budget, 5s);
|
||||
auto published = publish_environmental_batch(*r.runtime, *r.objects, *lease, head, batch,
|
||||
filled<16>(seed++), build, budget, 5s);
|
||||
const auto us = std::chrono::duration_cast<std::chrono::microseconds>(
|
||||
std::chrono::steady_clock::now() - started)
|
||||
.count();
|
||||
CHECK(published && published->head.sequence == head.head.sequence + 1);
|
||||
CHECK(published->head.end_tick == original->progress().tick);
|
||||
std::printf(
|
||||
"checkpoint_publish_us batch=%d ticks=%llu payload_bytes=%llu us=%lld\n", batch,
|
||||
static_cast<unsigned long long>(published->head.end_tick - head.head.end_tick),
|
||||
static_cast<unsigned long long>(decode_manifest(published->manifest)->payload_bytes),
|
||||
static_cast<long long>(us));
|
||||
head = std::move(*published);
|
||||
CHECK(published && published->committed.head.sequence == head.head.sequence + 1);
|
||||
CHECK(published->committed.head.end_tick == original->progress().tick);
|
||||
CHECK(published->decisions.empty());
|
||||
const auto m = *decode_manifest(published->committed.manifest);
|
||||
CHECK(m.manifest_version == 2 &&
|
||||
m.batch.trace_bytes == trace_bytes_for(m.end_tick - m.begin_tick));
|
||||
std::printf("batch_publish_us batch=%d ticks=%llu payload_bytes=%llu us=%lld\n", n,
|
||||
static_cast<unsigned long long>(m.end_tick - m.begin_tick),
|
||||
static_cast<unsigned long long>(m.payload_bytes), static_cast<long long>(us));
|
||||
CHECK(batch.confirm(m.end_tick));
|
||||
head = std::move(published->committed);
|
||||
// The trace object is retained, bound to this link of the chain.
|
||||
auto trace = r.objects->get({m.batch.trace_sha256, m.batch.trace_bytes});
|
||||
auto decoded = trace ? decode_trace(*trace) : std::unexpected{BatchError::format};
|
||||
CHECK(decoded && decoded->header.begin_tick == m.begin_tick &&
|
||||
decoded->header.predecessor_digest == m.predecessor_digest);
|
||||
}
|
||||
// 65 ticks is more than one batch: refused before upload.
|
||||
const auto before = count_objects(r.objects->directory());
|
||||
evo::test::advance_region(*original, 65);
|
||||
auto oversized = publish_environmental_checkpoint(
|
||||
*r.runtime, *r.objects, *lease, head, *original, filled<16>(0xCA), build, budget, 5s);
|
||||
CHECK(!oversized && oversized.error().kind == PublicationError::candidate);
|
||||
CHECK(count_objects(r.objects->directory()) == before);
|
||||
// State advanced outside the recorded batch cannot be published: the replay of
|
||||
// the recorded inputs no longer reproduces it.
|
||||
auto drifted = evo::test::region_community();
|
||||
EnvBatch drift{*drifted};
|
||||
run_batch(drift, 3);
|
||||
CHECK(drifted->advance(4, evo::test::region_input(4)));
|
||||
auto diverged =
|
||||
publish_environmental_batch(*r.runtime, *r.objects, *lease, recovered->committed, drift,
|
||||
filled<16>(0xCD), build, budget, 5s);
|
||||
CHECK(!diverged && diverged.error().kind == PublicationError::candidate);
|
||||
// A batch replayed from the wrong predecessor cannot reproduce its state.
|
||||
run_batch(batch, 4);
|
||||
auto stale = publish_environmental_batch(*r.runtime, *r.objects, *lease, recovered->committed,
|
||||
batch, filled<16>(0xCB), build, budget, 5s);
|
||||
CHECK(!stale && stale.error().kind == PublicationError::candidate);
|
||||
|
||||
// A new process recovers the committed head; its trajectory matches the original's.
|
||||
auto second = FilesystemObjectStore::open(r.objects->directory().parent_path().parent_path(),
|
||||
@@ -222,11 +257,11 @@ void publish_recover_and_continue() {
|
||||
auto restored =
|
||||
recover_environmental_checkpoint(*r.runtime, **second, r.key, retained, budget, 5s);
|
||||
CHECK(restored && restored->committed.head == head.head);
|
||||
CHECK(restored->state->progress().tick == 96);
|
||||
CHECK(restored->state->progress().tick == 95);
|
||||
auto reference = evo::test::region_community();
|
||||
evo::test::advance_region(*reference, 96);
|
||||
evo::test::advance_region(*reference, 95);
|
||||
CHECK(reference->snapshot() == restored->state->snapshot());
|
||||
for (std::uint64_t tick = 97; tick <= 140; ++tick) {
|
||||
for (std::uint64_t tick = 96; tick <= 140; ++tick) {
|
||||
auto a = reference->advance(tick, evo::test::region_input(tick, true));
|
||||
auto b = restored->state->advance(tick, evo::test::region_input(tick, true));
|
||||
CHECK(a && b && *a == *b);
|
||||
@@ -238,11 +273,9 @@ void publish_recover_and_continue() {
|
||||
auto foreign =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, other, budget, 5s);
|
||||
CHECK(!foreign && foreign.error().kind == RecoveryError::unsupported);
|
||||
// Unretained build is refused by the database; the upload stays an orphan.
|
||||
auto next = evo::test::region_community();
|
||||
evo::test::advance_region(*next, 100);
|
||||
auto refused = publish_environmental_checkpoint(*r.runtime, *r.objects, *lease, head, *next,
|
||||
filled<16>(0xCB), filled<32>(0xB2), budget, 5s);
|
||||
// Unretained build is refused by the database; the uploads stay orphans.
|
||||
auto refused = publish_environmental_batch(*r.runtime, *r.objects, *lease, head, batch,
|
||||
filled<16>(0xCC), filled<32>(0xB2), budget, 5s);
|
||||
CHECK(!refused && refused.error().kind == PublicationError::rejected &&
|
||||
refused.error().store.kind == LeaseStoreError::unsupported);
|
||||
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
|
||||
@@ -256,32 +289,34 @@ void failures_leave_the_previous_head() {
|
||||
auto head = load_region_checkpoint(*r.runtime, r.key, 5s);
|
||||
CHECK(head);
|
||||
auto state = evo::test::region_community();
|
||||
evo::test::advance_region(*state, 8);
|
||||
auto first = publish_environmental_checkpoint(*r.runtime, *r.objects, *lease, *head, *state,
|
||||
filled<16>(0xD1), build, budget, 5s);
|
||||
CHECK(first);
|
||||
const auto manifest = *decode_manifest(first->manifest);
|
||||
EnvBatch batch{*state};
|
||||
run_batch(batch, 8);
|
||||
auto first = publish_environmental_batch(*r.runtime, *r.objects, *lease, *head, batch,
|
||||
filled<16>(0xD1), build, budget, 5s);
|
||||
CHECK(first && batch.confirm(8));
|
||||
const auto manifest = *decode_manifest(first->committed.manifest);
|
||||
const auto object = r.objects->directory() / hex(manifest.payload_sha256);
|
||||
|
||||
// Upload failure: no manifest is sent; the head does not move.
|
||||
evo::test::advance_region(*state, 8);
|
||||
run_batch(batch, 8);
|
||||
if (::geteuid() != 0) {
|
||||
fs::permissions(r.objects->directory(), fs::perms::owner_read | fs::perms::owner_exec,
|
||||
fs::perm_options::replace);
|
||||
auto upload = publish_environmental_checkpoint(*r.runtime, *r.objects, *lease, *first,
|
||||
*state, filled<16>(0xD2), build, budget, 5s);
|
||||
auto upload = publish_environmental_batch(*r.runtime, *r.objects, *lease, first->committed,
|
||||
batch, filled<16>(0xD2), build, budget, 5s);
|
||||
fs::permissions(r.objects->directory(), fs::perms::owner_all, fs::perm_options::replace);
|
||||
CHECK(!upload && upload.error().kind == PublicationError::upload);
|
||||
CHECK(load_region_head(*r.runtime, r.key, 5s)->sequence == 1);
|
||||
}
|
||||
|
||||
// Manifest transaction failure after a verified upload: fenced writer, orphan retained.
|
||||
// Manifest transaction failure after verified uploads: fenced writer, orphans retained.
|
||||
auto stale = *lease;
|
||||
stale.holder[0] ^= 1U;
|
||||
auto fenced = publish_environmental_checkpoint(*r.runtime, *r.objects, stale, *first, *state,
|
||||
filled<16>(0xD3), build, budget, 5s);
|
||||
auto fenced = publish_environmental_batch(*r.runtime, *r.objects, stale, first->committed,
|
||||
batch, filled<16>(0xD3), build, budget, 5s);
|
||||
CHECK(!fenced && fenced.error().kind == PublicationError::rejected &&
|
||||
fenced.error().store.kind == LeaseStoreError::fenced);
|
||||
CHECK(fenced.error().prepared.commit == filled<16>(0xD3));
|
||||
auto orphan_bytes = state->snapshot();
|
||||
auto orphan = r.objects->get({*sha256(*orphan_bytes), orphan_bytes->size()});
|
||||
CHECK(orphan && *orphan == *orphan_bytes); // Uploaded, verified, unreferenced.
|
||||
@@ -296,6 +331,11 @@ void failures_leave_the_previous_head() {
|
||||
auto missing =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(!missing && missing.error().kind == RecoveryError::missing_payload);
|
||||
// Staging replays from the committed predecessor bytes; without them it stops.
|
||||
auto no_base = publish_environmental_batch(*r.runtime, *r.objects, *lease, first->committed,
|
||||
batch, filled<16>(0xD4), build, budget, 5s);
|
||||
CHECK(!no_base && no_base.error().kind == PublicationError::predecessor &&
|
||||
no_base.error().object == ObjectError::missing);
|
||||
CHECK(r.objects->put(*saved));
|
||||
fs::permissions(object, fs::perms::owner_all, fs::perm_options::replace);
|
||||
{
|
||||
@@ -310,18 +350,26 @@ void failures_leave_the_previous_head() {
|
||||
CHECK(r.objects->put(*saved));
|
||||
CHECK(recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s));
|
||||
|
||||
// The database refuses tampering with committed manifests even by the owner.
|
||||
// Committed history is immutable even to the owner (trigger), and the head
|
||||
// cannot be repointed (composite foreign key).
|
||||
auto tamper = r.owner->query("UPDATE evolution.region_commits SET manifest = "
|
||||
"overlay(manifest placing '\\x02'::bytea from 121 for 1) "
|
||||
"WHERE world_id = 7 AND region_id = 4 AND commit_sequence = 1",
|
||||
{}, 5s);
|
||||
CHECK(!tamper && std::string(tamper.error().sqlstate.data()) == "23514");
|
||||
CHECK(!tamper && std::string(tamper.error().sqlstate.data()) == "EVL09");
|
||||
auto erase = r.owner->query("DELETE FROM evolution.region_commits WHERE world_id = 7 AND "
|
||||
"region_id = 4 AND commit_sequence = 1",
|
||||
{}, 5s);
|
||||
CHECK(!erase && std::string(erase.error().sqlstate.data()) == "EVL09");
|
||||
auto repoint = r.owner->query("UPDATE evolution.region_heads SET digest = sha256('\\x00') "
|
||||
"WHERE world_id = 7 AND region_id = 4",
|
||||
{}, 5s);
|
||||
CHECK(!repoint && std::string(repoint.error().sqlstate.data()) == "23503");
|
||||
// Defense in depth: even if that constraint were lost, recovery re-digests the
|
||||
// manifest and quarantines. Restores the constraint before continuing.
|
||||
// Defense in depth: even with the trigger and digest constraint removed, recovery
|
||||
// re-digests the manifest and quarantines. Both guards are restored afterwards.
|
||||
CHECK(r.owner->query("ALTER TABLE evolution.region_commits DISABLE TRIGGER "
|
||||
"region_commits_immutable",
|
||||
{}, 5s));
|
||||
CHECK(r.owner->query("ALTER TABLE evolution.region_commits DROP CONSTRAINT "
|
||||
"region_commits_manifest_digest",
|
||||
{}, 5s));
|
||||
@@ -333,13 +381,16 @@ void failures_leave_the_previous_head() {
|
||||
auto tampered =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(!tampered && tampered.error().kind == RecoveryError::manifest);
|
||||
const std::array<std::string, 1> original{"\\x" + hex(first->manifest)};
|
||||
const std::array<std::string, 1> original{"\\x" + hex(first->committed.manifest)};
|
||||
CHECK(r.owner->query("UPDATE evolution.region_commits SET manifest = $1::bytea "
|
||||
"WHERE world_id = 7 AND region_id = 4 AND commit_sequence = 1",
|
||||
original, 5s));
|
||||
CHECK(r.owner->query("ALTER TABLE evolution.region_commits ADD CONSTRAINT "
|
||||
"region_commits_manifest_digest CHECK (commit_digest = sha256(manifest))",
|
||||
{}, 5s));
|
||||
CHECK(r.owner->query("ALTER TABLE evolution.region_commits ENABLE TRIGGER "
|
||||
"region_commits_immutable",
|
||||
{}, 5s));
|
||||
CHECK(recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s));
|
||||
|
||||
// Recovery does not create genesis for an unknown region.
|
||||
@@ -347,6 +398,7 @@ void failures_leave_the_previous_head() {
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, {7, 99}, retained, budget, 5s);
|
||||
CHECK(!none && none.error().kind == RecoveryError::not_provisioned);
|
||||
}
|
||||
|
||||
// History-capable profile: deaths commit with the state that caused them, the
|
||||
// archive survives recovery, and a region never switches profile.
|
||||
void lineage_publish_and_recover() {
|
||||
@@ -369,50 +421,260 @@ void lineage_publish_and_recover() {
|
||||
CHECK(lease);
|
||||
|
||||
auto region = evo::test::lineage_region();
|
||||
evo::test::advance_lineage(*region, 24, true); // Consumer dies at tick 20.
|
||||
auto first = publish_lineage_checkpoint(*runtime, **objects, *lease, *genesis, *region,
|
||||
filled<16>(0xE1), build, budget, 5s);
|
||||
CHECK(first && first->head.end_tick == 24);
|
||||
LineageBatch batch{*region};
|
||||
run_batch(batch, 64, true); // The consumer's death at tick 20 seals the batch there.
|
||||
CHECK(batch.end_tick() == 20);
|
||||
auto first = publish_lineage_batch(*runtime, **objects, *lease, *genesis, batch,
|
||||
filled<16>(0xE1), build, budget, 5s);
|
||||
CHECK(first && first->committed.head.end_tick == 20);
|
||||
CHECK(first->decisions.size() == 1 &&
|
||||
first->decisions[0] == BatchDecision{20, Actor::consumer, Outcome::death, 11});
|
||||
CHECK(batch.confirm(20));
|
||||
// The environmental path refuses a lineage predecessor before uploading.
|
||||
auto plain = evo::test::region_community();
|
||||
evo::test::advance_region(*plain, 30);
|
||||
auto crossed = publish_environmental_checkpoint(*runtime, **objects, *lease, *first, *plain,
|
||||
filled<16>(0xE2), build, budget, 5s);
|
||||
EnvBatch plain_batch{*plain};
|
||||
run_batch(plain_batch, 30);
|
||||
auto crossed = publish_environmental_batch(*runtime, **objects, *lease, first->committed,
|
||||
plain_batch, filled<16>(0xE2), build, budget, 5s);
|
||||
CHECK(!crossed && crossed.error().kind == PublicationError::candidate);
|
||||
// The database refuses a profile switch even when the manifest is well formed.
|
||||
auto switched = *decode_manifest(first->manifest);
|
||||
auto next = successor_manifest(switched, first->head.digest, filled<16>(0xE3), *lease, 30,
|
||||
build, 100, filled<32>(0xA2));
|
||||
auto switched = *decode_manifest(first->committed.manifest);
|
||||
auto next = successor_manifest(switched, first->committed.head.digest, filled<16>(0xE3), *lease,
|
||||
30, build, 100, filled<32>(0xA2),
|
||||
{trace_bytes_for(10), filled<32>(0x7C), 0, {}});
|
||||
CHECK(next);
|
||||
next->profile = CheckpointProfile::environmental_v1;
|
||||
next->history_profile = 0;
|
||||
auto switched_bytes = encode_manifest(*next);
|
||||
CHECK(switched_bytes);
|
||||
auto refused = publish_region_checkpoint(*runtime, *switched_bytes, 5s);
|
||||
auto refused = publish_region_checkpoint(*runtime, *switched_bytes, {}, 5s);
|
||||
CHECK(!refused && refused.error().kind == LeaseStoreError::unsupported);
|
||||
|
||||
const std::array<Sha256Digest, 1> retained{build};
|
||||
auto recovered = recover_lineage_checkpoint(*runtime, **objects, key, retained, budget, 5s);
|
||||
CHECK(recovered && recovered->committed.head == first->head);
|
||||
CHECK(recovered && recovered->committed.head == first->committed.head);
|
||||
CHECK(recovered->decisions.size() == 1 && recovered->decisions[0].commit_sequence == 1);
|
||||
const auto consumer = recovered->state->archive().organism(11);
|
||||
CHECK(consumer && !consumer->alive && consumer->died.value == 20);
|
||||
CHECK(recovered->state->archive().ancestry(2)->size() == 2); // Retained ancestry.
|
||||
CHECK(recovered->state->snapshot() == region->snapshot());
|
||||
for (std::uint64_t tick = 25; tick <= 60; ++tick) {
|
||||
LineageBatch resumed{*recovered->state};
|
||||
for (std::uint64_t tick = 21; tick <= 60; ++tick) {
|
||||
auto a = region->advance(tick, evo::test::region_input(tick));
|
||||
auto b = recovered->state->advance(tick, evo::test::region_input(tick));
|
||||
auto b = resumed.advance(evo::test::region_input(tick));
|
||||
CHECK(a && b && *a == *b);
|
||||
}
|
||||
CHECK(recovered->state->snapshot() == region->snapshot());
|
||||
auto second =
|
||||
publish_lineage_checkpoint(*runtime, **objects, *lease, recovered->committed,
|
||||
*recovered->state, filled<16>(0xE4), build, budget, 5s);
|
||||
CHECK(second && second->head.sequence == 2 && second->head.end_tick == 60);
|
||||
resumed.seal();
|
||||
auto second = publish_lineage_batch(*runtime, **objects, *lease, recovered->committed, resumed,
|
||||
filled<16>(0xE4), build, budget, 5s);
|
||||
CHECK(second && second->committed.head.sequence == 2 && second->committed.head.end_tick == 60 &&
|
||||
second->decisions.empty());
|
||||
// A host restoring the other profile refuses rather than misreading the payload.
|
||||
auto wrong = recover_environmental_checkpoint(*runtime, **objects, key, retained, budget, 5s);
|
||||
CHECK(!wrong && wrong.error().kind == RecoveryError::unsupported);
|
||||
CHECK(release_region_lease(*runtime, key, *lease, 5s));
|
||||
}
|
||||
|
||||
std::string runtime_user() {
|
||||
return env("EVO_PG_RUNTIME_USER");
|
||||
}
|
||||
// Polls resolution on a fresh connection until the prepared commit is committed.
|
||||
ResolvedCommit await_resolution(const PreparedCommit& prepared, Resolution wanted) {
|
||||
auto db = connect_as(runtime_user());
|
||||
for (int attempt = 0; attempt < 200; ++attempt) {
|
||||
auto resolved = resolve_prepared(*db, prepared, 5s);
|
||||
CHECK(resolved);
|
||||
if (resolved->status == wanted)
|
||||
return *resolved;
|
||||
std::this_thread::sleep_for(25ms);
|
||||
}
|
||||
CHECK(false);
|
||||
return {};
|
||||
}
|
||||
// Holds the lease row lock so a publication's outcome becomes ambiguous to its client.
|
||||
struct LeaseRowLock {
|
||||
std::unique_ptr<Postgres> db;
|
||||
LeaseRowLock(RegionKey key) : db(connect_as(env("PGUSER"))) {
|
||||
const std::array<std::string, 2> p{std::to_string(key.world), std::to_string(key.region)};
|
||||
CHECK(db->query("BEGIN", {}, 5s));
|
||||
CHECK(db->query("SELECT 1 FROM evolution.region_leases WHERE world_id = $1::evolution.u64 "
|
||||
"AND region_id = $2::evolution.u64 FOR UPDATE",
|
||||
p, 5s));
|
||||
}
|
||||
// Terminates backends of the runtime login queued behind this lock.
|
||||
void kill_waiters() {
|
||||
const std::array<std::string, 1> p{runtime_user()};
|
||||
CHECK(db->query("SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE "
|
||||
"usename = $1 AND wait_event_type = 'Lock'",
|
||||
p, 5s));
|
||||
}
|
||||
void release() {
|
||||
CHECK(db->query("ROLLBACK", {}, 5s));
|
||||
}
|
||||
};
|
||||
|
||||
// M7.5 durability: permanent outcomes commit atomically with their checkpoint,
|
||||
// ambiguity is resolved by exact bytes, and nothing is ever applied twice.
|
||||
void durable_outcomes_and_ambiguity() {
|
||||
auto r = provisioned("outcomes", 7, 0xF0);
|
||||
const std::array<Sha256Digest, 1> retained{build};
|
||||
auto lease = acquire_region_lease(*r.runtime, r.key, 20000ms, 5s);
|
||||
CHECK(lease);
|
||||
auto genesis = load_region_checkpoint(*r.runtime, r.key, 5s);
|
||||
CHECK(genesis);
|
||||
auto state = evo::test::region_community();
|
||||
EnvBatch batch{*state};
|
||||
run_batch(batch, 64, true); // Sealed by the consumer's death at tick 20.
|
||||
CHECK(batch.end_tick() == 20);
|
||||
auto died = publish_environmental_batch(*r.runtime, *r.objects, *lease, *genesis, batch,
|
||||
filled<16>(0xF1), build, budget, 5s);
|
||||
CHECK(died && died->decisions.size() == 1 &&
|
||||
died->decisions[0] == BatchDecision{20, Actor::consumer, Outcome::death, 0});
|
||||
CHECK(batch.confirm(20));
|
||||
const CommittedDecision final_death{1, died->decisions[0]};
|
||||
auto decisions = load_region_decisions(*r.runtime, r.key, 5s);
|
||||
CHECK(decisions && decisions->size() == 1 && (*decisions)[0] == final_death);
|
||||
|
||||
// Outbox: at-least-once delivery, idempotent acknowledgement, no simulation effect.
|
||||
auto pending = pending_region_outbox(*r.runtime, r.key, 64, 5s);
|
||||
CHECK(pending && pending->size() == 1 && (*pending)[0] == final_death);
|
||||
CHECK(pending_region_outbox(*r.runtime, r.key, 64, 5s)->size() == 1); // Redelivered.
|
||||
auto ack = acknowledge_region_outbox(*r.runtime, r.key, died->decisions[0], 5s);
|
||||
CHECK(ack && *ack);
|
||||
ack = acknowledge_region_outbox(*r.runtime, r.key, died->decisions[0], 5s);
|
||||
CHECK(ack && !*ack);
|
||||
CHECK(pending_region_outbox(*r.runtime, r.key, 64, 5s)->empty());
|
||||
CHECK(!pending_region_outbox(*r.runtime, r.key, 65, 5s));
|
||||
const BatchDecision never{21, Actor::producer, Outcome::death, 0};
|
||||
CHECK(!acknowledge_region_outbox(*r.runtime, r.key, never, 5s));
|
||||
|
||||
// Commit succeeded but the response was lost: the exact retry returns the same
|
||||
// receipt and records nothing twice; resolution confirms it under any epoch.
|
||||
auto death_block = encode_decisions(died->decisions, 0, 20);
|
||||
CHECK(death_block);
|
||||
const PreparedCommit lost{r.key, filled<16>(0xF1), died->committed.manifest, *death_block};
|
||||
auto retried = retry_prepared(*r.runtime, lost, 5s);
|
||||
CHECK(retried && *retried == died->committed.head);
|
||||
auto resolved = resolve_prepared(*r.runtime, lost, 5s);
|
||||
CHECK(resolved && resolved->status == Resolution::committed &&
|
||||
resolved->receipt == died->committed.head);
|
||||
CHECK(load_region_decisions(*r.runtime, r.key, 5s)->size() == 1);
|
||||
// Changed bytes under the same ID conflict, whether manifest or decisions differ.
|
||||
auto changed = lost;
|
||||
changed.manifest[288] ^= 1U; // Trace digest.
|
||||
CHECK(retry_prepared(*r.runtime, changed, 5s).error().kind == LeaseStoreError::commit_conflict);
|
||||
CHECK(resolve_prepared(*r.runtime, changed, 5s)->status == Resolution::conflict);
|
||||
auto undecided = lost;
|
||||
undecided.decisions.clear(); // No longer what the manifest digests.
|
||||
CHECK(retry_prepared(*r.runtime, undecided, 5s).error().kind == LeaseStoreError::invalid);
|
||||
CHECK(resolve_prepared(*r.runtime, undecided, 5s)->status == Resolution::conflict);
|
||||
auto elsewhere = lost;
|
||||
elsewhere.key.region = 3;
|
||||
CHECK(resolve_prepared(*r.runtime, elsewhere, 5s)->status == Resolution::conflict);
|
||||
auto unknown_id = lost;
|
||||
unknown_id.commit = filled<16>(0xFE);
|
||||
CHECK(resolve_prepared(*r.runtime, unknown_id, 5s)->status == Resolution::absent);
|
||||
|
||||
// The committed dead stay dead: a later batch cannot finalize the death again.
|
||||
const auto head = died->committed;
|
||||
const std::array again{BatchDecision{25, Actor::consumer, Outcome::death, 0}};
|
||||
auto again_bytes = encode_decisions(again, 20, 30);
|
||||
CHECK(again_bytes);
|
||||
auto relapse = successor_manifest(
|
||||
*decode_manifest(head.manifest), head.head.digest, filled<16>(0xF2), *lease, 30, build, 100,
|
||||
filled<32>(0xA2), {trace_bytes_for(10), filled<32>(0x7C), 1, *sha256(*again_bytes)});
|
||||
CHECK(relapse);
|
||||
auto relapse_bytes = encode_manifest(*relapse);
|
||||
auto repeated = publish_region_checkpoint(*r.runtime, *relapse_bytes, *again_bytes, 5s);
|
||||
CHECK(!repeated && repeated.error().kind == LeaseStoreError::outcome_final);
|
||||
// A block of the right size that is not the one the manifest digests is refused.
|
||||
auto forged_block = *again_bytes;
|
||||
forged_block[0] ^= 1U; // Tick 25 -> 24: still well formed, different bytes.
|
||||
CHECK(publish_region_checkpoint(*r.runtime, *relapse_bytes, forged_block, 5s).error().kind ==
|
||||
LeaseStoreError::invalid);
|
||||
// Partial or reordered decision blocks are refused whole; nothing is recorded.
|
||||
CHECK(publish_region_checkpoint(*r.runtime, *relapse_bytes, {}, 5s).error().kind ==
|
||||
LeaseStoreError::invalid);
|
||||
const std::array two{BatchDecision{25, Actor::producer, Outcome::death, 0},
|
||||
BatchDecision{24, Actor::consumer, Outcome::death, 0}};
|
||||
std::vector<std::uint8_t> reordered(two.size() * decision_bytes);
|
||||
auto ordered = encode_decisions(std::array{two[1], two[0]}, 20, 30);
|
||||
CHECK(ordered);
|
||||
std::copy_n(ordered->begin() + 24, 24, reordered.begin());
|
||||
std::copy_n(ordered->begin(), 24, reordered.begin() + 24);
|
||||
auto reorder = *relapse;
|
||||
reorder.batch = {trace_bytes_for(10), filled<32>(0x7C), 2, *sha256(reordered)};
|
||||
CHECK(publish_region_checkpoint(*r.runtime, *encode_manifest(reorder), reordered, 5s)
|
||||
.error()
|
||||
.kind == LeaseStoreError::invalid);
|
||||
// A gap: a link from an older predecessor is not the committed head's successor.
|
||||
auto gap = successor_manifest(*decode_manifest(genesis->manifest), genesis->head.digest,
|
||||
filled<16>(0xF3), *lease, 30, build, 100, filled<32>(0xA2),
|
||||
{trace_bytes_for(30), filled<32>(0x7C), 0, {}});
|
||||
CHECK(publish_region_checkpoint(*r.runtime, *encode_manifest(*gap), {}, 5s).error().kind ==
|
||||
LeaseStoreError::head_mismatch);
|
||||
CHECK(load_region_head(*r.runtime, r.key, 5s)->sequence == 1);
|
||||
CHECK(load_region_decisions(*r.runtime, r.key, 5s)->size() == 1);
|
||||
|
||||
// Keep authority comfortably valid for the ambiguity cases (sanitizer builds are slow).
|
||||
CHECK(renew_region_lease(*r.runtime, r.key, *lease, 20000ms, 5s));
|
||||
// Ambiguous outcome that did commit: the client times out while queued, the
|
||||
// server finishes after the lock is released. Resolution adopts the receipt.
|
||||
run_batch(batch, 10);
|
||||
std::optional<LeaseRowLock> lock{std::in_place, r.key};
|
||||
auto ambiguous_db = connect_as(runtime_user());
|
||||
auto ambiguous = publish_environmental_batch(*ambiguous_db, *r.objects, *lease, head, batch,
|
||||
filled<16>(0xF4), build, budget, 300ms);
|
||||
CHECK(!ambiguous && ambiguous.error().kind == PublicationError::unknown);
|
||||
CHECK(batch.sealed()); // Still paused: never assume rollback.
|
||||
lock->release();
|
||||
lock.reset();
|
||||
const auto committed = await_resolution(ambiguous.error().prepared, Resolution::committed);
|
||||
CHECK(committed.receipt.sequence == 2 && committed.receipt.end_tick == 30);
|
||||
auto adopted = retry_prepared(*r.runtime, ambiguous.error().prepared, 5s);
|
||||
CHECK(adopted && *adopted == committed.receipt);
|
||||
CHECK(batch.confirm(30));
|
||||
|
||||
// Ambiguous outcome that did not commit: the orphaned request is terminated
|
||||
// before it reaches the row. Absent, so the same bytes are retried while authorized.
|
||||
auto after = load_region_checkpoint(*r.runtime, r.key, 5s);
|
||||
CHECK(after);
|
||||
run_batch(batch, 5);
|
||||
lock.emplace(r.key);
|
||||
auto orphan_db = connect_as(runtime_user());
|
||||
auto orphaned = publish_environmental_batch(*orphan_db, *r.objects, *lease, *after, batch,
|
||||
filled<16>(0xF5), build, budget, 300ms);
|
||||
CHECK(!orphaned && orphaned.error().kind == PublicationError::unknown);
|
||||
lock->kill_waiters();
|
||||
lock->release();
|
||||
lock.reset();
|
||||
std::this_thread::sleep_for(200ms);
|
||||
CHECK(resolve_prepared(*r.runtime, orphaned.error().prepared, 5s)->status ==
|
||||
Resolution::absent);
|
||||
auto recommitted = retry_prepared(*r.runtime, orphaned.error().prepared, 5s);
|
||||
CHECK(recommitted && recommitted->sequence == 3 && recommitted->end_tick == 35);
|
||||
CHECK(batch.confirm(35));
|
||||
|
||||
// Recovery reconciles durable outcomes with the restored lifecycle.
|
||||
auto recovered =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(recovered && recovered->decisions.size() == 1 && recovered->decisions[0] == final_death &&
|
||||
recovered->state->progress().consumer_terminal == 1);
|
||||
CHECK(recovered->state->snapshot() == state->snapshot());
|
||||
// A durable death the state does not show is a history disagreement: quarantine.
|
||||
CHECK(r.owner->query("INSERT INTO evolution.region_decisions(world_id, region_id, outcome, "
|
||||
"actor, tick, organism, commit_sequence) VALUES (7, 7, 1, 0, 30, 0, 2)",
|
||||
{}, 5s));
|
||||
auto disagreed =
|
||||
recover_environmental_checkpoint(*r.runtime, *r.objects, r.key, retained, budget, 5s);
|
||||
CHECK(!disagreed && disagreed.error().kind == RecoveryError::history);
|
||||
auto undo = r.owner->query("DELETE FROM evolution.region_decisions WHERE world_id = 7 AND "
|
||||
"region_id = 7 AND actor = 0",
|
||||
{}, 5s);
|
||||
CHECK(!undo && std::string(undo.error().sqlstate.data()) == "EVL09"); // Stays quarantined.
|
||||
CHECK(release_region_lease(*r.runtime, r.key, *lease, 5s));
|
||||
}
|
||||
} // namespace
|
||||
|
||||
void run_region_checkpoint_store_tests() {
|
||||
@@ -421,4 +683,5 @@ void run_region_checkpoint_store_tests() {
|
||||
publish_recover_and_continue();
|
||||
failures_leave_the_previous_head();
|
||||
lineage_publish_and_recover();
|
||||
durable_outcomes_and_ambiguity();
|
||||
}
|
||||
|
||||
@@ -33,9 +33,14 @@ CheckpointManifest genesis(std::uint64_t payload_bytes = 100) {
|
||||
CHECK(g);
|
||||
return *g;
|
||||
}
|
||||
// Synthetic batch evidence of the right size for `ticks` (content is not checked here).
|
||||
BatchDigests batch_for(std::uint64_t ticks, std::uint32_t decisions = 0) {
|
||||
return {trace_bytes_for(ticks), filled<32>(0x7C), decisions,
|
||||
decisions == 0 ? Sha256Digest{} : filled<32>(0x7D)};
|
||||
}
|
||||
CheckpointManifest next(std::uint64_t end_tick = 16) {
|
||||
auto n = successor_manifest(genesis(), filled<32>(0xD0), filled<16>(0x02), lease(), end_tick,
|
||||
build, 200, filled<32>(0xA1));
|
||||
build, 200, filled<32>(0xA1), batch_for(end_tick, 1));
|
||||
CHECK(n);
|
||||
return *n;
|
||||
}
|
||||
@@ -52,7 +57,7 @@ void layout_and_round_trip() {
|
||||
CHECK(bytes && bytes->size() == checkpoint_manifest_bytes);
|
||||
// Golden offsets from schemas/region-commit.md.
|
||||
CHECK(std::string_view(reinterpret_cast<const char*>(bytes->data()), 8) == "AEVORCM1");
|
||||
CHECK(at(*bytes, 8, 4) == 1 && at(*bytes, 12, 4) == 1);
|
||||
CHECK(at(*bytes, 8, 4) == 2 && at(*bytes, 12, 4) == 1);
|
||||
CHECK(at(*bytes, 16, 8) == 7 && at(*bytes, 24, 8) == 3 && at(*bytes, 32, 8) == 9);
|
||||
CHECK((*bytes)[40] == 0x02 && at(*bytes, 56, 8) == 1 && (*bytes)[64] == 0x01);
|
||||
CHECK((*bytes)[80] == 0xD0 && at(*bytes, 112, 8) == 0 && at(*bytes, 120, 8) == 16);
|
||||
@@ -60,6 +65,9 @@ void layout_and_round_trip() {
|
||||
CHECK(at(*bytes, 160, 4) == 2 && at(*bytes, 164, 4) == 2 && at(*bytes, 192, 4) == 1);
|
||||
CHECK((*bytes)[196] == 0xB1 && at(*bytes, 228, 4) == 0 && at(*bytes, 232, 4) == 0);
|
||||
CHECK(at(*bytes, 236, 4) == 1 && at(*bytes, 240, 8) == 200 && (*bytes)[248] == 0xA1);
|
||||
CHECK(at(*bytes, 8, 4) == 2 && at(*bytes, 280, 8) == trace_bytes_for(16));
|
||||
CHECK((*bytes)[288] == 0x7C && at(*bytes, 320, 4) == 1 && at(*bytes, 324, 4) == 0);
|
||||
CHECK((*bytes)[328] == 0x7D);
|
||||
auto decoded = decode_manifest(*bytes);
|
||||
CHECK(decoded && *decoded == m && supported(*decoded));
|
||||
auto g = encode_manifest(genesis());
|
||||
@@ -68,7 +76,20 @@ void layout_and_round_trip() {
|
||||
|
||||
void malformed_and_unsupported() {
|
||||
auto bytes = *encode_manifest(next());
|
||||
CHECK(decode_manifest(std::span{bytes}.first(279)).error() == CheckpointError::format);
|
||||
CHECK(decode_manifest(std::span{bytes}.first(359)).error() == CheckpointError::format);
|
||||
// Version 1 (M7.4 commits) still decodes with no batch evidence; v2 is never
|
||||
// written as v1 and v1 bytes cannot carry batch fields.
|
||||
std::vector<std::uint8_t> v1(bytes.begin(), bytes.begin() + 280);
|
||||
v1[8] = 1;
|
||||
auto old = decode_manifest(v1);
|
||||
CHECK(old && old->manifest_version == 1 && old->batch == BatchDigests{});
|
||||
CHECK(!encode_manifest(*old));
|
||||
auto stretched = bytes;
|
||||
stretched[8] = 1;
|
||||
CHECK(decode_manifest(stretched).error() == CheckpointError::format);
|
||||
auto reserved = bytes;
|
||||
reserved[324] = 1;
|
||||
CHECK(decode_manifest(reserved).error() == CheckpointError::format);
|
||||
auto grown = bytes;
|
||||
grown.push_back(0);
|
||||
CHECK(decode_manifest(grown).error() == CheckpointError::format);
|
||||
@@ -76,7 +97,7 @@ void malformed_and_unsupported() {
|
||||
magic[0] ^= 1U;
|
||||
CHECK(decode_manifest(magic).error() == CheckpointError::format);
|
||||
auto version = bytes;
|
||||
version[8] = 2;
|
||||
version[8] = 3;
|
||||
CHECK(decode_manifest(version).error() == CheckpointError::unsupported);
|
||||
// Structurally invalid: zero world, 65-tick batch, oversized payload, zero digest.
|
||||
for (const auto& [offset, value] :
|
||||
@@ -113,24 +134,38 @@ void successor_chain() {
|
||||
const auto g = genesis();
|
||||
const auto d = filled<32>(0xD0);
|
||||
const auto payload = filled<32>(0xA1);
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 0, build, 1, payload).error() ==
|
||||
CheckpointError::chain);
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 65, build, 1, payload).error() ==
|
||||
CheckpointError::chain);
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 0, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::chain);
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 65, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::chain);
|
||||
auto foreign = lease();
|
||||
foreign.frame = 8;
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), foreign, 1, build, 1, payload).error() ==
|
||||
CheckpointError::chain);
|
||||
CHECK(successor_manifest(g, d, g.commit, lease(), 1, build, 1, payload).error() ==
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), foreign, 1, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::chain);
|
||||
CHECK(successor_manifest(g, d, g.commit, lease(), 1, build, 1, payload, batch_for(1)).error() ==
|
||||
CheckpointError::format); // Reused commit identity.
|
||||
auto old = g;
|
||||
old.versions.rules = 1;
|
||||
CHECK(successor_manifest(old, d, filled<16>(2), lease(), 1, build, 1, payload).error() ==
|
||||
CheckpointError::unsupported);
|
||||
CHECK(successor_manifest(old, d, filled<16>(2), lease(), 1, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::unsupported);
|
||||
auto last = next();
|
||||
last.sequence = ~std::uint64_t{0};
|
||||
CHECK(successor_manifest(last, d, filled<16>(3), lease(), 17, build, 1, payload).error() ==
|
||||
CheckpointError::chain);
|
||||
CHECK(successor_manifest(last, d, filled<16>(3), lease(), 17, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::chain);
|
||||
// Batch evidence must match the tick range: one trace step per tick.
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 2, build, 1, payload, batch_for(1))
|
||||
.error() == CheckpointError::format);
|
||||
auto untraced = batch_for(1);
|
||||
untraced.trace_sha256 = {};
|
||||
CHECK(
|
||||
successor_manifest(g, d, filled<16>(2), lease(), 1, build, 1, payload, untraced).error() ==
|
||||
CheckpointError::format);
|
||||
auto unsigned_decisions = batch_for(1, 1);
|
||||
unsigned_decisions.decisions_sha256 = {};
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 1, build, 1, payload, unsigned_decisions)
|
||||
.error() == CheckpointError::format);
|
||||
CHECK(successor_manifest(g, d, filled<16>(2), lease(), 1, build, 1, payload, batch_for(1, 9))
|
||||
.error() == CheckpointError::format);
|
||||
const auto n = next(64);
|
||||
CHECK(n.sequence == 1 && n.predecessor_commit == g.commit && n.predecessor_digest == d);
|
||||
CHECK(n.begin_tick == 0 && n.holder == lease().holder && n.epoch == 4 && n.frame == 9);
|
||||
@@ -151,7 +186,7 @@ void payload_validation_and_restored_trajectory() {
|
||||
auto lease_at_frame = lease();
|
||||
lease_at_frame.frame = evo::test::region_identity.frame;
|
||||
auto m = successor_manifest(*g, filled<32>(0xD0), filled<16>(2), lease_at_frame, 16, build,
|
||||
payload->size(), filled<32>(0xA1));
|
||||
payload->size(), filled<32>(0xA1), batch_for(16));
|
||||
CHECK(m);
|
||||
auto restored = restore_environmental_payload(*payload, *m, 8000000);
|
||||
CHECK(restored);
|
||||
@@ -170,7 +205,12 @@ void payload_validation_and_restored_trajectory() {
|
||||
auto r = restore_environmental_payload(bytes, manifest, 8000000);
|
||||
return !r && r.error() == error;
|
||||
};
|
||||
CHECK(refused([](auto&, auto& x) { x.end_tick = 15; }, CheckpointError::payload));
|
||||
CHECK(refused(
|
||||
[](auto&, auto& x) {
|
||||
x.end_tick = 15;
|
||||
x.batch = batch_for(15);
|
||||
},
|
||||
CheckpointError::payload));
|
||||
CHECK(refused([](auto&, auto& x) { x.key.world = 8; }, CheckpointError::payload));
|
||||
CHECK(refused([](auto&, auto& x) { x.frame = 10; }, CheckpointError::payload));
|
||||
CHECK(refused([](auto&, auto& x) { x.tick_microseconds = 25000; }, CheckpointError::payload));
|
||||
@@ -227,13 +267,14 @@ void lineage_profile() {
|
||||
auto lease_at_frame = lease();
|
||||
lease_at_frame.frame = evo::test::region_identity.frame;
|
||||
auto m = successor_manifest(*g, filled<32>(0xD0), filled<16>(2), lease_at_frame, 24, build,
|
||||
payload->size(), filled<32>(0xA1));
|
||||
payload->size(), filled<32>(0xA1), batch_for(24));
|
||||
CHECK(m && m->profile == CheckpointProfile::environmental_lineage_v1 &&
|
||||
m->history_profile == 1);
|
||||
auto later = restore_lineage_payload(*payload, *m, 8000000);
|
||||
CHECK(later && !(*later)->archive().organism(11)->alive);
|
||||
auto wrong_tick = *m;
|
||||
wrong_tick.end_tick = 23;
|
||||
wrong_tick.batch = batch_for(23);
|
||||
CHECK(restore_lineage_payload(*payload, wrong_tick, 8000000).error() ==
|
||||
CheckpointError::payload);
|
||||
auto nested = *payload;
|
||||
|
||||
@@ -103,11 +103,13 @@ CheckpointManifest link(const CommittedCheckpoint& from, const LeaseGrant& lease
|
||||
m->holder = lease.holder;
|
||||
m->epoch = lease.epoch;
|
||||
m->payload_sha256 = digest(0xA1);
|
||||
// Synthetic trace evidence of the right size; no decisions.
|
||||
m->batch = {trace_bytes_for(end_tick - from.head.end_tick), digest(0x7C), 0, {}};
|
||||
return *m;
|
||||
}
|
||||
std::expected<CommittedCheckpoint, LeaseStoreFailure> publish(Postgres& db,
|
||||
std::vector<std::uint8_t> bytes) {
|
||||
auto head = publish_region_checkpoint(db, bytes, 5s);
|
||||
auto head = publish_region_checkpoint(db, bytes, {}, 5s);
|
||||
if (!head)
|
||||
return std::unexpected{head.error()};
|
||||
return CommittedCheckpoint{*head, std::move(bytes)};
|
||||
@@ -215,8 +217,9 @@ void acquire_renew_release_and_publish() {
|
||||
CHECK(head && head->head.sequence == 1 && head->head.end_tick == 64 &&
|
||||
head->head.commit == id(21));
|
||||
CHECK(failed(publish(*db, link(*head, forged, 22, 65)), LeaseStoreError::fenced));
|
||||
// Exact retry after an applied publication: mismatch, then the head proves it.
|
||||
CHECK(failed(publish(*db, first), LeaseStoreError::head_mismatch));
|
||||
// Exact retry after an applied publication returns the original receipt (M7.5).
|
||||
auto retried = publish(*db, first);
|
||||
CHECK(retried && retried->head == head->head);
|
||||
auto current = committed(*db, key);
|
||||
CHECK(current.head == head->head && current.manifest == head->manifest);
|
||||
auto changed = link(*head, *lease, 22, 65);
|
||||
|
||||
@@ -43,7 +43,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
|
||||
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
|
||||
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4
|
||||
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5
|
||||
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
|
||||
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
|
||||
v2_sha="${v2_sha%% *}"
|
||||
@@ -64,7 +64,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
|
||||
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
|
||||
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
|
||||
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 4:1,2,3,4
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 5:1,2,3,4,5
|
||||
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
|
||||
test "$(PGDATABASE=v2_heads psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2
|
||||
# Inherited privileges bypass fencing, so every migrate run audits effective rights.
|
||||
@@ -99,9 +99,11 @@ expect_audit_failure 'GRANT EXECUTE ON FUNCTION evolution.provision_region_check
|
||||
expect_audit_failure 'GRANT SELECT ON evolution.region_commits TO evo_runtime_test' 'REVOKE SELECT ON evolution.region_commits FROM evo_runtime_test'
|
||||
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
|
||||
# Unknown future versions and checksum drift must not be repaired silently.
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=5 WHERE version=4'
|
||||
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=6 WHERE version=5'
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=5 WHERE version=6"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=5"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=4 WHERE version=5"
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=4"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=3"
|
||||
@@ -110,4 +112,4 @@ psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=rep
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64)"
|
||||
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
|
||||
echo 'postgres migrations, rollback, constraints, permissions, fencing and checkpoints passed'
|
||||
echo 'postgres migrations, rollback, constraints, permissions, fencing, checkpoints and outcomes passed'
|
||||
|
||||
@@ -20,8 +20,10 @@ digest_3="$(sha256sum "$root/migrations/0003_region_checkpoints.sql")"
|
||||
digest_3="${digest_3%% *}"
|
||||
digest_4="$(sha256sum "$root/migrations/0004_lineage_profile.sql")"
|
||||
digest_4="${digest_4%% *}"
|
||||
digest_5="$(sha256sum "$root/migrations/0005_batch_decisions.sql")"
|
||||
digest_5="${digest_5%% *}"
|
||||
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
|
||||
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
|
||||
--set="migration_sha_4=$digest_4" \
|
||||
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
|
||||
--set="runtime_role=$runtime_role" \
|
||||
--file="$root/migrations/apply.sql"
|
||||
|
||||
Reference in New Issue
Block a user