M12.6: capture consistent regional census snapshots
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m20s
Foundation / checks (network) (push) Successful in 21m3s
Foundation / checks (client) (push) Successful in 22m14s
Foundation / checks (tsan) (push) Successful in 24m52s
Foundation / checks (postgres) (push) Failing after 36m19s
Foundation / checks (checks) (push) Successful in 1h10m25s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m20s
Foundation / checks (network) (push) Successful in 21m3s
Foundation / checks (client) (push) Successful in 22m14s
Foundation / checks (tsan) (push) Successful in 24m52s
Foundation / checks (postgres) (push) Failing after 36m19s
Foundation / checks (checks) (push) Successful in 1h10m25s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
This commit is contained in:
@@ -258,6 +258,7 @@ if(EVO_BUILD_POSTGRES)
|
||||
src/persistence/checkpoint_store.cpp src/persistence/primitive_checkpoint_store.cpp
|
||||
src/persistence/cellular_checkpoint_store.cpp src/persistence/population_checkpoint_store.cpp
|
||||
src/persistence/cellular_history_store.cpp
|
||||
src/persistence/cellular_census_snapshot.cpp
|
||||
src/persistence/region_clone.cpp src/persistence/cellular_migration_store.cpp)
|
||||
add_library(evo::region_store ALIAS evo_region_store)
|
||||
target_link_libraries(evo_region_store PUBLIC evo::postgres evo::region_checkpoint
|
||||
|
||||
@@ -136,7 +136,11 @@ current cellular owners. Retained departed history is excluded from owned stock.
|
||||
derives bounded, fully replayed census pages from the existing atomic biological
|
||||
journal. [Scoped multi-region queries](schemas/ecology/scoped-cellular-census-set-v1.md)
|
||||
require exact declared coverage/boundaries, consistent immutable genomes and
|
||||
disjoint resident identities before summing. Complete world/branch reconciliation,
|
||||
disjoint resident identities before summing.
|
||||
[Consistent scoped snapshots](schemas/ecology/consistent-cellular-census-snapshot-v1.md)
|
||||
capture declared current heads in one short database snapshot, then fully replay
|
||||
the immutable cuts; concurrent atomic migration cannot split a selected pair.
|
||||
Complete world/branch reconciliation,
|
||||
client queries and terminal extinction remain later gates.
|
||||
`apps/evo-sim` owns CLI/file I/O
|
||||
and reporting. `tests` holds contract,
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# Project status and next steps
|
||||
|
||||
## M12.6d consistent scoped census snapshots - 2026-10-18
|
||||
|
||||
Scoped census reads now select all declared regional heads from one short
|
||||
read-only database snapshot, then fully replay the exact immutable cuts after
|
||||
the metadata transaction ends. A concurrent atomic migration cannot split the
|
||||
selected source/destination heads. Missing or failed evidence returns no partial
|
||||
census; unsafe capture cleanup closes its connection.
|
||||
|
||||
Private raw/cohort migration races and TLS BEGIN/COMMIT request/reply-loss tests
|
||||
pass in Release and ASan/UBSan, alongside existing history and migration/storage
|
||||
faults. Native default Release passes **79/79**; Clang analysis and formatting
|
||||
pass. [Evidence and limits](engineering/VALIDATION.md) retain durable world/branch
|
||||
coverage, outside transfer/viability reconciliation, client access, target-host
|
||||
query costs and later ecology/soak/product gates. A caller-declared consistent
|
||||
cut cannot accept terminal extinction. No deployment or World0 action.
|
||||
|
||||
## M12.6c scoped multi-region census queries - 2026-10-18
|
||||
|
||||
The portable census now queries 1..64 explicitly declared regions at an exact
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
# ADR-0153: Consistent database cuts for scoped cellular census queries
|
||||
|
||||
## Status
|
||||
|
||||
Accepted for M12.6d, 2026-10-18. Backend read adapter under ADR-0151/0152;
|
||||
no durable world catalog, external branch identity or extinction acceptance.
|
||||
|
||||
## Context and decision
|
||||
|
||||
Equal-tick regional observations can straddle an atomic migration. A source-after
|
||||
and destination-before selection can omit stock without duplicating an organism.
|
||||
The scoped portable query cannot detect that omission from detached totals alone.
|
||||
|
||||
Capture the current committed heads of 1..64 declared regions in one explicitly
|
||||
owned PostgreSQL `REPEATABLE READ READ ONLY` transaction on an idle connection.
|
||||
Requests are nonzero, distinct, ascending region IDs in one world, with object
|
||||
stores and grants for the existing replay readers. Keep exact head/manifest pairs,
|
||||
then commit the short metadata transaction before reading biological objects.
|
||||
An atomic migration committed between head reads is either visible at both
|
||||
endpoints or neither when both are in the declared set.
|
||||
|
||||
After capture, use ADR-0151's existing exact-cut history reader for each selected
|
||||
head. Verify the returned head and manifest still match the captured cut. Complete
|
||||
semantic replay, object hashes and fenced evidence checks remain mandatory; no
|
||||
cached census or older-head fallback. Combine observations through ADR-0152's
|
||||
coverage/boundary/genome/resident checks. Different head ticks fail rather than
|
||||
advancing biology or silently selecting an older common tick. Return exact cuts
|
||||
alongside the scoped census only after every step succeeds.
|
||||
|
||||
The adapter refuses all caller-owned transactions, including read-only ones.
|
||||
It rolls back failed metadata capture; failed completion/cleanup closes the
|
||||
connection. An exception during capture also closes the connection immediately,
|
||||
releasing server resources when disconnect is observed. Add an explicit noexcept
|
||||
disconnect method to the libpq owner for this cleanup. No automatic retry.
|
||||
Per-I/O deadlines retain existing semantics, not a whole-query latency guarantee.
|
||||
|
||||
## Alternatives, benefits, costs and risks
|
||||
|
||||
Independent head reads can mix the sides of a paired publication. Holding one
|
||||
transaction through full recovery is unnecessarily long, and existing fenced
|
||||
evidence functions take lease row locks that a read-only transaction forbids.
|
||||
Short read-only metadata capture plus immutable replay avoids that coupling.
|
||||
Ordinary replay still briefly takes existing per-call lease locks; this does not
|
||||
eliminate all contention or establish a deployment capacity claim.
|
||||
|
||||
Capture uses indexed regional head lookups. Work and retained observations remain
|
||||
bounded by 64 regions; full replay/object cost is inherited from ADR-0151. A long
|
||||
metadata snapshot can retain old database versions until completion. No new SQL
|
||||
objects, cache/index, durable writes, biological rule/codec or dependency.
|
||||
|
||||
## Consequences and qualification
|
||||
|
||||
Require an actual paired migration on an independent connection between the first
|
||||
and second head reads: both captured cuts must remain before the pair while a
|
||||
later snapshot sees both after it. Recover the earlier cuts after publication,
|
||||
compare exact census stock, and repeat for raw/cohort histories. Missing regions,
|
||||
stale/missing/corrupt replay evidence, mismatched boundaries, invalid/over-bound
|
||||
requests and caller-owned transactions must return no partial census. Confirm
|
||||
transaction cleanup/connection closure, unchanged heads and no object writes.
|
||||
|
||||
This is one connected database's consistent committed cut for a caller-declared
|
||||
region set. It neither proves that the list includes every world owner nor
|
||||
reconciles branches, outside transfers, dormant/reserve viability or global
|
||||
freshness. Current grants support recovery, not new observation authority. No
|
||||
terminal-extinction outcome or public client permission follows. Durable
|
||||
world/branch coverage and complete M12.7 reconciliation remain open.
|
||||
@@ -1,5 +1,7 @@
|
||||
# Architecture decision records
|
||||
|
||||
- [ADR-0153: Consistent scoped cellular census snapshots](0153-consistent-cellular-census-snapshots.md) - short read-only repeatable-read head capture followed by exact immutable replay; atomic pairs cannot straddle a declared-region cut, without claiming world/branch completeness.
|
||||
|
||||
- [ADR-0152: Scoped multi-region cellular census queries](0152-scoped-cellular-census-sets.md) - bounded declared coverage, exact boundaries, complete shared-genome equality and disjoint resident identities; no world-completeness or extinction authority.
|
||||
|
||||
- [ADR-0151: Committed cellular census history](0151-committed-cellular-census-history.md) - derives bounded, exact-cut census pages from the existing atomic immutable biological journal, with full historical replay; no duplicate census writes or world-extinction authority.
|
||||
|
||||
@@ -1,5 +1,14 @@
|
||||
# Persistence and region authority
|
||||
|
||||
M12.6d captures declared regional heads in one short read-only repeatable-read
|
||||
transaction, closes it, then fully replays each exact immutable cut before
|
||||
constructing a scoped census. Atomic migration commits cannot split the selected
|
||||
head set when both endpoints are declared. Failed metadata capture rolls back;
|
||||
failed cleanup or an exception disconnects the session. Full replay remains
|
||||
outside the metadata transaction, with existing per-call evidence locks.
|
||||
See [ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md). The caller's
|
||||
list is still not complete world coverage or external branch identity.
|
||||
|
||||
M12.6c's portable [scoped census set](../../schemas/ecology/scoped-cellular-census-set-v1.md)
|
||||
combines a bounded declared set of regional observations, including replayed
|
||||
history entries. It verifies exact common boundaries, complete shared genome
|
||||
|
||||
@@ -22,6 +22,13 @@ It refuses missing/stale/conflicting evidence. The list remains caller-declared;
|
||||
neither a durable world catalog nor reconciled migration cuts or branch proof
|
||||
is supplied, so this query cannot accept world absence or terminal extinction.
|
||||
|
||||
M12.6d's [consistent scoped snapshots](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md)
|
||||
select current regional heads from one database snapshot, then verify each exact
|
||||
immutable cut through full recovery. A pair cannot commit between selected sides
|
||||
when both regions are declared. Durable required-world coverage, outside transfer
|
||||
and reserve reconciliation, external branch identity and extinction acceptance
|
||||
remain separate; this is no complete world evidence envelope.
|
||||
|
||||
## Identity and classification
|
||||
|
||||
An organism has a permanent identity and terminal death. An immutable genome has
|
||||
|
||||
@@ -13,7 +13,7 @@ Commit and push each validated scoped segment separately.
|
||||
| 12.3 | Exact population profile/trace/store and opt-in host integration implemented (12.3b) | ADR-0142, schema 44; exact publication/recovery, mutation race guards, ambiguous replies and process kills. Executable/network population fault drills and target-host qualification remain open. |
|
||||
| 12.4 | Owned intervals and durable paired migration/fault qualification implemented in scoped segments through 12.4l | ADR-0143–0149/schema 49; bounded admitted work, source freeze, staged ghost, atomic pair and exact recovery, local TLS and storage faults. Multi-host/orchestrator, linked ecology and integrated stock audits remain open. |
|
||||
| 12.5 | Blocked on 12.2-12.4 | Finite M6 process fixtures do not qualify coupled persistent cellular/cohort atmosphere/water budgets. |
|
||||
| 12.6 | Regional census/lineage queries, replayed commit-bound history pages and scoped multi-region queries implemented (12.6a-c) | ADR-0150–0152; atomic biological commits are the durable events. Fixed regional cuts and full replay bind observations; declared sets reject missing/stale/conflicting identity or duplicate residents. Durable world/branch coverage, reconciled migration cuts, target-host query/index costs and client path remain required. |
|
||||
| 12.6 | Regional queries, replayed history, scoped multi-region queries and consistent database head snapshots implemented (12.6a-d) | ADR-0150–0153; short read-only snapshot capture prevents atomic pairs from straddling declared head sets, with exact immutable replay. Durable world/branch coverage, outside transfer/viability reconciliation, target-host query/index costs and client path remain required. |
|
||||
| 12.7 | Evidence policy defined; reconciler open | Complete durable world watermark and every viable/in-flight store remain required. |
|
||||
| 12.8 | Not qualified | Actual multi-day integrated ecology/restart/clone evidence remains required. |
|
||||
| 12.9 | Not accepted | Returning-player product evidence and expansion review remain required; World0 stays separately blocked. |
|
||||
|
||||
@@ -27,7 +27,7 @@ or authorize production enablement.
|
||||
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
|
||||
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
|
||||
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
|
||||
| 12.6 | Retained archive, regional census/lineage queries, replayed commit-bound history and bounded scoped multi-region queries (12.6a-c) | Durable world/branch history context and required-region coverage, reconciled migration cuts, target-host query/index qualification and client permission/presentation; no account power or invented precise statistical ancestry. |
|
||||
| 12.6 | Retained archive, regional census/lineage queries, replayed history, scoped multi-region queries and consistent database head snapshots (12.6a-d) | Durable world/branch history context and required-region coverage, outside transfer/viability reconciliation, target-host query/index qualification and client permission/presentation; no account power or invented precise statistical ancestry. |
|
||||
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
|
||||
| 12.8 | Deterministic seed/regime drivers and separate-process exact continuation | Reproducible integrated worlds running for actual days, real restarts/clone comparisons and retained accounting/diversity/cost audits. Accelerated tests cannot supply elapsed days. |
|
||||
| 12.9 | M11 feedback and current UI review retained | Returning-player samples on the integrated ecology and explicit product/3D readiness decision after soak evidence. No M12 product acceptance or World0 promotion is inferred. |
|
||||
@@ -114,6 +114,13 @@ before querying. Nonresident historical records add no living stock. A required
|
||||
list supplied by a caller is still not durable world coverage; equal ticks alone
|
||||
cannot prove a consistent migration cut or branch. This does not close M12.7.
|
||||
|
||||
M12.6d's [consistent scoped snapshot](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md)
|
||||
now captures all declared heads in one database read snapshot before exact replay.
|
||||
An atomic pair cannot straddle that selected set when both endpoints are declared.
|
||||
The short metadata transaction ends before object I/O and fenced recovery; failed
|
||||
evidence still returns no partial census. Durable world/branch identity, required
|
||||
owner coverage, outside transfers and viable-store reconciliation remain open.
|
||||
|
||||
Current expansion readiness: **no-go for M13 integration or World0 promotion**
|
||||
while these M12 integration/evidence gates remain open. This records the present
|
||||
engineering evidence, not a substitute for the later maintainer product review.
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
# Validation strategy
|
||||
|
||||
M12.6d's private PostgreSQL snapshot tests publish a real atomic migration from
|
||||
an independent connection between the first and second captured head reads.
|
||||
Both selected cuts must remain before the pair; the next snapshot sees both
|
||||
after it. Recover the earlier cuts after publication and compare full regional
|
||||
rows plus exact/descendant count/body/reserve, in both directions and raw/cohort
|
||||
profiles. Verify actual repeatable-read/read-only settings, no metadata transaction
|
||||
held over replay, exact cut binding and unchanged biological stock. Invalid,
|
||||
missing, unretained, stale, unreadable or caller-transaction evidence returns no
|
||||
partial result. Capture errors roll back; lost completion/throwing hooks close
|
||||
the session. Sanitize/inspect the new adapter and existing libpq owner together.
|
||||
This does not qualify complete world/branch coverage or terminal extinction.
|
||||
|
||||
M12.6c's scoped census-set contract compares declared-region queries with direct
|
||||
resident count/body/reserve oracles. Require missing/extra/duplicate regions,
|
||||
every boundary field, unknown/invalid scope, conflicting immutable successor IDs,
|
||||
|
||||
@@ -1,5 +1,73 @@
|
||||
# Validation report
|
||||
|
||||
## 2026-10-18 - M12.6d consistent scoped census snapshots
|
||||
|
||||
[ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md) captures the current
|
||||
heads of 1..64 declared regions in one owned read-only repeatable-read database
|
||||
transaction. It completes that short metadata transaction before fully replaying
|
||||
each immutable selected cut through the existing history reader. Exact cut binding
|
||||
and the scoped census-set checks remain mandatory. Failed capture rolls back;
|
||||
failed completion/cleanup or a capture exception disconnects the session. No
|
||||
partial result or older-head fallback. See
|
||||
[the contract](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md).
|
||||
|
||||
Executed locally:
|
||||
|
||||
- Native MSVC **19.44.35217.0** default Release build/CTest **79/79 passed**,
|
||||
**13.76 s**, with optional Linux persistence disabled.
|
||||
- GCC 14 Release private population/history integration passed **17 s** (15 s
|
||||
population phase). Final combined snapshot/history/migration **TLS and storage
|
||||
fault qualification passed, exit 0, 39 s** (38 s population phase).
|
||||
- Clang 19 Linux **ASan/UBSan** private population integration passed **46 s**;
|
||||
the final combined TLS/storage/snapshot matrix passed **exit 0, 70 s** (66 s
|
||||
population phase), without sanitizer reports.
|
||||
- Explicit clang-tidy 19 covered the libpq owner, snapshot adapter and integration
|
||||
tests, then rechecked the final TLS assertions; no owned warnings. Repository
|
||||
clang-format 19 and `git diff --check` passed.
|
||||
|
||||
Each raw/cohort migration race uses an independent publishing connection after
|
||||
the first captured head read. Actual database settings are checked as repeatable
|
||||
read and read-only. Both captured heads remain before the atomic pair; a later
|
||||
snapshot sees both new heads. Earlier cuts fully replay after publication, retain
|
||||
identical complete regional rows, and preserve exact/descendant resident counts,
|
||||
body and reserve in both migration directions. The metadata transaction is closed
|
||||
before the existing idle-only historical reader runs.
|
||||
|
||||
Tests also retain exact head/manifest binding on repeated reads, direct biology
|
||||
oracles, request bounds/order/null validation, missing regions, incompatible
|
||||
current ticks/owner versions, stale grants, unretained builds, missing/unreadable
|
||||
payloads and unchanged current heads. Caller-owned read-only transactions and a
|
||||
writer's pending publication are refused without taking over its transaction.
|
||||
Capture failure rolls back; throwing hooks and failed completion close the
|
||||
connection. Disconnect is idempotent and later queries return `closed`.
|
||||
|
||||
Actual loopback TCP with end-to-end TLS independently drops requests and replies
|
||||
at both BEGIN and COMMIT. Every uncertain outcome returns no census, closes the
|
||||
query session and preserves observed heads. Existing migration owner expiry,
|
||||
replacement/heal, exact recovery and local object-failure checks still pass.
|
||||
All database runners used fresh private temporary clusters and stopped them on
|
||||
exit; no live database, deployment or World0 action.
|
||||
|
||||
The test allowlist initially needed an explicit nested digest-array type; analysis
|
||||
also required consuming the expected result in the throwing-hook test. Both were
|
||||
corrected and the final matrix rebuilt. No checks/goldens were suppressed or
|
||||
rewritten. No SQL schema, biological rule/codec, dependency or ownership change.
|
||||
|
||||
Limits: this is a consistent cut of a caller-declared set in one connected
|
||||
database, not durable world coverage or external branch identity. Outside-region
|
||||
transfers, reserve viability and complete global freshness/reconciliation remain
|
||||
open. SQL deadlines do not preempt local object reads or promise whole-query
|
||||
latency. Existing replay evidence calls retain their per-call lease locks. No
|
||||
deployment-host capacity claim, new TSan, complete SQL/backup/DR rerun, remote CI,
|
||||
multi-host drill or actual multi-day soak. Prior full persistence regression is
|
||||
retained in M12.6b. No terminal-extinction acceptance or client endpoint.
|
||||
|
||||
Logs: `build/m126d-native.log`, `build/m126d-postgres.log`,
|
||||
`build/m126d-asan.log`, `build/m126d-final-postgres-build.log`,
|
||||
`build/m126d-final-postgres.log`, `build/m126d-final-asan-build.log`,
|
||||
`build/m126d-final-asan.log`, `build/m126d-final-tidy.log`,
|
||||
`build/m126d-final-integration-tidy.log`, `build/m126d-final-format.log`.
|
||||
|
||||
## 2026-10-18 - M12.6c scoped multi-region census queries
|
||||
|
||||
[ADR-0152](../adr/0152-scoped-cellular-census-sets.md) adds a portable detached
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
#pragma once
|
||||
|
||||
#include "evo/persistence/cellular_history_store.hpp"
|
||||
#include "evo/server/lifecycle/cellular_census_set.hpp"
|
||||
|
||||
#include <chrono>
|
||||
#include <cstddef>
|
||||
#include <cstdint>
|
||||
#include <expected>
|
||||
#include <functional>
|
||||
#include <span>
|
||||
#include <vector>
|
||||
|
||||
namespace evo::persistence {
|
||||
struct CellularCensusRegionRequest {
|
||||
std::uint64_t region{};
|
||||
LeaseGrant lease{};
|
||||
const FilesystemObjectStore* objects{}; // Non-owning, valid throughout the call.
|
||||
};
|
||||
struct CellularCensusSnapshot {
|
||||
std::vector<CommittedCheckpoint> cuts; // Sorted regions, exact captured head/manifest pairs.
|
||||
server::CellularCensusSet census;
|
||||
};
|
||||
enum class CellularSnapshotError : std::uint8_t { invalid_request, transaction, history, census };
|
||||
struct CellularSnapshotFailure {
|
||||
CellularSnapshotError kind{};
|
||||
std::uint64_t region{}; // Failed region, or zero for a request/transaction/set error.
|
||||
PgFailure transaction{}; // Meaningful for transaction.
|
||||
CellularHistoryFailure history{}; // Meaningful for history.
|
||||
server::CellularCensusSetError census{}; // Meaningful for census.
|
||||
};
|
||||
// Test/fault seam after each head has been captured (zero-based request index).
|
||||
// Production callers pass nothing. A throwing hook closes the capture connection.
|
||||
struct CellularCensusSnapshotHooks {
|
||||
std::function<void(std::size_t)> after_head;
|
||||
};
|
||||
// Idle connected session only. Requests: 1..64, nonzero strictly ascending regions
|
||||
// in one world; all stores nonnull. Capture current heads in a short READ ONLY
|
||||
// REPEATABLE READ transaction, finish it, then fully replay each immutable cut.
|
||||
// All cuts must share frame/tick/owner/rules/quantum. No partial census on error.
|
||||
// Per-I/O timeout 1..30000 ms; no automatic retry or whole-query latency promise.
|
||||
// Metadata failures roll back; failed cleanup or an exception closes the session.
|
||||
// Caller-declared coverage, NOT world/branch completeness or live authority.
|
||||
[[nodiscard]] std::expected<CellularCensusSnapshot, CellularSnapshotFailure>
|
||||
load_cellular_census_snapshot(Postgres&, std::uint64_t world,
|
||||
std::span<const CellularCensusRegionRequest>,
|
||||
std::span<const Sha256Digest> retained_builds,
|
||||
std::chrono::milliseconds timeout,
|
||||
const CellularCensusSnapshotHooks& hooks = {});
|
||||
} // namespace evo::persistence
|
||||
@@ -40,6 +40,9 @@ class Postgres {
|
||||
// True only for a connected, idle session outside an explicit transaction.
|
||||
// Read-only status inspection; no query, lock or transaction is started.
|
||||
[[nodiscard]] bool transaction_idle() const noexcept;
|
||||
// Drop the owned connection and any server-side transaction. Idempotent;
|
||||
// future queries return closed. No SQL, allocation or automatic reconnect.
|
||||
void disconnect() noexcept;
|
||||
[[nodiscard]] std::expected<PgRows, PgFailure> query(const std::string& sql,
|
||||
std::span<const std::string> parameters,
|
||||
std::chrono::milliseconds timeout);
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
# Consistent scoped cellular census snapshot v1
|
||||
|
||||
Semantic backend read contract under
|
||||
[ADR-0153](../../docs/adr/0153-consistent-cellular-census-snapshots.md), not a
|
||||
persistent/wire codec. Linux API: `persistence::load_cellular_census_snapshot`.
|
||||
|
||||
Input names one nonzero world and 1..64 requests with strictly ascending nonzero
|
||||
region IDs. Each request retains a non-owning object-store pointer and a grant
|
||||
for the existing historical recovery checks. Stores must remain alive throughout
|
||||
the call. The connected PostgreSQL session must be idle; all caller transactions,
|
||||
including read-only transactions and pending publications, are refused unchanged.
|
||||
The timeout for each PostgreSQL query is 1..30000 ms; local object reads retain
|
||||
their existing blocking semantics. No automatic retry or whole-query SLO.
|
||||
|
||||
1. Begin an owned `REPEATABLE READ READ ONLY` transaction and read every requested
|
||||
current committed head plus exact manifest. The first read establishes the
|
||||
snapshot; later head reads use the same database view. Retain request order.
|
||||
2. Commit the metadata transaction before any object restoration/replay. Metadata
|
||||
errors roll back. Failure to begin/finish/clean up safely closes a non-idle
|
||||
connection; an exception during capture also disconnects it. Results from a
|
||||
failed capture never escape as a partial census.
|
||||
3. On the now-idle session, recover each exact selected cut through the committed
|
||||
history reader. Validate full manifest binding, retained builds, strong object
|
||||
digests, complete payload/trace/transition/paired replay and existing fenced
|
||||
evidence. The returned head and manifest must equal their captured values.
|
||||
4. Build the scoped census set, requiring all captured states to agree on frame,
|
||||
completed tick, owner format, rules and quantum, with consistent complete
|
||||
shared-genome records and disjoint resident organism identities. Different
|
||||
current ticks fail; no catch-up or earlier common-tick selection is inferred.
|
||||
5. Return exact captured head/manifest pairs and the detached scoped census only
|
||||
after all regions pass. Any later failure discards the entire result. There is
|
||||
no fallback, skipped region or missing-as-zero behavior.
|
||||
|
||||
When both endpoints are declared, one atomic migration committed between head
|
||||
reads appears at neither endpoint of that captured cut; a later capture sees
|
||||
both new heads. More generally each atomic pair is visible at both endpoints or
|
||||
neither in the selected database snapshot. Replay remains pinned to those exact
|
||||
immutable cuts even if the current heads advance before replay begins. Frozen
|
||||
source stock remains owned; staged ghosts and uncommitted uploads are excluded.
|
||||
Existing per-call lease locks still apply during recovery outside the metadata
|
||||
transaction. No long read transaction is held over payload I/O.
|
||||
|
||||
Returned references are historical evidence, not current lease/activation proof.
|
||||
The region list remains caller-declared and need not include outside migration
|
||||
endpoints or all world owners. This supplies neither a durable world catalog nor
|
||||
external branch/rewind identity, reserve viability reconciliation, freshness or
|
||||
complete global watermark. No world absence/terminal-extinction acceptance,
|
||||
account advantage, new client permissions or public endpoint is provided.
|
||||
@@ -0,0 +1,102 @@
|
||||
#include "evo/persistence/cellular_census_snapshot.hpp"
|
||||
|
||||
#include <utility>
|
||||
|
||||
namespace evo::persistence {
|
||||
namespace {
|
||||
using Failure = CellularSnapshotFailure;
|
||||
using Error = CellularSnapshotError;
|
||||
auto transaction_failure(PgFailure error) {
|
||||
return std::unexpected(Failure{Error::transaction, 0, error, {}, {}});
|
||||
}
|
||||
auto history_failure(std::uint64_t region, CellularHistoryFailure error) {
|
||||
return std::unexpected(Failure{Error::history, region, {}, error, {}});
|
||||
}
|
||||
// Exception-safe capture ownership: explicit completion disarms this guard.
|
||||
// Closing cannot allocate/throw or leave a caller with an open failed transaction.
|
||||
struct CaptureGuard {
|
||||
Postgres& db;
|
||||
bool active = true;
|
||||
~CaptureGuard() {
|
||||
if (active)
|
||||
db.disconnect();
|
||||
}
|
||||
};
|
||||
std::expected<std::vector<CommittedCheckpoint>, Failure>
|
||||
capture(Postgres& db, std::uint64_t world, std::span<const CellularCensusRegionRequest> requests,
|
||||
std::chrono::milliseconds timeout, const CellularCensusSnapshotHooks& hooks) {
|
||||
const auto begun = db.query("BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY", {}, timeout);
|
||||
if (!begun) {
|
||||
if (!db.transaction_idle())
|
||||
db.disconnect();
|
||||
return transaction_failure(begun.error());
|
||||
}
|
||||
CaptureGuard guard{db};
|
||||
auto result = [&]() -> std::expected<std::vector<CommittedCheckpoint>, Failure> {
|
||||
std::vector<CommittedCheckpoint> cuts;
|
||||
cuts.reserve(requests.size());
|
||||
for (std::size_t i = 0; i < requests.size(); ++i) {
|
||||
auto head = load_region_checkpoint(db, {world, requests[i].region}, timeout);
|
||||
if (!head)
|
||||
return history_failure(requests[i].region, {CellularHistoryError::recovery,
|
||||
{RecoveryError::store, head.error()}});
|
||||
cuts.push_back(std::move(*head));
|
||||
if (hooks.after_head)
|
||||
hooks.after_head(i);
|
||||
}
|
||||
return cuts;
|
||||
}();
|
||||
const auto done = db.query(result ? "COMMIT" : "ROLLBACK", {}, timeout);
|
||||
if (!done)
|
||||
return transaction_failure(done.error());
|
||||
if (!db.transaction_idle())
|
||||
return transaction_failure({PgError::sql, {}});
|
||||
guard.active = false;
|
||||
return result;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<CellularCensusSnapshot, CellularSnapshotFailure> load_cellular_census_snapshot(
|
||||
Postgres& db, std::uint64_t world, std::span<const CellularCensusRegionRequest> requests,
|
||||
std::span<const Sha256Digest> builds, std::chrono::milliseconds timeout,
|
||||
const CellularCensusSnapshotHooks& hooks) {
|
||||
if (!db.transaction_idle() || world == 0 || requests.empty() ||
|
||||
requests.size() > server::max_cellular_census_regions || timeout.count() <= 0 ||
|
||||
timeout.count() > 30000)
|
||||
return std::unexpected(Failure{Error::invalid_request, 0, {}, {}, {}});
|
||||
std::vector<std::uint64_t> required;
|
||||
required.reserve(requests.size());
|
||||
for (const auto& request : requests) {
|
||||
if (!request.objects || request.region == 0 || request.lease.frame == 0 ||
|
||||
(!required.empty() && request.region <= required.back()))
|
||||
return std::unexpected(Failure{Error::invalid_request, 0, {}, {}, {}});
|
||||
required.push_back(request.region);
|
||||
}
|
||||
auto cuts = capture(db, world, requests, timeout, hooks);
|
||||
if (!cuts)
|
||||
return std::unexpected(cuts.error());
|
||||
std::vector<server::CellularCensus> observations;
|
||||
observations.reserve(requests.size());
|
||||
for (std::size_t i = 0; i < requests.size(); ++i) {
|
||||
const auto& request = requests[i];
|
||||
const auto& cut = (*cuts)[i];
|
||||
auto page = load_cellular_census_history(db, *request.objects, {world, request.region},
|
||||
request.lease, cut.head, cut.head.sequence, 1,
|
||||
builds, timeout);
|
||||
if (!page)
|
||||
return history_failure(request.region, page.error());
|
||||
if (page->entries.size() != 1 || page->entries.front().committed.head != cut.head ||
|
||||
page->entries.front().committed.manifest != cut.manifest)
|
||||
return history_failure(request.region, {CellularHistoryError::cut_mismatch, {}});
|
||||
observations.push_back(std::move(page->entries.front().census));
|
||||
}
|
||||
const auto& first = observations.front().boundary();
|
||||
auto census = server::CellularCensusSet::observe({first.world, first.frame, first.tick,
|
||||
first.owner_format, first.rules,
|
||||
first.quantum_microseconds},
|
||||
required, observations);
|
||||
if (!census)
|
||||
return std::unexpected(Failure{Error::census, 0, {}, {}, census.error()});
|
||||
return CellularCensusSnapshot{std::move(*cuts), std::move(*census)};
|
||||
}
|
||||
} // namespace evo::persistence
|
||||
@@ -119,6 +119,9 @@ struct Postgres::Impl {
|
||||
};
|
||||
Postgres::Postgres(std::unique_ptr<Impl> impl) : impl_(std::move(impl)) {}
|
||||
Postgres::~Postgres() = default;
|
||||
void Postgres::disconnect() noexcept {
|
||||
impl_->connection.reset();
|
||||
}
|
||||
bool Postgres::transaction_idle() const noexcept {
|
||||
return impl_->connection && PQstatus(impl_->connection.get()) == CONNECTION_OK &&
|
||||
PQtransactionStatus(impl_->connection.get()) == PQTRANS_IDLE;
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
#include "evo/persistence/account_control_store.hpp"
|
||||
#include "evo/persistence/cellular_census_snapshot.hpp"
|
||||
#include "evo/persistence/cellular_history_store.hpp"
|
||||
#include "evo/persistence/population_checkpoint_store.hpp"
|
||||
#include "evo/persistence/population_interval_store.hpp"
|
||||
@@ -790,6 +791,158 @@ void history_pages(Fixture& f) {
|
||||
"history transaction requirement changed for read-only callers");
|
||||
std::puts("committed census history pagination, replay, transaction and failure guards passed");
|
||||
}
|
||||
void snapshot_transport_faults(Fixture& f) {
|
||||
if (!std::getenv("EVO_MIGRATION_TCP_PORT"))
|
||||
return;
|
||||
const std::array<Sha256Digest, 1> builds{f.build};
|
||||
const std::array requests{CellularCensusRegionRequest{f.key.region, f.lease, f.objects.get()}};
|
||||
for (unsigned mode = 0; mode < 4; ++mode) {
|
||||
test::PostgresRelay relay(migration_backend_port());
|
||||
auto db = migration_tls_connection(relay);
|
||||
const bool at_completion = mode >= 2;
|
||||
const bool lost_request = mode % 2 == 0;
|
||||
const auto partition = [&] {
|
||||
if (lost_request)
|
||||
relay.drop_requests(true);
|
||||
else
|
||||
relay.drop_responses(true);
|
||||
};
|
||||
if (!at_completion)
|
||||
partition();
|
||||
const CellularCensusSnapshotHooks hook{[&](std::size_t) {
|
||||
if (at_completion)
|
||||
partition();
|
||||
}};
|
||||
const auto lost =
|
||||
load_cellular_census_snapshot(*db, f.key.world, requests, builds, 300ms, hook);
|
||||
require(!lost && lost.error().kind == CellularSnapshotError::transaction &&
|
||||
lost.error().transaction.kind == PgError::unknown && !db->transaction_idle() &&
|
||||
relay.healthy() &&
|
||||
(lost_request ? relay.dropped_requests() : relay.dropped_responses()) > 0 &&
|
||||
take(load_region_checkpoint(*f.db, f.key, 5s), "head after lost capture I/O")
|
||||
.head == f.head.head,
|
||||
"lost snapshot BEGIN/COMMIT request/reply returned evidence or kept a session");
|
||||
}
|
||||
std::puts("snapshot TLS BEGIN/COMMIT request/reply loss fails closed");
|
||||
}
|
||||
void snapshot_guards(Fixture& f) {
|
||||
const std::array<Sha256Digest, 1> builds{f.build};
|
||||
const std::array requests{CellularCensusRegionRequest{f.key.region, f.lease, f.objects.get()}};
|
||||
const auto read = [&] {
|
||||
return load_cellular_census_snapshot(*f.db, f.key.world, requests, builds, 5s);
|
||||
};
|
||||
const auto before = f.head;
|
||||
const auto first = take(read(), "single-region captured snapshot");
|
||||
const auto second = take(read(), "repeat captured snapshot");
|
||||
require(first.cuts.size() == 1 && first.cuts[0].head == before.head &&
|
||||
first.cuts[0].manifest == before.manifest &&
|
||||
second.cuts[0].head == first.cuts[0].head &&
|
||||
std::ranges::equal(
|
||||
first.census.regions()[0].rows(),
|
||||
take(CellularCensus::observe(*f.raw), "snapshot direct oracle").rows()) &&
|
||||
f.db->transaction_idle(),
|
||||
"captured snapshot lost its exact committed cut/biology or retained a transaction");
|
||||
const auto invalid = [&](auto region_requests, std::uint64_t world = 30201,
|
||||
std::chrono::milliseconds timeout = 5s) {
|
||||
const auto result =
|
||||
load_cellular_census_snapshot(*f.db, world, region_requests, builds, timeout);
|
||||
require(!result && result.error().kind == CellularSnapshotError::invalid_request &&
|
||||
f.db->transaction_idle(),
|
||||
"invalid snapshot request started a transaction or returned a census");
|
||||
};
|
||||
invalid(std::vector<CellularCensusRegionRequest>{});
|
||||
invalid(requests, 0);
|
||||
invalid(requests, f.key.world, 0ms);
|
||||
invalid(requests, f.key.world, 30001ms);
|
||||
auto bad = requests;
|
||||
bad[0].objects = nullptr;
|
||||
invalid(bad);
|
||||
bad = requests;
|
||||
bad[0].region = 0;
|
||||
invalid(bad);
|
||||
bad = requests;
|
||||
bad[0].lease.frame = 0;
|
||||
invalid(bad);
|
||||
invalid(std::array{requests[0], requests[0]});
|
||||
auto higher = requests[0];
|
||||
++higher.region;
|
||||
invalid(std::array{higher, requests[0]});
|
||||
invalid(std::vector<CellularCensusRegionRequest>(max_cellular_census_regions + 1, requests[0]));
|
||||
higher.region = 999'999;
|
||||
const auto missing = load_cellular_census_snapshot(*f.db, f.key.world,
|
||||
std::array{requests[0], higher}, builds, 5s);
|
||||
require(!missing && missing.error().kind == CellularSnapshotError::history &&
|
||||
missing.error().region == higher.region && f.db->transaction_idle(),
|
||||
"missing region became partial coverage or left capture open");
|
||||
for (const auto* begin : {"BEGIN", "BEGIN READ ONLY"}) {
|
||||
take(f.db->query(begin, {}, 5s), "caller snapshot transaction");
|
||||
if (std::string_view{begin} == "BEGIN")
|
||||
take(migrate_cellular_to_population(*f.db, *f.objects, f.lease, f.head, f.id(), f.build,
|
||||
{false, 1200, 1800}, 5s),
|
||||
"pending caller snapshot publication");
|
||||
const auto refused = read();
|
||||
require(!refused && refused.error().kind == CellularSnapshotError::invalid_request &&
|
||||
!f.db->transaction_idle(),
|
||||
"snapshot took ownership of a caller transaction");
|
||||
take(f.db->query("ROLLBACK", {}, 5s), "caller snapshot rollback");
|
||||
}
|
||||
const auto manifest = take(decode_manifest(before.manifest), "snapshot retained payload");
|
||||
const auto path = f.objects->directory() / hexadecimal(manifest.payload_sha256, false);
|
||||
const auto saved = path.string() + ".snapshot-test";
|
||||
std::filesystem::rename(path, saved);
|
||||
const auto missing_object = read();
|
||||
std::filesystem::rename(saved, path);
|
||||
require(!missing_object && missing_object.error().kind == CellularSnapshotError::history &&
|
||||
f.db->transaction_idle(),
|
||||
"missing replay payload became a snapshot or left capture open");
|
||||
{
|
||||
const UnreadableObject denied{path};
|
||||
const auto unreadable = read();
|
||||
require(!unreadable && unreadable.error().kind == CellularSnapshotError::history &&
|
||||
f.db->transaction_idle(),
|
||||
"unreadable captured payload became a census");
|
||||
}
|
||||
auto stale = requests;
|
||||
--stale[0].lease.epoch;
|
||||
const auto fenced = load_cellular_census_snapshot(*f.db, f.key.world, stale, builds, 5s);
|
||||
require(!fenced && fenced.error().history.recovery.store.kind == LeaseStoreError::fenced,
|
||||
"snapshot ignored stale evidence authority");
|
||||
const std::array<Sha256Digest, 1> foreign_builds{};
|
||||
require(!load_cellular_census_snapshot(*f.db, f.key.world, requests, foreign_builds, 5s),
|
||||
"snapshot accepted unretained build evidence");
|
||||
auto closed = connection();
|
||||
CellularCensusSnapshotHooks close_hook{[&](std::size_t) { closed->disconnect(); }};
|
||||
const auto lost =
|
||||
load_cellular_census_snapshot(*closed, f.key.world, requests, builds, 5s, close_hook);
|
||||
closed->disconnect();
|
||||
const auto closed_query = closed->query("SELECT 1", {}, 5s);
|
||||
require(!lost && lost.error().kind == CellularSnapshotError::transaction &&
|
||||
!closed->transaction_idle() && !closed_query &&
|
||||
closed_query.error().kind == PgError::closed,
|
||||
"failed capture completion reused a dead session or returned a partial census");
|
||||
auto throwing = connection();
|
||||
bool threw = false;
|
||||
bool hook_called = false;
|
||||
try {
|
||||
const CellularCensusSnapshotHooks hook{[&](std::size_t) {
|
||||
hook_called = true;
|
||||
throw std::runtime_error("capture hook test exception");
|
||||
}};
|
||||
const auto result =
|
||||
load_cellular_census_snapshot(*throwing, f.key.world, requests, builds, 5s, hook);
|
||||
require(!result, "throwing capture hook returned a snapshot");
|
||||
} catch (const std::runtime_error&) {
|
||||
threw = hook_called;
|
||||
}
|
||||
const auto after_throw = throwing->query("SELECT 1", {}, 5s);
|
||||
require(threw && !after_throw && after_throw.error().kind == PgError::closed &&
|
||||
!throwing->transaction_idle() &&
|
||||
take(load_region_checkpoint(*f.db, f.key, 5s), "head after snapshot guards").head ==
|
||||
before.head,
|
||||
"capture exception retained its connection/transaction or changed a head");
|
||||
std::puts("consistent census snapshot guards and connection cleanup passed");
|
||||
snapshot_transport_faults(f);
|
||||
}
|
||||
void boundary_upgrades(Fixture& f) {
|
||||
const std::array<Sha256Digest, 1> builds{f.build};
|
||||
const auto original = f.head;
|
||||
@@ -1180,6 +1333,87 @@ void migration_owner_partitions(Fixture& f, RegionKey destination_key,
|
||||
std::puts("migration independent TLS owner partitions, expiry, replacement and heal passed");
|
||||
}
|
||||
|
||||
void migration_snapshot_race(Fixture& f, RegionKey destination_key,
|
||||
FilesystemObjectStore& destination_objects,
|
||||
const LeaseGrant& destination_lease,
|
||||
CommittedCheckpoint& destination_head, std::uint64_t genome) {
|
||||
const std::array<Sha256Digest, 1> builds{f.build};
|
||||
const std::array requests{CellularCensusRegionRequest{f.key.region, f.lease, f.objects.get()},
|
||||
CellularCensusRegionRequest{destination_key.region, destination_lease,
|
||||
&destination_objects}};
|
||||
auto reader = connection();
|
||||
for (const bool reverse : {false, true}) {
|
||||
auto& sh = reverse ? destination_head : f.head;
|
||||
auto& dh = reverse ? f.head : destination_head;
|
||||
auto& so = reverse ? destination_objects : *f.objects;
|
||||
auto& dst = reverse ? *f.objects : destination_objects;
|
||||
const auto& sl = reverse ? destination_lease : f.lease;
|
||||
const auto& dl = reverse ? f.lease : destination_lease;
|
||||
const auto sk = reverse ? destination_key : f.key;
|
||||
const auto dk = reverse ? f.key : destination_key;
|
||||
const auto before =
|
||||
take(load_cellular_census_snapshot(*reader, f.key.world, requests, builds, 5s),
|
||||
"before snapshot migration race");
|
||||
const CellularMigrationRequest request{f.id(),
|
||||
{sk.region, sh.head.commit, f.id(), sh.head.digest},
|
||||
{dk.region, dh.head.commit, f.id(), dh.head.digest},
|
||||
genome};
|
||||
const auto frozen =
|
||||
take(prepare_cellular_migration_freeze(*f.db, so, dst, sl, sh, dh, request, 5s),
|
||||
"snapshot race source freeze");
|
||||
take(stage_cellular_migration(*f.db, so, dst, dl, sh, dh, frozen.trace, 5s),
|
||||
"snapshot race destination stage");
|
||||
std::optional<CellularMigrationCommitted> published;
|
||||
const CellularCensusSnapshotHooks hook{[&](std::size_t index) {
|
||||
if (index != 0)
|
||||
return;
|
||||
const auto settings =
|
||||
take(reader->query("SELECT current_setting('transaction_isolation'), "
|
||||
"current_setting('transaction_read_only')",
|
||||
{}, 5s),
|
||||
"capture transaction properties");
|
||||
require(settings.size() == 1 && settings[0].size() == 2 &&
|
||||
take(settings[0][0], "capture isolation") == "repeatable read" &&
|
||||
take(settings[0][1], "capture access") == "on",
|
||||
"capture did not own a repeatable read read-only snapshot");
|
||||
auto publisher = connection();
|
||||
published = take(publish_cellular_migration(*publisher, so, dst, sl, dl, sh, dh,
|
||||
frozen.trace, f.build, 5s),
|
||||
"publish real pair between snapshot head reads");
|
||||
}};
|
||||
const auto raced =
|
||||
take(load_cellular_census_snapshot(*reader, f.key.world, requests, builds, 5s, hook),
|
||||
"coherent census across a concurrent pair");
|
||||
const auto committed =
|
||||
take(std::move(published), "pair was not published in snapshot hook");
|
||||
sh = committed.source;
|
||||
dh = committed.destination;
|
||||
require(reader->transaction_idle() && raced.cuts.size() == 2 && before.cuts.size() == 2,
|
||||
"raced capture retained its transaction or lost coverage");
|
||||
for (std::size_t i = 0; i < before.cuts.size(); ++i)
|
||||
require(raced.cuts[i].head == before.cuts[i].head &&
|
||||
raced.cuts[i].manifest == before.cuts[i].manifest &&
|
||||
std::ranges::equal(raced.census.regions()[i].rows(),
|
||||
before.census.regions()[i].rows()),
|
||||
"atomic pair split a captured cut or changed historical replay");
|
||||
const auto after =
|
||||
take(load_cellular_census_snapshot(*reader, f.key.world, requests, builds, 5s),
|
||||
"snapshot after pair publication");
|
||||
require(after.cuts[0].head == f.head.head && after.cuts[1].head == destination_head.head &&
|
||||
after.cuts[0].head.sequence == before.cuts[0].head.sequence + 1 &&
|
||||
after.cuts[1].head.sequence == before.cuts[1].head.sequence + 1,
|
||||
"later snapshot omitted one committed migration endpoint");
|
||||
for (const auto scope :
|
||||
{CellularCensusScope::exact_genome, CellularCensusScope::descendants}) {
|
||||
const auto initial = take(before.census.query(genome, scope), "before pair lineage");
|
||||
const auto captured = take(raced.census.query(genome, scope), "captured pair lineage");
|
||||
const auto settled = take(after.census.query(genome, scope), "settled pair lineage");
|
||||
require(initial == captured && initial.resident == settled.resident,
|
||||
"snapshot migration race lost/duplicated living count/body/reserve");
|
||||
}
|
||||
}
|
||||
std::puts("consistent raw/cohort snapshots across atomic migration commits passed");
|
||||
}
|
||||
void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
FilesystemObjectStore& destination_objects,
|
||||
LeaseGrant& destination_lease, CommittedCheckpoint& destination_head) {
|
||||
@@ -1693,6 +1927,8 @@ void migration_publication(Fixture& f, const RegionKey destination_key,
|
||||
"return migration rewrote earlier committed ownership observations");
|
||||
migration_owner_partitions(f, destination_key, destination_objects, destination_lease,
|
||||
destination_head, genome);
|
||||
migration_snapshot_race(f, destination_key, destination_objects, destination_lease,
|
||||
destination_head, genome);
|
||||
}
|
||||
std::puts("migration object sync/read failures, quarantine and exact recovery passed");
|
||||
// Race the actual two-head transaction against abort on independent connections.
|
||||
@@ -1796,6 +2032,16 @@ void migration_freezes(Fixture& f) {
|
||||
auto destination_lease =
|
||||
take(acquire_region_lease(*f.db, destination_key, f.build, 60000ms, 5s),
|
||||
"migration destination authority");
|
||||
const auto unequal_cut = load_cellular_census_snapshot(
|
||||
*f.db, f.key.world,
|
||||
std::array{CellularCensusRegionRequest{f.key.region, f.lease, f.objects.get()},
|
||||
CellularCensusRegionRequest{destination_key.region, destination_lease,
|
||||
destination_objects.get()}},
|
||||
std::array<Sha256Digest, 1>{f.build}, 5s);
|
||||
require(!unequal_cut && unequal_cut.error().kind == CellularSnapshotError::census &&
|
||||
unequal_cut.error().census == CellularCensusSetError::boundary &&
|
||||
f.db->transaction_idle(),
|
||||
"snapshot silently reconciled different current ticks/owner versions");
|
||||
while (destination->tick() < f.raw->tick()) {
|
||||
const auto count = std::min(std::uint64_t{64}, f.raw->tick() - destination->tick());
|
||||
CellularBatch inputs{{f.key.world, 2, 1, destination_head.head.commit,
|
||||
@@ -2606,6 +2852,7 @@ void run_population_publication_tests() {
|
||||
Fixture fixture;
|
||||
funded_source(fixture);
|
||||
history_pages(fixture);
|
||||
snapshot_guards(fixture);
|
||||
conversions_and_retries(fixture);
|
||||
interval_contract(fixture);
|
||||
ambiguous_publication(fixture, false);
|
||||
|
||||
Reference in New Issue
Block a user