M12.4: retain and settle owned population intervals
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m28s
Foundation / checks (tsan) (push) Failing after 18m22s
Foundation / checks (network) (push) Successful in 20m38s
Foundation / checks (client) (push) Failing after 21m6s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 46m54s
Foundation / checks (checks) (push) Canceled after 47m3s

This commit is contained in:
dingo
2026-10-17 17:31:05 -04:00
parent 627411983f
commit 0a8962b57e
34 changed files with 1032 additions and 75 deletions
+2 -1
View File
@@ -242,7 +242,8 @@ if(EVO_BUILD_POSTGRES)
endif()
find_package(PostgreSQL 18.6 EXACT REQUIRED)
find_package(Threads REQUIRED)
add_library(evo_postgres STATIC src/persistence/postgres.cpp src/persistence/region_lease_store.cpp)
add_library(evo_postgres STATIC src/persistence/postgres.cpp src/persistence/region_lease_store.cpp
src/persistence/population_interval_store.cpp)
add_library(evo::postgres ALIAS evo_postgres)
target_include_directories(evo_postgres PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include")
target_link_libraries(evo_postgres PUBLIC evo::region_lease PRIVATE PostgreSQL::PostgreSQL)
+4 -2
View File
@@ -135,8 +135,10 @@ founder creation and an in-memory cellular tick owner over the finite habitat.
The cellular owner has durable profile-4 recovery/publication and the cellular
worker/session path. Opt-in exact population profile 5 is described in
[M12 durable populations](docs/engineering/M12_DURABLE_POPULATIONS.md);
`--cellular-populations on|off` defaults to off. Durable interval admission,
environmental coupling and population qualification remain in progress. The
`--cellular-populations on|off` defaults to off. Its
[owned intervals](docs/engineering/M12_OWNED_POPULATION_INTERVALS.md) survive
replacement owners; biological migration, environmental coupling and production
qualification remain in progress. The
microscope client is the current M11 playtest path. `evo::simulation` remains
graphics-, network- and database-independent.
+13 -29
View File
@@ -1067,19 +1067,19 @@ int serve_cellular(Options& o) {
held_births.push_back(birth);
}
const bool birth_waiting = !held_births.empty();
const bool want_population =
o.cellular_populations && state &&
state->rules_version() == server::cellular_lives_foraging_rules && !player_present &&
!birth_waiting && host.queued_mutations().empty() &&
std::ranges::any_of(state->lives(), [](const auto& life) {
return life.account == server::LifeAccount{};
});
const bool want_population = o.cellular_populations && state &&
server::population_interval_safe(*state) && !player_present &&
!birth_waiting && host.queued_mutations().empty() &&
std::ranges::any_of(state->lives(), [](const auto& life) {
return life.account == server::LifeAccount{};
});
if (status.ready && state && want_population != host.population_mode()) {
[[maybe_unused]] const auto changed = host.set_population_mode(want_population);
state = host.state();
}
const bool needed = state && (player_present || birth_waiting ||
server::cellular_unattended_tick_needed(*state));
server::cellular_unattended_tick_needed(*state) ||
status.ecological_pending_ticks != 0);
const auto monotonic_us = [] {
return static_cast<std::uint64_t>(
std::chrono::duration_cast<std::chrono::microseconds>(
@@ -1088,32 +1088,16 @@ int serve_cellular(Options& o) {
};
const auto paced = schedule.observe(monotonic_us(), status.epoch,
state ? state->tick() : status.committed_tick, needed,
status.ready && state);
status.ready && state, status.ecological_pending_ticks);
if (!paced) {
std::puts("error cellular-schedule");
return finish(4);
}
if (status.ready && state && needed) {
if (status.ready && state && (needed || host.population_mode())) {
if (host.population_mode()) {
// Collect one second of due time before an unattended publication.
// Overload services at most 64 fixed steps per bounded batch. Debt
// remains visible and is consumed only by a confirmed head.
if (paced->due && paced->debt >= 20) {
const auto& head = host.committed()->head;
const server::CellularBatchHeader header{
o.host.key.world,
o.host.key.region,
o.host.frame,
head.commit,
head.digest,
head.end_tick,
head.end_tick + std::min<std::uint64_t>(paced->debt, 64)};
const auto inputs = server::unattended_population_inputs(*state, header);
const auto end = header.end_tick;
if (inputs) {
[[maybe_unused]] const auto published = host.publish(*inputs);
}
if (!host.state() || host.state()->tick() < end)
if (paced->retry_ready && (paced->due || status.ecological_pending_ticks != 0)) {
const auto published = host.advance_unattended(paced->due ? paced->debt : 0);
if (!published)
schedule.defer(monotonic_us());
}
} else if (paced->due && state->tick() != UINT64_MAX) {
+16
View File
@@ -1,5 +1,21 @@
# Project status and next steps
## M12.4b owned population intervals - 2026-10-17
ADR-0143/schema 45 records one bounded interval before computation and settles it
atomically with its exact checkpoint. Replacement owners retain the same pending
work; zero newly due ticks can finish a recovered interval. Unknown admission
pauses readiness and retries its immutable source/end. A separate durable pending
tick metric complements local debt and checkpoint age.
New entry/work requires fully suspended controlled cells: the existing 200-tick
vulnerable linger and permanent-ending path stays on profile 4. Old traces retain
their meaning, and either profile's recovery reconciles owned corpses. No new
biological version, immunity, dependency or deployment was introduced. See
[scope and evidence](engineering/M12_OWNED_POPULATION_INTERVALS.md). Biological
migration escrow and linked-boundary/resource contracts remain the next M12.4
work; this does not close all of M12 or enable production automatically.
## M12.3b durable exact populations - 2026-10-17
ADR-0142 and schema 44 add dedicated population profile 5, trace 1, immutable
@@ -0,0 +1,71 @@
# ADR-0143: Owned bounded population intervals
## Status
Accepted for M12.4's opt-in engineering path, 2026-10-17. Extends ADR-0133,
0141 and 0142 without changing biology, controller decisions or World0 policy.
M12.4b's interval segment is implemented and validated. Biological migration and
the rest of M12.4 remain separate work; this does not close the entire gate.
## Context
Local confirmed-head pacing preserves debt while an owner runs, but loses its
unpublished interval on process loss. Inferring a replacement interval from elapsed
wall time would violate the accepted outage pause and could change controlled
biology. Profile 5 already supplies exact frozen-player/wildlife replay and bounded
publication. It needs a durable operational admission boundary.
## Decision
Admit one interval per region under its current lease and exact profile-5 head.
Bind the source commit/digest, begin/end ticks, controller version, admitting
holder and epoch. Intervals contain 1..64 unchanged 50 ms biological steps. Keep
admission records immutable. Identical retries return the same interval; different
end boundaries for one source conflict. New admissions require the exact current
head and no unsettled predecessor interval.
Write its immutable settlement with the checkpoint in the same transaction.
Normal profile-5 biological publication must consume precisely its admission;
manual conversion or profile exit waits while any interval is unsettled. Old
committed records remain recoverable and exactly retryable without fabricated
retroactive admission records. Generic publishers cannot bypass these guards.
A replacement fenced owner loads pending admission before selecting fresh work.
It advances from the verified source using the recorded controller version and
frozen controlled intentions, then publishes the exact trace/payload. Unknown
admission replies retain the same request until reconciliation; unknown checkpoint
replies retain their existing prepared bytes. Read-only resolution grants no
authority. A returning player waits for this bounded admitted interval, then the
host materializes before admitting player operations.
New profile entry and interval preparation require every controlled cell to be
fully suspended. Frozen intentions alone are insufficient: during the existing
200-tick linger period a cell is still vulnerable to predation, and its permanent
ending must stay on the existing profile-4 path. That path finishes linger and
records any death before entry; no new immunity or biological timer is introduced.
Earlier profile-5 traces remain verifiable with their original semantics and can
be materialized. Recovery reconciles retained owned corpses for either profile,
so an older post-commit/pre-ending crash cannot leave its account bound forever.
Collect one second of due time for routine unattended admissions, service at most
64 ticks per batch, and preserve debt until the confirmed head consumes it. A
short terminal-u64 remainder is admitted without waiting for 20 ticks. Report
pending admitted ticks separately from local due debt and checkpoint freshness.
Capacity pressure pauses admission; it cannot drop an admitted interval or cull
organisms. Unadmitted wall time remains paused across outage (ADR-0133).
## Alternatives
Keeping only a local deadline loses admitted work on restart. Reconstructing it
from database timestamps adds an unaccepted catch-up policy. A transaction per
tick violates the founding durability requirement. An unbounded queue conceals
resource commitments and makes returning-player latency unbounded.
## Consequences and limits
This adds a bounded operational journal and transaction guards, not a new time
integrator or ecological rule. Require duplicate/conflicting admissions, lost
replies, stale writers, process loss before/after admission and publication,
restart with zero new due time, and exact uninterrupted continuation comparisons.
Migration escrow, delayed boundary coupling, environmental ownership and multi-day
evidence remain separate M12 work. Production enablement stays off by default.
+9
View File
@@ -40,6 +40,15 @@ state, materializes before player operations and never treats read-only commit
resolution as current authority. See [the contract](../../schemas/cellular-population-batch.md)
and [qualification limits](../engineering/M12_DURABLE_POPULATIONS.md).
ADR-0143/schema 45 adds one durable 1..64-tick admission per population head,
with immutable source/digest/ticks/controller evidence and an atomically maintained
pending pointer. Settlement and checkpoint publication share one transaction.
Recovery services that exact pending prefix before fresh work; conversion/profile
exit cannot skip it. Unknown admission retains the same request and pauses
readiness. New work requires fully suspended controlled cells, leaving vulnerable
linger/death on profile 4. Earlier trace semantics remain readable. See the
[interval contract](../../schemas/ecology/population-interval-v1.md).
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them.
+9 -4
View File
@@ -20,7 +20,7 @@ removes stale ownership tags; funded births begin as individuals. Suspension
continues to freeze controlled biology regardless of representation elsewhere.
The model retains exact members and makes no aggregation speedup claim. There is
no regional population approximation, process migration or production selector.
no regional population approximation or biological cross-region migration.
The existing 256-life ceiling remains; target-host service capacity is separate.
M12.3 adds [population envelope 1](../../schemas/cellular-population.md), detached
@@ -34,9 +34,14 @@ biology restore together; loading a checkpoint does not reselect members.
`conversion_candidate` leaves the predecessor intact and returns the complete
exact successor for fenced publication. Successful candidate creation and local
checkpoint file writes are not durable production acknowledgements. A dedicated
PostgreSQL profile/trace/publication adapter is still required; existing raw
cellular profile 4 cannot contain this envelope. The separate-process test
demonstrates identical continuation, not power-loss or database-failover safety.
PostgreSQL adapter now uses dedicated profile 5 (ADR-0142), leaving raw profile 4
unchanged. ADR-0143 adds owned bounded unattended intervals. New entry and interval
publication require fully suspended controlled cells; active/lingering cells stay
on the existing raw path so permanent endings remain correct. Older recorded
traces remain verifiable. The selector is opt-in and off by default; see
[durable populations](../engineering/M12_DURABLE_POPULATIONS.md) and
[owned intervals](../engineering/M12_OWNED_POPULATION_INTERVALS.md) for evidence
and the remaining production qualification.
Run `evo-cellular-cohorts --verify` for the CTest matrix or omit the option for
8192 ticks per case. The executable compares complete canonical state every tick,
+1 -1
View File
@@ -11,7 +11,7 @@ Commit and push each validated scoped segment separately.
| 12.1 | Engineering evidence policy and explicit deferrals documented | ADR-0133, identity/time contract and worked histories; no new game rules or runtime versions. |
| 12.2 | Exact local model selected and paired qualification implemented (12.2d); production cost gate open | ADR-0139, exact canonical equality across 196,608 paired ticks per platform. Regional approximation disabled; no compression claim. Deployment-host capacity and fuller interaction coverage remain open. |
| 12.3 | Exact population profile/trace/store and opt-in host integration implemented (12.3b) | ADR-0142, schema 44; exact publication/recovery, mutation race guards, ambiguous replies and process kills. Executable/network population fault drills and target-host qualification remain open. |
| 12.4 | Current-worker unattended demand and confirmed-head pacing implemented (12.4a) | ADR-0141. Population durable intervals and biological migration escrow remain unimplemented. |
| 12.4 | Unattended demand/pacing plus owned population interval admission/recovery implemented (12.4b) | ADR-0143/schema 45; bounded durable pending work, exact settlement, replacement-owner and uncertainty tests. Biological migration escrow and boundary/stock-in-transit evidence remain open. |
| 12.5 | Blocked on 12.2-12.4 | Finite M6 process fixtures do not qualify coupled persistent cellular/cohort atmosphere/water budgets. |
| 12.6 | Existing local archive retained; integrated population history open | World-level durable event/query/client path remains required. |
| 12.7 | Evidence policy defined; reconciler open | Complete durable world watermark and every viable/in-flight store remain required. |
@@ -61,6 +61,11 @@ services at most 64 fixed steps per batch. The portable input builder matches an
independent per-tick fixture. This removes per-tick SQL publication from the new
population path; it does not yet make local pacing debt a durable admission queue.
M12.4b subsequently adds the [owned interval journal](M12_OWNED_POPULATION_INTERVALS.md).
It restricts new entry/work to fully suspended controlled cells, preserving the
vulnerable linger/death path on profile 4. Earlier traces remain verifiable, and
retained owned corpses are reconciled after either profile's recovery.
Still open: profile-5 executable/network fault drills, deployment-host tail costs,
durable ecological interval admission and biological migration escrow, jointly
owned environmental budgets, world-level history/extinction reconciliation and
@@ -0,0 +1,64 @@
# M12.4b owned unattended population intervals
ADR-0143 and schema 45 add one durable admitted interval per population region.
The source commit/digest and exact begin/end ticks identify it; immutable admission
and settlement journals retain its history. A bounded pending pointer makes work
lookup independent of accumulated history. Publication settles that exact interval
atomically with its checkpoint. An aborting transaction leaves it pending.
The optional worker collects at least 20 due ticks (one second), services at most
64 unchanged 50 ms steps, and consumes debt only from a confirmed head. The final
short u64 remainder cannot wrap or wait forever. A recovered pending interval runs
with zero newly due time; it does not consume or delay fresh wall-time accounting.
Admission uncertainty retains the same source/end request and pauses readiness.
Checkpoint uncertainty retains its complete prepared bytes. Neither a timeout,
duplicate request nor new lease may resample work, skip its prefix or reopen it.
The host reports `evo_region_ecological_pending_ticks` (0..64), separately from
local tick debt and checkpoint age. Fresh unadmitted wall time remains paused
during outage/uncertainty. This does not introduce whole-world catch-up.
## Suspension and retained compatibility
Inspection of ADR-0128 and the cellular kernel found an important restriction:
frozen intentions stop controlled biology, but lingering cells remain vulnerable
for 200 ticks. New population entry and interval preparation therefore require
every controlled cell fully suspended. Linger/death stays on the existing raw
cellular path, which records account endings before population entry. This changes
operational admission, not immunity, movement, funding, biology versions or goldens.
Existing profile-5 envelopes and recorded frozen/lingering traces retain their
meaning. Recovery verifies them and allows reverse materialization; new biological
publication from such an unsafe source is refused. Either profile's recovery now
reconciles retained owned corpses, including a legacy post-commit/pre-ending crash.
The portable general population model and its active-player qualification remain
unchanged; the durable unattended profile has the stricter admission contract.
## Evidence and limits
The private PostgreSQL fixture uses funded reserved founders and descendants. It
publishes the real 201-tick raw suspension prefix before new profile entry. Tests
cover duplicate/conflicting/oversized admissions, partial settlement refusal,
conversion/profile-exit refusal while pending, a replacement epoch retaining the
same interval, exact settlement, old retry without reopening work, and revoked
private publisher access. Publisher SIGKILLs before transaction/after commit/after
pipe ACK also verify whether the interval remains pending or settled.
An actual owning host is drained/recreated with pending work. It restores the
same head, refuses early materialization, services the recorded interval with
zero new due ticks, and then permits player operations. Lease-row contention
causes a real admission timeout; readiness and biology stay paused until the
identical request reconciles, then its exact reference trajectory publishes.
The 65-tick local request is capped at 64; a 19-tick routine window waits.
Fixed-clock tests check recovered-prefix accounting, bounded retry without fresh
debt, overflow/refusal and metrics shape/cardinality. Legacy trace replay and new
active/lingering-source refusal are separate contract checks. Executed compiler,
sanitizer and database results are recorded in [VALIDATION](VALIDATION.md).
This closes the scoped interval-admission/recovery segment. Biological cross-region
escrow, delayed boundary coupling and stock-in-transit audits remain M12.4 work.
No probe inventory stands in for biological stock. Deployment-host costs, mixed
controlled/cohort publication, environmental coupling, integrated history/global
extinction and actual multi-day/product qualification remain open. Production
entry stays disabled by default; no live region or World0 was changed.
+7 -5
View File
@@ -14,8 +14,10 @@ now support exact population membership with entry disabled by default. Private
PostgreSQL tests cover ambiguity, fencing, profile races and publisher process
kills. [Scope and limits](M12_DURABLE_POPULATIONS.md) distinguish those tests from
executable/network fault drills and deployment-host qualification still required.
This resolves the architectural profile conflict and permits the next durable
interval work. It does not close all of M12.3 or authorize production enablement.
This resolves the architectural profile conflict. ADR-0143/schema 45 now adds
bounded durable interval admission and exact settlement/recovery, with fully
suspended controlled cells required for new work. It does not close all of M12.3
or authorize production enablement.
## Dependent ecological integration
@@ -23,7 +25,7 @@ interval work. It does not close all of M12.3 or authorize production enablement
|---|---|---|
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Current worker advances unattended wildlife and preserves running scheduling debt | Durable ecological interval admission/backlog recovery; exact biological source/destination escrow, partition/duplicate settlement and stock-in-transit audits. |
| 12.4 | Current-worker demand/debt plus owned bounded population intervals and restart backlog recovery | Exact biological source/destination escrow, partition/duplicate settlement, linked-boundary contracts and stock-in-transit audits. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
@@ -40,8 +42,8 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics.
The engineering constitution requires stopping expansion at this architectural
boundary and documenting the resolution. That does not prevent the independent
worker scheduling correction or the dedicated exact population adapter above.
Durable ecological intervals are the next substantive coding segment; none of
the rows above is closed by adding a placeholder service.
Biological migration escrow is the next M12.4 boundary; none of the rows above is
closed by adding a placeholder service or relabeling the non-biological probe.
Current expansion readiness: **no-go for M13 integration or World0 promotion**
while these M12 integration/evidence gates remain open. This records the present
+45
View File
@@ -1,5 +1,50 @@
# Validation report
## 2026-10-17 - M12.4b owned population intervals
ADR-0143/schema 45 affects population admission/settlement journals, the owning
host, confirmed-head pacing and bounded operational metrics. New entry/work waits
for full controlled suspension; legacy trace replay and owned-corpse recovery are
retained. Biology versions, old goldens and the general population kernel remain
unchanged. No live region, World0 operation or new dependency.
Executed:
- MSVC 19.44 Release configure/build and **76/76 CTests** (final 10.39 s), including
active/lingering-source refusal, retained legacy trace replay, recovered-prefix
clock accounting and fixed-cardinality pending-work metrics.
- Analysis-preset clang-tidy 19 and repository format check; direct portable
population/pacing/metrics CTests **3/3 passed** (4.40 s).
- Actual GCC QUIC/PostgreSQL server build; focused private PostgreSQL suite with
the final suspension source **passed, exit 0, 6 s**. Real reserved founders fund
descendants; 201 raw frozen steps finish linger before new profile entry.
- Tests reject conflicting/oversized admissions, shortened settlement, conversion
or profile exit while pending, and the private pre-journal publisher. A new
epoch restores the same admission; committed retries do not reopen it. Real
publisher SIGKILLs verify pending versus settled work at each tested boundary.
- Recreated actual host finishes a pending interval with zero new due ticks.
Lease-row contention causes a real admission timeout; readiness and biology
stay paused until the exact request reconciles, then match uninterrupted
reference continuation. A 65-tick request is capped at 64; a routine 19 waits.
- Actual PostgreSQL host/adapter tests under Clang 19 **ASan/UBSan passed, exit 0,
19 s**, default sanitizer environment, no reports. The earlier 21 s run disabled
leak checking; the final default-environment run above supersedes it.
- Actual optional Clang server build and explicit clang-tidy 19 on the interval
store, population publication store, host, integration tests and worker. The
final repeated-branch cleanup in error classification passes its follow-up
tidy check; no owned warnings remain.
Final frozen-binary `postgres-check.sh --full` **passed, exit 0, 497 s**: complete
C++ suite (422 s), independent reservation connections, backup/restore, host-loss
DR, upgrade, migration-history, permission and checksum-drift checks. Migration
history and rollback drills include schema 45; private old publishers are revoked.
Limits: these remain private-cluster publisher/host tests, not population-specific
network ACK fault qualification or actual multi-day ecology. Biological migration
escrow, delayed linked-boundary coupling and stock-in-transit audits remain open;
the M10 probe is not biological evidence. Details:
[M12_OWNED_POPULATION_INTERVALS](M12_OWNED_POPULATION_INTERVALS.md).
## 2026-10-17 - M12.3b durable exact population publication
ADR-0142 affects portable lifecycle batches/checkpoint manifests, optional
@@ -0,0 +1,21 @@
#pragma once
#include "evo/persistence/region_lease_store.hpp"
namespace evo::persistence {
// ADR-0143 / schema 45. Exactly one pending bounded interval, identified by its
// immutable source head. A new holder may finish it; admission grants no lease.
struct PopulationInterval {
Uuid source_commit{};
Sha256Digest source_manifest_sha256{};
std::uint64_t begin_tick{}, end_tick{};
std::uint32_t controller_version{1};
bool operator==(const PopulationInterval&) const = default;
};
[[nodiscard]] std::expected<PopulationInterval, LeaseStoreFailure>
admit_population_interval(Postgres&, RegionKey, const LeaseGrant&, const RegionHead& source,
std::uint64_t end_tick, std::chrono::milliseconds timeout);
[[nodiscard]] std::expected<std::optional<PopulationInterval>, LeaseStoreFailure>
load_population_interval(Postgres&, RegionKey, const LeaseGrant&,
std::chrono::milliseconds timeout);
} // namespace evo::persistence
@@ -4,6 +4,7 @@
#include "evo/persistence/checkpoint_store.hpp"
#include "evo/persistence/life_reservation_store.hpp"
#include "evo/persistence/population_checkpoint_store.hpp"
#include "evo/persistence/population_interval_store.hpp"
#include "evo/server/lifecycle/cellular_batch.hpp"
#include "evo/server/ops.hpp"
@@ -93,6 +94,11 @@ class CellularRegionHost {
// Explicit no-tick fenced transition. Uncertain publication pauses the host.
[[nodiscard]] std::expected<void, CellularPublishFailure> set_population_mode(bool enabled);
[[nodiscard]] bool population_mode() const noexcept;
// Service a recorded interval even when new due time is zero after restart.
// Otherwise admit a bounded prefix of due ticks before computing it.
[[nodiscard]] std::expected<std::optional<persistence::CommittedCheckpoint>,
CellularPublishFailure>
advance_unattended(std::uint64_t due_ticks);
[[nodiscard]] std::expected<persistence::PublishedCellularBatch, CellularPublishFailure>
publish(const CellularBatch& batch, std::span<const CellularActionRecord> actions = {},
std::span<const CellularMutationActionRecord> mutation_actions = {});
@@ -147,6 +153,8 @@ class CellularRegionHost {
void resolve();
void refresh();
bool recover_state(const persistence::LeaseGrant&);
void reconcile_interval();
void finish_interval();
[[nodiscard]] std::expected<persistence::PublishedCellularBatch, CellularPublishFailure>
publish_population(const CellularBatch&);
void finalize_births(std::span<const CellularTickIntents::Birth> births);
@@ -207,6 +215,15 @@ class CellularRegionHost {
std::expected<persistence::CommittedCheckpoint, persistence::PopulationPublicationFailure>,
OwnedState candidate);
OwnedState state_;
bool interval_reload_{};
std::optional<persistence::PopulationInterval> admitted_interval_;
// An uncertain admission must retry its exact source/target, never select again.
struct IntervalRequest {
persistence::RegionHead source;
std::uint64_t end_tick{};
};
std::optional<IntervalRequest> interval_request_;
std::optional<persistence::LeaseStoreFailure> interval_failure_;
std::optional<persistence::CommittedCheckpoint> committed_;
std::unique_ptr<Pending> pending_;
std::vector<FinalCellularAction> finalized_actions_;
@@ -30,6 +30,9 @@ struct CellularPopulationExecution {
CellularPopulationBatch batch;
std::unique_ptr<CellularPopulation> state;
};
// Lingering controlled cells remain vulnerable. Their permanent death path must
// stay on profile 4; unattended population intervals require full suspension.
[[nodiscard]] bool population_interval_safe(const CellularLives&) noexcept;
// Build an unattended interval from a detached biological copy. Controlled cells
// receive frozen-only intentions; the unchanged wildlife controller supplies the
// rest. The returned recorded intentions replay without rerunning that controller.
@@ -11,9 +11,15 @@ namespace evo::server {
// Historical profiles without offline freeze are not opted into unattended work.
[[nodiscard]] bool cellular_unattended_tick_needed(const CellularLives& state) noexcept;
enum class CellularScheduleError : std::uint8_t { clock_regression, head_regression, authority };
enum class CellularScheduleError : std::uint8_t {
clock_regression,
head_regression,
authority,
admission
};
struct CellularScheduleStatus {
bool due{};
bool retry_ready{true}; // Also permits servicing a recovered durable interval with zero debt.
std::uint64_t debt{}, max_debt{};
std::uint64_t head_age_us{}; // Since the last confirmed head change/recovery.
};
@@ -22,11 +28,13 @@ struct CellularScheduleStatus {
// be attempted per poll. Rejected work does not consume debt. Authority loss,
// restart or absence of tick demand pauses wall-time (ADR-0133); committed
// intervals and uncertain outcomes remain the fenced host's responsibility.
// On a new anchor, previously_admitted_ticks excludes a recovered durable prefix
// from fresh wall-time accounting. It is bounded to 64 and cannot wrap the head.
class CellularTickSchedule {
public:
[[nodiscard]] std::expected<CellularScheduleStatus, CellularScheduleError>
observe(std::uint64_t now_us, std::uint64_t epoch, std::uint64_t committed_tick, bool needed,
bool authoritative);
bool authoritative, std::uint64_t previously_admitted_ticks = 0);
void defer(std::uint64_t now_us) noexcept;
[[nodiscard]] CellularScheduleStatus status() const noexcept;
+1
View File
@@ -34,6 +34,7 @@ struct HostStatus {
std::uint64_t simulated_tick{}, committed_tick{}, committed_sequence{};
std::size_t queued{}, queued_bytes{};
std::uint64_t tick_debt{}, max_tick_debt{}; // Due minus simulated since running.
std::uint64_t ecological_pending_ticks{}; // Durable admitted interval, 0..64.
std::uint64_t commits{}, store_failures{}, unknown_outcomes{}, fences{};
// Local lease safety: time until this host must stop originating (0 = none).
std::uint64_t lease_remaining_ms{};
@@ -0,0 +1,161 @@
-- ADR-0143: one owned bounded ecological interval, settled with its checkpoint.
CREATE TABLE evolution.population_interval_admissions (
source_commit uuid PRIMARY KEY REFERENCES evolution.region_commits(commit_id),
world_id evolution.u64 NOT NULL,
region_id evolution.u64 NOT NULL,
source_manifest_sha256 bytea NOT NULL CHECK (octet_length(source_manifest_sha256) = 32),
begin_tick evolution.u64 NOT NULL,
end_tick evolution.u64 NOT NULL CHECK (end_tick > begin_tick AND end_tick - begin_tick <= 64),
controller_version integer NOT NULL CHECK (controller_version = 1),
admitted_holder uuid NOT NULL,
admitted_epoch evolution.u64 NOT NULL CHECK (admitted_epoch > 0)
);
CREATE INDEX population_interval_region ON evolution.population_interval_admissions(world_id,region_id);
CREATE TABLE evolution.population_interval_settlements (
source_commit uuid PRIMARY KEY REFERENCES evolution.population_interval_admissions(source_commit),
commit_id uuid UNIQUE NOT NULL REFERENCES evolution.region_commits(commit_id) DEFERRABLE INITIALLY DEFERRED
);
-- Bounded operational pointer, maintained atomically with the immutable journal.
CREATE TABLE evolution.population_interval_pending (
world_id evolution.u64 NOT NULL, region_id evolution.u64 NOT NULL,
source_commit uuid UNIQUE NOT NULL REFERENCES evolution.population_interval_admissions(source_commit),
PRIMARY KEY(world_id,region_id)
);
REVOKE ALL ON evolution.population_interval_admissions,evolution.population_interval_settlements,
evolution.population_interval_pending FROM PUBLIC;
CREATE TRIGGER population_interval_admissions_immutable BEFORE UPDATE OR DELETE ON evolution.population_interval_admissions
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER population_interval_admissions_no_truncate BEFORE TRUNCATE ON evolution.population_interval_admissions
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER population_interval_settlements_immutable BEFORE UPDATE OR DELETE ON evolution.population_interval_settlements
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE TRIGGER population_interval_settlements_no_truncate BEFORE TRUNCATE ON evolution.population_interval_settlements
FOR EACH STATEMENT EXECUTE FUNCTION evolution.refuse_history_rewrite();
CREATE FUNCTION evolution.admit_population_interval(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64,
source uuid,source_digest bytea,target_tick evolution.u64)
RETURNS TABLE(source_commit uuid,source_manifest_sha256 bytea,begin_tick evolution.u64,end_tick evolution.u64,controller_version integer)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE a evolution.population_interval_admissions%ROWTYPE; h evolution.region_heads%ROWTYPE;
c evolution.checkpoint_manifest; m bytea;
BEGIN
IF source IS NULL OR source_digest IS NULL OR octet_length(source_digest) <> 32 OR target_tick IS NULL THEN
RAISE EXCEPTION 'invalid population interval' USING ERRCODE = '22023';
END IF;
PERFORM evolution.check_region_lease(w,r,holder,epoch);
SELECT * INTO a FROM evolution.population_interval_admissions x WHERE x.source_commit = source;
IF FOUND THEN
IF a.world_id <> w OR a.region_id <> r OR a.source_manifest_sha256 <> source_digest OR a.end_tick <> target_tick THEN
RAISE EXCEPTION 'population interval identity reused' USING ERRCODE = 'EVL07';
END IF;
RETURN QUERY SELECT a.source_commit,a.source_manifest_sha256,a.begin_tick,a.end_tick,a.controller_version;
RETURN;
END IF;
SELECT * INTO h FROM evolution.region_heads x WHERE x.world_id = w AND x.region_id = r FOR UPDATE;
IF NOT FOUND OR h.commit_id <> source OR h.digest <> source_digest THEN
RAISE EXCEPTION 'population interval head moved' USING ERRCODE = 'EVL03';
END IF;
SELECT x.manifest INTO m FROM evolution.region_commits x WHERE x.commit_id = source;
c := evolution.parse_checkpoint_manifest(m);
IF c.profile <> 5 OR target_tick <= h.end_tick OR target_tick - h.end_tick > 64 THEN
RAISE EXCEPTION 'unsupported population interval' USING ERRCODE = 'EVL06';
END IF;
IF EXISTS (SELECT 1 FROM evolution.population_interval_pending x WHERE x.world_id = w AND x.region_id = r) THEN
RAISE EXCEPTION 'population interval already pending' USING ERRCODE = 'EVL18';
END IF;
INSERT INTO evolution.population_interval_admissions VALUES(source,w,r,source_digest,h.end_tick,target_tick,1,holder,epoch)
RETURNING * INTO a;
INSERT INTO evolution.population_interval_pending VALUES(w,r,source);
RETURN QUERY SELECT a.source_commit,a.source_manifest_sha256,a.begin_tick,a.end_tick,a.controller_version;
END $$;
REVOKE ALL ON FUNCTION evolution.admit_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,bytea,evolution.u64) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.admit_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,bytea,evolution.u64) TO :"runtime_role";
CREATE FUNCTION evolution.load_population_interval(w evolution.u64,r evolution.u64,holder uuid,epoch evolution.u64)
RETURNS TABLE(source_commit uuid,source_manifest_sha256 bytea,begin_tick evolution.u64,end_tick evolution.u64,controller_version integer)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE a evolution.population_interval_admissions%ROWTYPE;
BEGIN
PERFORM evolution.check_region_lease(w,r,holder,epoch);
SELECT x.* INTO a FROM evolution.population_interval_pending p JOIN evolution.population_interval_admissions x
ON x.source_commit = p.source_commit WHERE p.world_id = w AND p.region_id = r;
IF NOT FOUND THEN RETURN; END IF;
IF a.world_id <> w OR a.region_id <> r OR EXISTS (SELECT 1 FROM evolution.population_interval_settlements s
WHERE s.source_commit = a.source_commit) THEN
RAISE EXCEPTION 'invalid population interval pointer' USING ERRCODE = 'XX000';
END IF;
IF NOT EXISTS (SELECT 1 FROM evolution.region_heads h WHERE h.world_id = w AND h.region_id = r
AND h.commit_id = a.source_commit AND h.digest = a.source_manifest_sha256 AND h.end_tick = a.begin_tick) THEN
RAISE EXCEPTION 'population interval source lost' USING ERRCODE = 'EVL03';
END IF;
RETURN QUERY SELECT a.source_commit,a.source_manifest_sha256,a.begin_tick,a.end_tick,a.controller_version;
END $$;
REVOKE ALL ON FUNCTION evolution.load_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.load_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64) TO :"runtime_role";
ALTER FUNCTION evolution.publish_population_checkpoint(bytea,bytea) RENAME TO publish_population_checkpoint_v44;
REVOKE ALL ON FUNCTION evolution.publish_population_checkpoint_v44(bytea,bytea) FROM PUBLIC, :"runtime_role";
CREATE FUNCTION evolution.publish_population_checkpoint(m bytea,t bytea)
RETURNS TABLE(region_frame evolution.u64,head_sequence evolution.u64,head_commit uuid,head_digest text,head_tick evolution.u64)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE c evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(m); a evolution.population_interval_admissions%ROWTYPE;
BEGIN
PERFORM evolution.check_population_trace(m,t);
PERFORM evolution.check_region_lease(c.world_id,c.region_id,c.holder,c.epoch);
IF EXISTS (SELECT 1 FROM evolution.region_commits x WHERE x.commit_id = c.commit_id) THEN
RETURN QUERY SELECT * FROM evolution.publish_population_checkpoint_v44(m,t);
RETURN;
END IF;
IF c.end_tick > c.begin_tick THEN
SELECT x.* INTO a FROM evolution.population_interval_pending p JOIN evolution.population_interval_admissions x
ON x.source_commit = p.source_commit WHERE p.world_id = c.world_id AND p.region_id = c.region_id
AND p.source_commit = c.predecessor_commit;
IF NOT FOUND OR a.world_id <> c.world_id OR a.region_id <> c.region_id
OR a.source_manifest_sha256 <> c.predecessor_digest OR a.begin_tick <> c.begin_tick OR a.end_tick <> c.end_tick
OR EXISTS (SELECT 1 FROM evolution.population_interval_settlements s WHERE s.source_commit = a.source_commit) THEN
RAISE EXCEPTION 'population publication requires exact interval' USING ERRCODE = 'EVL18';
END IF;
INSERT INTO evolution.population_interval_settlements VALUES(a.source_commit,c.commit_id);
DELETE FROM evolution.population_interval_pending p WHERE p.world_id = c.world_id AND p.region_id = c.region_id
AND p.source_commit = a.source_commit;
ELSIF EXISTS (SELECT 1 FROM evolution.population_interval_pending x
WHERE x.world_id = c.world_id AND x.region_id = c.region_id) THEN
RAISE EXCEPTION 'population conversion waits for interval' USING ERRCODE = 'EVL18';
END IF;
RETURN QUERY SELECT * FROM evolution.publish_population_checkpoint_v44(m,t);
END $$;
REVOKE ALL ON FUNCTION evolution.publish_population_checkpoint(bytea,bytea) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.publish_population_checkpoint(bytea,bytea) TO :"runtime_role";
ALTER FUNCTION evolution.publish_population_profile_transition(bytea) RENAME TO publish_population_profile_transition_v44;
REVOKE ALL ON FUNCTION evolution.publish_population_profile_transition_v44(bytea) FROM PUBLIC, :"runtime_role";
CREATE FUNCTION evolution.publish_population_profile_transition(m bytea)
RETURNS TABLE(region_frame evolution.u64,head_sequence evolution.u64,head_commit uuid,head_digest text,head_tick evolution.u64)
LANGUAGE plpgsql SECURITY DEFINER SET search_path = pg_catalog, pg_temp AS $$
DECLARE c evolution.checkpoint_manifest := evolution.parse_checkpoint_manifest(m);
BEGIN
PERFORM evolution.check_region_lease(c.world_id,c.region_id,c.holder,c.epoch);
IF NOT EXISTS (SELECT 1 FROM evolution.region_commits x WHERE x.commit_id = c.commit_id)
AND EXISTS (SELECT 1 FROM evolution.population_interval_pending x
WHERE x.world_id = c.world_id AND x.region_id = c.region_id) THEN
RAISE EXCEPTION 'profile transition waits for interval' USING ERRCODE = 'EVL18';
END IF;
RETURN QUERY SELECT * FROM evolution.publish_population_profile_transition_v44(m);
END $$;
REVOKE ALL ON FUNCTION evolution.publish_population_profile_transition(bytea) FROM PUBLIC;
GRANT EXECUTE ON FUNCTION evolution.publish_population_profile_transition(bytea) TO :"runtime_role";
CREATE FUNCTION evolution.refuse_unsettled_population_interval() RETURNS trigger
LANGUAGE plpgsql SECURITY INVOKER SET search_path = pg_catalog, pg_temp AS $$
BEGIN
IF EXISTS (SELECT 1 FROM evolution.population_interval_pending a
WHERE a.world_id = NEW.world_id AND a.region_id = NEW.region_id) THEN
RAISE EXCEPTION 'unsettled population interval blocks publication' USING ERRCODE = 'EVL18';
END IF;
RETURN NEW;
END $$;
REVOKE ALL ON FUNCTION evolution.refuse_unsettled_population_interval() FROM PUBLIC;
CREATE TRIGGER region_population_interval BEFORE INSERT ON evolution.region_commits
FOR EACH ROW EXECUTE FUNCTION evolution.refuse_unsettled_population_interval();
INSERT INTO evolution.schema_migrations(version,sha256) VALUES(45,:'migration_sha_45');
+14 -3
View File
@@ -120,6 +120,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
SELECT :'history_v41' || ',42:' || :'migration_sha_42' AS history_v42 \gset
SELECT :'history_v42' || ',43:' || :'migration_sha_43' AS history_v43 \gset
SELECT :'history_v43' || ',44:' || :'migration_sha_44' AS history_v44 \gset
SELECT :'history_v44' || ',45:' || :'migration_sha_45' AS history_v45 \gset
SELECT :'history' = :'history_v22' AS at_v22,
:'history' = :'history_v23' AS at_v23,
:'history' = :'history_v24' AS at_v24,
@@ -142,8 +143,10 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution')
:'history' = :'history_v41' AS at_v41,
:'history' = :'history_v42' AS at_v42,
:'history' = :'history_v43' AS at_v43,
:'history' = :'history_v44' AS at_v44 \gset
\if :at_v44
:'history' = :'history_v44' AS at_v44,
:'history' = :'history_v45' AS at_v45 \gset
\if :at_v45
\elif :at_v44
\elif :at_v43
\elif :at_v42
\elif :at_v41
@@ -525,7 +528,8 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
coalesce(max(version), 0) = 41 AS at_v41,
coalesce(max(version), 0) = 42 AS at_v42,
coalesce(max(version), 0) = 43 AS at_v43,
coalesce(max(version), 0) = 44 AS at_v44
coalesce(max(version), 0) = 44 AS at_v44,
coalesce(max(version), 0) = 45 AS at_v45
FROM evolution.schema_migrations \gset
\if :at_v24
\ir 0025_cellular_v2_bundle.sql
@@ -738,6 +742,7 @@ SELECT coalesce(max(version), 0) = 24 AS at_v24,
\ir 0043_cellular_v9_foraging_bundle.sql
\elif :at_v43
\elif :at_v44
\elif :at_v45
\else
\echo 'Unknown or changed migration history after v43'
DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$;
@@ -746,6 +751,10 @@ SELECT coalesce(max(version), 0) = 43 AS needs_v44 FROM evolution.schema_migrati
\if :needs_v44
\ir 0044_exact_population_publication.sql
\endif
SELECT coalesce(max(version), 0) = 44 AS needs_v45 FROM evolution.schema_migrations \gset
\if :needs_v45
\ir 0045_owned_population_intervals.sql
\endif
-- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes
-- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC
-- grants or SET-able memberships would bypass the fenced functions. Checks the
@@ -765,6 +774,8 @@ BEGIN
UNION
SELECT m.member FROM pg_catalog.pg_auth_members m JOIN closure c ON m.roleid = c.oid
), fenced(oid) AS (VALUES
('evolution.admit_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64,uuid,bytea,evolution.u64)'::regprocedure::oid),
('evolution.load_population_interval(evolution.u64,evolution.u64,uuid,evolution.u64)'::regprocedure::oid),
('evolution.publish_population_checkpoint(bytea,bytea)'::regprocedure::oid),
('evolution.publish_population_profile_transition(bytea)'::regprocedure::oid),
('evolution.resolve_population_commit(evolution.u64,evolution.u64,uuid,bytea,bytea)'::regprocedure::oid),
+5
View File
@@ -29,6 +29,11 @@ every controlled cell has exactly its frozen-only intent. Recorded wildlife
intentions replay through the same biological kernel. The persisted policy must
reproduce the exact boundary, including sequence and every member.
ADR-0143 further restricts **new** interval preparation/publication to fully
suspended controlled cells. Linger remains vulnerable under the existing rules
and stays on profile 4. Earlier recorded frozen/lingering traces still decode and
replay unchanged for retained recovery; their wire format is not reinterpreted.
Kind 1 contains no biological trace, one boundary and a nonzero genome; begin and
end tick are equal and nonzero. Conversion moves the whole eligible wildlife
genome group. It cannot change biological bytes or select a random sample.
+34
View File
@@ -0,0 +1,34 @@
# Owned population interval journal 1
Authority: ADR-0143, PostgreSQL schema 45. Operational admission is separate from
the biological checkpoint. Each interval names one exact profile-5 source commit
and SHA-256, its begin/end ticks, controller version 1, and admitting holder/epoch.
It spans 1..64 unchanged fixed 50 ms steps. No timestamp grants biological time.
The immutable admissions table keys by source commit. Repeating the same source
and end returns its record; changing its digest, region or end conflicts. New
admission requires the current region lease and exact head. The pending pointer
has one `(world,region)` primary key and one unique source; looking up pending
work never scans a growing history. Runtime receives no direct table privileges.
Settlement appends the source-to-target commit record and removes the pending
pointer in the same transaction that publishes its exact checkpoint/trace.
Settlement's target foreign key is deferred until commit. Rollback restores the
pointer and leaves the interval pending. Retrying a committed publication or
settled admission never reopens its work. Manual conversion/profile exit cannot
skip a pending interval; private old publishers are revoked and a commit trigger
also refuses insertion while the pointer is unsettled.
A new holder loads and verifies the source pointer against the restored head,
retaining the original admission identity and controller version. It may finish
this interval with zero newly due wall time. Fresh pacing excludes that recovered
prefix; only fresh elapsed time funds new local debt. Unknown admission replies
retain the identical source/end request and pause readiness until reconciled.
Unknown checkpoint replies retain the complete prepared manifest/trace as before.
Current preparation requires all controlled lives fully suspended. Older recorded
population traces remain readable; no lineage, genome, funding, age, quantity,
archive or biological rule version changes. The worker reports durable pending
ticks separately from local tick debt and checkpoint age. Unadmitted wall time
remains paused across outages. Biological migration escrow is a separate journal
and is not supplied by this interval record.
@@ -71,6 +71,8 @@ transition(Postgres& db, const FilesystemObjectStore& objects, const LeaseGrant&
auto state = restore_cellular_payload(*source, *previous);
if (!state)
return std::unexpected(failure(PublicationError::predecessor));
if (!server::population_interval_safe(**state))
return std::unexpected(failure(PublicationError::candidate));
auto population = server::CellularPopulation::create(std::move(*state), policy);
if (!population || (*population)->state().snapshot() != *source)
return std::unexpected(failure(PublicationError::candidate));
@@ -160,6 +162,8 @@ std::expected<CommittedCheckpoint, PopulationPublicationFailure> publish_populat
auto base = restore_population_payload(*source, *previous);
if (!base)
return std::unexpected(failure(PublicationError::predecessor));
if (!batch.conversion && !server::population_interval_safe((*base)->state()))
return std::unexpected(failure(PublicationError::candidate));
auto replay = server::replay_population_batch(**base, batch);
const auto payload = candidate.snapshot();
if (!replay || (*replay)->snapshot() != payload)
@@ -0,0 +1,127 @@
#include "evo/persistence/population_interval_store.hpp"
#include <charconv>
#include <string_view>
namespace evo::persistence {
namespace {
LeaseStoreFailure failure(const PgFailure& cause) {
const std::string_view state(cause.sqlstate.data(), 5);
auto kind = LeaseStoreError::store;
if (cause.kind == PgError::unknown)
kind = LeaseStoreError::unknown;
else if (cause.kind == PgError::config || cause.kind == PgError::bounds || state == "22023")
kind = LeaseStoreError::invalid;
else if (state == "EVL02")
kind = LeaseStoreError::fenced;
else if (state == "EVL03")
kind = LeaseStoreError::head_mismatch;
else if (state == "EVL04")
kind = LeaseStoreError::not_provisioned;
else if (state == "EVL05")
kind = LeaseStoreError::exhausted;
else if (state == "EVL06")
kind = LeaseStoreError::unsupported;
else if (state == "EVL07" || state == "EVL18")
kind = LeaseStoreError::commit_conflict;
return {kind, cause};
}
std::string hex(std::span<const std::uint8_t> bytes) {
constexpr std::string_view digits = "0123456789abcdef";
std::string out;
for (const auto b : bytes) {
out += digits[b >> 4U];
out += digits[b & 15U];
}
return out;
}
std::string uuid(const Uuid& id) {
const auto h = hex(id);
return h.substr(0, 8) + '-' + h.substr(8, 4) + '-' + h.substr(12, 4) + '-' + h.substr(16, 4) +
'-' + h.substr(20);
}
template <std::size_t N> std::optional<std::array<std::uint8_t, N>> unhex(std::string_view value) {
if (value.size() != N * 2)
return std::nullopt;
std::array<std::uint8_t, N> out{};
for (std::size_t i = 0; i < N; ++i) {
unsigned byte{};
const auto [end, error] =
std::from_chars(value.data() + i * 2, value.data() + i * 2 + 2, byte, 16);
if (error != std::errc{} || end != value.data() + i * 2 + 2 || byte > 255)
return std::nullopt;
out[i] = static_cast<std::uint8_t>(byte);
}
return out;
}
std::optional<std::uint64_t> number(std::string_view value) {
std::uint64_t out{};
const auto [end, error] = std::from_chars(value.data(), value.data() + value.size(), out);
if (error != std::errc{} || end != value.data() + value.size())
return std::nullopt;
return out;
}
std::optional<PopulationInterval> parse(const PgRows::value_type& row) {
if (row.size() != 5)
return std::nullopt;
const auto text = row[0].value_or("");
if (text.size() != 36 || text[8] != '-' || text[13] != '-' || text[18] != '-' ||
text[23] != '-')
return std::nullopt;
std::string compact;
for (const auto c : text)
if (c != '-')
compact += c;
const auto source = unhex<16>(compact);
const auto digest = unhex<32>(row[1].value_or(""));
const auto begin = number(row[2].value_or("")), end = number(row[3].value_or(""));
if (!source || !digest || !begin || !end || *end <= *begin || *end - *begin > 64 ||
*source == Uuid{} || *digest == Sha256Digest{} || row[4].value_or("") != "1")
return std::nullopt;
return PopulationInterval{*source, *digest, *begin, *end, 1};
}
constexpr auto columns = "source_commit::text,encode(source_manifest_sha256,'hex'),begin_tick::"
"text,end_tick::text,controller_version::text";
} // namespace
std::expected<PopulationInterval, LeaseStoreFailure>
admit_population_interval(Postgres& db, RegionKey key, const LeaseGrant& lease,
const RegionHead& source, std::uint64_t end_tick,
std::chrono::milliseconds timeout) {
const std::array<std::string, 7> parameters{
std::to_string(key.world), std::to_string(key.region), uuid(lease.holder),
std::to_string(lease.epoch), uuid(source.commit), "\\x" + hex(source.digest),
std::to_string(end_tick)};
auto rows = db.query(std::string("SELECT ") + columns +
" FROM evolution.admit_population_interval("
"$1::evolution.u64,$2::evolution.u64,$3::uuid,$4::evolution.u64,$5::"
"uuid,$6::bytea,$7::evolution.u64)",
parameters, timeout);
if (!rows)
return std::unexpected(failure(rows.error()));
const auto interval = rows->size() == 1 ? parse(rows->front()) : std::nullopt;
if (!interval || interval->source_commit != source.commit ||
interval->source_manifest_sha256 != source.digest ||
interval->begin_tick != source.end_tick || interval->end_tick != end_tick)
return std::unexpected(LeaseStoreFailure{LeaseStoreError::unknown, {}});
return *interval;
}
std::expected<std::optional<PopulationInterval>, LeaseStoreFailure>
load_population_interval(Postgres& db, RegionKey key, const LeaseGrant& lease,
std::chrono::milliseconds timeout) {
const std::array<std::string, 4> parameters{std::to_string(key.world),
std::to_string(key.region), uuid(lease.holder),
std::to_string(lease.epoch)};
auto rows = db.query(std::string("SELECT ") + columns +
" FROM evolution.load_population_interval("
"$1::evolution.u64,$2::evolution.u64,$3::uuid,$4::evolution.u64)",
parameters, timeout);
if (!rows)
return std::unexpected(failure(rows.error()));
if (rows->empty())
return std::optional<PopulationInterval>{};
const auto interval = rows->size() == 1 ? parse(rows->front()) : std::nullopt;
if (!interval)
return std::unexpected(LeaseStoreFailure{LeaseStoreError::store, {}});
return interval;
}
} // namespace evo::persistence
+133 -8
View File
@@ -199,15 +199,19 @@ void CellularRegionHost::acquire() {
}
if (!recover_state(*lease) || !committed_.has_value() || !state_)
return;
interval_reload_ = population_mode();
admitted_interval_.reset();
interval_request_.reset();
interval_failure_.reset();
queued_mutations_.clear();
mutations_uncertain_ = true;
pending_endings_.clear();
// Controlled cells cannot die in unattended profile 5; their earlier raw
// endings were drained before entry. Do not invent a new death commit.
if (!population_mode())
for (const auto& corpse : state_->corpses())
if (corpse.parent_life == 0)
pending_endings_.push_back({committed_->head.commit, corpse.life_id});
// Reconcile retained owned deaths after either profile's recovery, including
// older profile-5 records that admitted lingering cells. Existing endings
// are idempotent; no ending is inferred for a living suspended cell.
for (const auto& corpse : state_->corpses())
if (corpse.parent_life == 0)
pending_endings_.push_back({committed_->head.commit, corpse.life_id});
status_.phase = HostPhase::running;
status_.simulated_tick = committed_->head.end_tick;
status_.committed_tick = committed_->head.end_tick;
@@ -391,6 +395,7 @@ void CellularRegionHost::resolve() {
note_endings(*state_, *pending_->candidate, retried->head.commit);
state_ = std::move(pending_->candidate);
committed_ = std::move(*retried);
finish_interval();
for (const auto& action : pending_->actions)
finalized_actions_.push_back({action, committed_->head.sequence});
finalize_births(pending_->births);
@@ -406,7 +411,11 @@ void CellularRegionHost::resolve() {
void CellularRegionHost::refresh() {
const auto& grant = authority_.grant();
status_.ready = status_.phase == HostPhase::running && state_ && committed_ && grant &&
authority_.may_originate(LeaseClock::now());
authority_.may_originate(LeaseClock::now()) && !interval_reload_ &&
!interval_request_.has_value();
const auto& interval = admitted_interval_;
status_.ecological_pending_ticks =
interval.has_value() ? interval->end_tick - interval->begin_tick : 0;
status_.epoch = grant ? grant->epoch : 0;
status_.lease_remaining_ms =
status_.ready ? static_cast<std::uint64_t>(std::chrono::duration_cast<milliseconds>(
@@ -424,6 +433,8 @@ const HostStatus& CellularRegionHost::step() {
resolve();
else if (!state_)
acquire();
if (!pending_ && state_ && status_.phase == HostPhase::running)
reconcile_interval();
if (!pending_ && state_ && mutations_uncertain_ && status_.phase == HostPhase::running)
reload_mutations();
if (!pending_ && state_ && status_.phase == HostPhase::running)
@@ -839,6 +850,9 @@ void CellularRegionHost::drain() {
authority_.fence();
state_.reset();
committed_.reset();
admitted_interval_.reset();
interval_request_.reset();
interval_reload_ = false;
status_.phase = HostPhase::stopped;
status_.epoch = 0;
status_.lease_remaining_ms = 0;
@@ -891,6 +905,7 @@ CellularRegionHost::accept_population_publication(
note_endings(*state_, *candidate, result->head.commit);
state_ = std::move(candidate);
committed_ = *result;
finish_interval();
++status_.commits;
status_.committed_sequence = committed_->head.sequence;
status_.committed_tick = status_.simulated_tick = committed_->head.end_tick;
@@ -899,7 +914,8 @@ CellularRegionHost::accept_population_publication(
}
std::expected<void, CellularPublishFailure> CellularRegionHost::set_population_mode(bool enabled) {
if (!status_.ready || pending_ || mutations_uncertain_ || !state_ || !committed_.has_value() ||
if (!status_.ready || pending_ || mutations_uncertain_ || admitted_interval_.has_value() ||
interval_request_.has_value() || interval_reload_ || !state_ || !committed_.has_value() ||
!authority_.may_begin_commit(LeaseClock::now()) || !database())
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
if (population_mode() == enabled)
@@ -908,6 +924,8 @@ std::expected<void, CellularPublishFailure> CellularRegionHost::set_population_m
(!config_.population_policy || !queued_mutations_.empty() || !pending_endings_.empty() ||
state_->rules_version() != cellular_lives_foraging_rules))
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
if (enabled && !population_interval_safe(*state_))
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
const auto policy = config_.population_policy.value_or(CellularTierPolicy{});
std::array<std::uint8_t, 10> marker{};
marker[0] = enabled ? 1 : 0;
@@ -971,4 +989,111 @@ CellularRegionHost::publish_population(const CellularBatch& batch) {
return std::unexpected(accepted.error());
return persistence::PublishedCellularBatch{std::move(*accepted), {}};
}
void CellularRegionHost::finish_interval() {
const auto& interval = admitted_interval_;
if (interval.has_value() && committed_.has_value() &&
committed_->head.end_tick == interval->end_tick)
admitted_interval_.reset();
}
void CellularRegionHost::reconcile_interval() {
if (!interval_reload_ && !interval_request_.has_value())
return;
const auto grant = authority_.grant();
if (!population_mode() || !committed_.has_value() || !grant.has_value() ||
!authority_.may_begin_commit(LeaseClock::now()) || !database())
return;
auto result = [&]() -> std::expected<std::optional<persistence::PopulationInterval>,
persistence::LeaseStoreFailure> {
const auto& request = interval_request_;
if (request.has_value()) {
auto admitted = persistence::admit_population_interval(
*db_, config_.key, *grant, request->source, request->end_tick,
config_.lease.commit_timeout);
if (!admitted)
return std::unexpected(admitted.error());
return std::optional{*admitted};
}
return persistence::load_population_interval(*db_, config_.key, *grant,
config_.lease.commit_timeout);
}();
if (!result) {
interval_failure_ = result.error();
if (result.error().kind == LeaseStoreError::unknown ||
result.error().kind == LeaseStoreError::store) {
if (result.error().kind == LeaseStoreError::unknown)
++status_.unknown_outcomes;
else
++status_.store_failures;
db_.reset();
return;
}
authority_.fence();
state_.reset();
committed_.reset();
admitted_interval_.reset();
interval_request_.reset();
status_.ready = false;
status_.phase = result.error().kind == LeaseStoreError::fenced ? HostPhase::acquiring
: HostPhase::quarantined;
++status_.fences;
return;
}
auto interval = *result;
if (interval.has_value() &&
(interval->source_commit != committed_->head.commit ||
interval->source_manifest_sha256 != committed_->head.digest ||
interval->begin_tick != committed_->head.end_tick || interval->controller_version != 1)) {
authority_.fence();
status_.ready = false;
status_.phase = HostPhase::quarantined;
return;
}
admitted_interval_ = interval;
interval_request_.reset();
interval_failure_.reset();
interval_reload_ = false;
}
std::expected<std::optional<persistence::CommittedCheckpoint>, CellularPublishFailure>
CellularRegionHost::advance_unattended(std::uint64_t due_ticks) {
if (!status_.ready || !population_mode() || !committed_.has_value() || pending_ ||
!authority_.may_begin_commit(LeaseClock::now()) || !population_interval_safe(*state_))
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
if (!admitted_interval_.has_value()) {
const auto remaining = ~std::uint64_t{0} - committed_->head.end_tick;
const auto count = std::min({due_ticks, remaining, std::uint64_t{64}});
if (count == 0 || (due_ticks < 20 && count != remaining))
return std::optional<persistence::CommittedCheckpoint>{};
interval_request_ = IntervalRequest{committed_->head, committed_->head.end_tick + count};
reconcile_interval();
refresh();
if (!status_.ready || !admitted_interval_.has_value())
return std::unexpected(
CellularPublishFailure{CellularHostError::publication, {}, interval_failure_, {}});
}
const auto interval = admitted_interval_;
if (!interval.has_value())
return std::unexpected(CellularPublishFailure{CellularHostError::not_ready, {}, {}, {}});
const CellularBatchHeader header{config_.key.world,
config_.key.region,
config_.frame,
interval->source_commit,
interval->source_manifest_sha256,
interval->begin_tick,
interval->end_tick};
const auto inputs = unattended_population_inputs(*state_, header);
if (!inputs) {
authority_.fence();
status_.ready = false;
status_.phase = HostPhase::quarantined;
return std::unexpected(
CellularPublishFailure{CellularHostError::invalid_batch, {}, {}, {}});
}
auto published = publish(*inputs);
if (!published)
return std::unexpected(published.error());
return std::optional{std::move(published->committed)};
}
} // namespace evo::server
@@ -74,6 +74,13 @@ class Reader {
};
} // namespace
bool population_interval_safe(const CellularLives& state) noexcept {
return state.rules_version() == cellular_lives_foraging_rules &&
std::ranges::all_of(state.lives(), [](const auto& life) {
return life.account == LifeAccount{} || life.presence == CellularPresence::suspended;
});
}
std::expected<CellularBatch, CellularPopulationError>
unattended_population_inputs(const CellularLives& base, const CellularBatchHeader& header) {
if (header.world != base.world() || header.region != base.region() ||
@@ -178,6 +185,8 @@ execute_population_batch(const CellularPopulation& base, CellularPopulationBatch
std::expected<CellularPopulationExecution, CellularPopulationError>
prepare_population_batch(const CellularPopulation& base, CellularBatch biology) {
if (!population_interval_safe(base.state()))
return std::unexpected(CellularPopulationError::ineligible);
CellularPopulationBatch batch{biology.header, std::move(biology), {}, {}};
auto state = execute_population_batch(base, batch, true);
if (!state)
+12 -4
View File
@@ -16,7 +16,11 @@ bool cellular_unattended_tick_needed(const CellularLives& state) noexcept {
std::expected<CellularScheduleStatus, CellularScheduleError>
CellularTickSchedule::observe(std::uint64_t now_us, std::uint64_t epoch,
std::uint64_t committed_tick, bool needed, bool authoritative) {
std::uint64_t committed_tick, bool needed, bool authoritative,
std::uint64_t previously_admitted_ticks) {
if (previously_admitted_ticks > 64 ||
previously_admitted_ticks > std::numeric_limits<std::uint64_t>::max() - committed_tick)
return std::unexpected(CellularScheduleError::admission);
if (observed_ && now_us < last_us_)
return std::unexpected(CellularScheduleError::clock_regression);
if (authoritative && epoch == 0)
@@ -35,27 +39,30 @@ CellularTickSchedule::observe(std::uint64_t now_us, std::uint64_t epoch,
was_authoritative_ = authoritative;
status_.head_age_us = have_receipt_ ? now_us - receipt_us_ : 0;
status_.due = false;
status_.retry_ready = now_us >= retry_us_;
status_.debt = 0;
if (!authoritative || !needed) {
anchored_ = false;
retry_us_ = 0;
status_.retry_ready = true;
return status_;
}
if (!anchored_ || epoch != epoch_) {
anchored_ = true;
origin_us_ = now_us;
origin_tick_ = committed_tick;
origin_tick_ = committed_tick + previously_admitted_ticks;
epoch_ = epoch;
retry_us_ = 0;
}
head_ = committed_tick;
const auto elapsed_ticks = (now_us - origin_us_) / biology::cellular_motion_tick_microseconds;
const auto completed_ticks = committed_tick - origin_tick_;
const auto completed_ticks = committed_tick > origin_tick_ ? committed_tick - origin_tick_ : 0;
const auto remaining = std::numeric_limits<std::uint64_t>::max() - committed_tick;
status_.debt =
elapsed_ticks > completed_ticks ? std::min(elapsed_ticks - completed_ticks, remaining) : 0;
status_.max_debt = std::max(status_.max_debt, status_.debt);
status_.due = status_.debt > 0 && now_us >= retry_us_;
status_.retry_ready = now_us >= retry_us_;
status_.due = status_.debt > 0 && status_.retry_ready;
return status_;
}
void CellularTickSchedule::defer(std::uint64_t now_us) noexcept {
@@ -64,6 +71,7 @@ void CellularTickSchedule::defer(std::uint64_t now_us) noexcept {
? maximum
: now_us + biology::cellular_motion_tick_microseconds;
status_.due = false;
status_.retry_ready = false;
}
CellularScheduleStatus CellularTickSchedule::status() const noexcept {
return status_;
+2
View File
@@ -72,6 +72,8 @@ std::string render_metrics(const HostStatus& s, const OpsCounters& c) {
metric(out, "evo_region_checkpoint_age_milliseconds", "gauge",
"Time since the last confirmed commit or recovery", c.checkpoint_age_ms);
metric(out, "evo_region_tick_debt", "gauge", "Due ticks not yet simulated", s.tick_debt);
metric(out, "evo_region_ecological_pending_ticks", "gauge", "Durable admitted ecological ticks",
s.ecological_pending_ticks);
metric(out, "evo_region_tick_debt_max", "gauge", "Largest tick debt since recovery",
s.max_tick_debt);
metric(out, "evo_region_queued_inputs", "gauge", "Admitted inputs awaiting a tick", s.queued);
+16 -2
View File
@@ -20,9 +20,9 @@ void pacing() {
require(sample.head_age_us == 500'000, "stalled head appeared fresh");
schedule.defer(500'000);
sample = checked(schedule.observe(549'999, 1, 100, true, true), "bounded retry");
require(!sample.due && sample.debt == 10, "retry consumed debt");
require(!sample.due && !sample.retry_ready && sample.debt == 10, "retry consumed debt");
sample = checked(schedule.observe(550'000, 1, 100, true, true), "retry due");
require(sample.due && sample.debt == 11, "retry lost ticks");
require(sample.due && sample.retry_ready && sample.debt == 11, "retry lost ticks");
for (std::uint64_t tick = 101; tick <= 111; ++tick) {
sample = checked(schedule.observe(550'000, 1, tick, true, true), "committed catchup");
require(sample.debt == 111 - tick, "confirmed head did not consume exact debt");
@@ -87,9 +87,23 @@ void historical_profile_is_not_opted_in() {
require(!state->lives().empty() && !cellular_unattended_tick_needed(*state),
"legacy profile without offline freeze was enabled");
}
void admitted_recovery_does_not_spend_new_wall_time() {
CellularTickSchedule schedule;
auto sample =
checked(schedule.observe(0, 2, 100, true, true, 64), "recovered admission anchor");
require(!sample.due && sample.retry_ready && sample.debt == 0, "recovery invented fresh debt");
sample = checked(schedule.observe(10'000, 2, 164, true, true), "recovered interval settled");
require(!sample.due && sample.debt == 0, "recorded prefix used new wall time");
sample = checked(schedule.observe(50'000, 2, 164, true, true), "fresh post-recovery time");
require(sample.due && sample.debt == 1, "recorded prefix delayed fresh work");
require(!schedule.observe(50'000, 2, 164, true, true, 65), "oversized admission accepted");
require(!schedule.observe(50'000, 2, ~std::uint64_t{0}, true, true, 1),
"admission wrapped tick");
}
} // namespace
int main() try {
pacing();
admitted_recovery_does_not_spend_new_wall_time();
offline_demand();
historical_profile_is_not_opted_in();
return 0;
+4 -2
View File
@@ -71,6 +71,7 @@ void exposition_is_bounded_and_well_formed() {
s.simulated_tick = 50;
s.committed_tick = 41;
s.lease_remaining_ms = 1234;
s.ecological_pending_ticks = 64;
OpsCounters c;
c.queue_capacity_inputs = 64;
c.queue_capacity_bytes = 65536;
@@ -91,10 +92,11 @@ void exposition_is_bounded_and_well_formed() {
CHECK(text.find("evo_region_phase{phase=\"running\"} 1\n") != std::string::npos);
CHECK(text.find("evo_region_ready 1\n") != std::string::npos);
CHECK(text.find("evo_region_commit_lag_ticks 9\n") != std::string::npos);
CHECK(text.find("evo_region_ecological_pending_ticks 64\n") != std::string::npos);
CHECK(text.find("evo_region_lease_remaining_milliseconds 1234\n") != std::string::npos);
CHECK(text.find("evo_region_dropped_total{reason=\"fenced\"} 5\n") != std::string::npos);
// Series count is fixed: 6 phases + 4 reasons + 22 unlabelled families.
CHECK(samples.size() == 6 + 4 + 22);
// Series count is fixed: 6 phases + 4 reasons + 23 unlabelled families.
CHECK(samples.size() == 6 + 4 + 23);
// Worst-case values keep the size bounded and never underflow the lag.
HostStatus big;
+40 -3
View File
@@ -17,6 +17,42 @@ CellularBatchHeader header(const CellularPopulation& state, std::uint64_t ticks)
return {s.world(), s.region(), s.frame(), filled<16>(1),
filled<32>(2), s.tick(), s.tick() + ticks};
}
Owner interval_source() {
auto source = cellular_descendants();
const auto stop = source->tick() + cellular_linger_ticks + 1;
while (source->tick() < stop) {
const CellularBatchHeader h{source->world(),
source->region(),
source->frame(),
filled<16>(1),
filled<32>(2),
source->tick(),
std::min(stop, source->tick() + 64)};
auto inputs = checked(unattended_population_inputs(*source, h), "suspension prefix inputs");
source = checked(replay_cellular_batch(*source, inputs), "suspension prefix replay");
}
require(population_interval_safe(*source), "controlled life did not finish lingering");
return source;
}
void lingering_is_not_new_population_work() {
auto active =
checked(CellularPopulation::create(cellular_descendants()), "legacy active source");
require(!population_interval_safe(active->state()), "active controlled source admitted");
auto biology = checked(unattended_population_inputs(active->state(), header(*active, 1)),
"legacy frozen tick");
require(!prepare_population_batch(*active, biology),
"new interval bypassed vulnerable lingering");
auto biological =
checked(replay_cellular_batch(active->state(), biology), "legacy recorded biology");
require(!population_interval_safe(*biological), "lingering controlled source admitted");
auto expected =
checked(CellularPopulation::create(std::move(biological)), "legacy population result");
// Existing profile-5 records retain their meaning. They can be verified and
// materialized to profile 4; new interval preparation rejects this state.
const CellularPopulationBatch legacy{biology.header, biology, {}, {{0, {}}}};
auto replayed = checked(replay_population_batch(*active, legacy), "retained legacy trace");
require(replayed->snapshot() == expected->snapshot(), "old recorded prefix changed meaning");
}
CellularBatch inputs_batch(const CellularPopulation& base) {
CellularBatch batch{header(base, 64), {}};
auto biological = restore(base.state());
@@ -37,8 +73,8 @@ CellularBatch inputs_batch(const CellularPopulation& base) {
return batch;
}
void traces() {
auto population = checked(
CellularPopulation::create(cellular_descendants(), {true, 1200, 1800}), "batch population");
auto population = checked(CellularPopulation::create(interval_source(), {true, 1200, 1800}),
"batch population");
std::uint64_t converted = 0;
for (unsigned interval = 0; interval < 6; ++interval) {
auto inputs = inputs_batch(*population);
@@ -83,7 +119,7 @@ void traces() {
"manual materialization changed biology");
}
void manifests() {
auto state = cellular_descendants();
auto state = interval_source();
auto population = checked(CellularPopulation::create(restore(*state), {true, 1200, 1800}),
"manifest population");
CheckpointManifest source;
@@ -151,6 +187,7 @@ void manifests() {
} // namespace
int main() try {
traces();
lingering_is_not_new_population_work();
manifests();
return 0;
} catch (const std::exception& error) {
+163 -3
View File
@@ -1,5 +1,6 @@
#include "evo/persistence/account_control_store.hpp"
#include "evo/persistence/population_checkpoint_store.hpp"
#include "evo/persistence/population_interval_store.hpp"
#include "evo/persistence/sha256.hpp"
#include "evo/server/cellular_region_host.hpp"
#include "fixtures/cellular_ecology.hpp"
@@ -157,6 +158,28 @@ void funded_source(Fixture& f) {
require(std::ranges::any_of(f.raw->lives(),
[](const auto& life) { return life.account == LifeAccount{}; }),
"durable fixture has no funded player-derived wildlife");
require(!migrate_cellular_to_population(*f.db, *f.objects, f.lease, f.head, f.id(), f.build,
{true, 1200, 1800}, 5s),
"population entry bypassed vulnerable controlled life");
const auto suspended_at = f.raw->tick() + cellular_linger_ticks + 1;
while (f.raw->tick() < suspended_at) {
auto h = f.header(std::min(std::uint64_t{64}, suspended_at - f.raw->tick()));
auto prefix = take(unattended_population_inputs(*f.raw, h), "raw linger inputs");
f.raw = take(replay_cellular_batch(*f.raw, prefix), "raw linger biology");
f.head = take(publish_cellular_batch(*f.db, *f.objects, f.lease, f.head, prefix, *f.raw,
f.id(), f.build, 5s),
"raw linger publication")
.committed;
std::vector<std::uint64_t> endings;
for (const auto& corpse : f.raw->corpses())
if (corpse.parent_life == 0)
endings.push_back(corpse.life_id);
if (!endings.empty())
take(end_cellular_lives(*f.db, f.key, f.lease, f.head.head.commit, endings, 5s),
"raw linger endings");
}
require(population_interval_safe(*f.raw),
"population source retained a vulnerable controlled life");
}
CellularPopulationExecution unattended(Fixture& f) {
auto working = restore(f.population->state());
@@ -265,6 +288,9 @@ void conversions_and_retries(Fixture& f) {
}
void process_loss(Fixture& f, int point) {
auto next = unattended(f);
take(admit_population_interval(*f.db, f.key, f.lease, f.head.head, next.batch.header.end_tick,
5s),
"admit process-loss interval");
const auto predecessor = f.head.head;
const auto expected = next.state->snapshot();
const auto commit = f.id();
@@ -306,6 +332,9 @@ void process_loss(Fixture& f, int point) {
require((point == 2 && acknowledged == 1 && ack == 'A') || (point != 2 && acknowledged == 0),
"unexpected permanent acknowledgement");
recover(f);
const auto pending =
take(load_population_interval(*f.db, f.key, f.lease, 5s), "recover process interval");
require(pending.has_value() == (point == 0), "process loss changed interval settlement");
if (point == 0)
require(f.head.head == predecessor, "uncommitted process published a head");
else
@@ -314,6 +343,9 @@ void process_loss(Fixture& f, int point) {
}
void ambiguous_publication(Fixture& f, bool terminate) {
auto next = unattended(f);
take(admit_population_interval(*f.db, f.key, f.lease, f.head.head, next.batch.header.end_tick,
5s),
"admit ambiguous interval");
const auto expected = next.state->snapshot();
auto locker = connection(true);
take(locker->query("BEGIN", {}, 5s), "population lease row transaction");
@@ -352,6 +384,73 @@ void ambiguous_publication(Fixture& f, bool terminate) {
recover(f);
require(f.population->snapshot() == expected, "ambiguous publication changed retained sample");
}
void interval_contract(Fixture& f) {
require(!take(load_population_interval(*f.db, f.key, f.lease, 5s), "empty interval journal")
.has_value(),
"settled source retained pending work");
auto next = unattended(f);
const auto source = f.head.head;
const auto admitted =
take(admit_population_interval(*f.db, f.key, f.lease, source, source.end_tick + 64, 5s),
"durable interval admission");
require(take(admit_population_interval(*f.db, f.key, f.lease, source, source.end_tick + 64, 5s),
"duplicate interval admission") == admitted,
"admission retry changed boundary");
auto changed =
admit_population_interval(*f.db, f.key, f.lease, source, source.end_tick + 63, 5s);
require(!changed && changed.error().kind == LeaseStoreError::commit_conflict,
"source admitted two different intervals");
const auto expected = next.state->snapshot();
require(!f.population->members().empty(), "interval fixture lost its exact cohort");
const auto genome = f.population->members().front().genome;
auto conversion =
take(prepare_population_conversion(*f.population, f.header(0), {genome, false}),
"conversion during admitted work");
require(!publish_population_batch(*f.db, *f.objects, f.lease, f.head, conversion.batch,
*conversion.state, f.id(), f.build, 5s),
"conversion skipped an admitted interval");
require(
!materialize_population_profile(*f.db, *f.objects, f.lease, f.head, f.id(), f.build, 5s),
"profile exit skipped an admitted interval");
auto shortened_header = f.header(63);
auto shortened =
take(prepare_population_batch(
*f.population,
take(unattended_population_inputs(f.population->state(), shortened_header),
"shortened input")),
"shortened interval");
require(!publish_population_batch(*f.db, *f.objects, f.lease, f.head, shortened.batch,
*shortened.state, f.id(), f.build, 5s),
"publication partially consumed an interval");
take(release_region_lease(*f.db, f.key, f.lease, 5s), "release interval owner");
const auto epoch = f.lease.epoch;
f.lease =
take(acquire_region_lease(*f.db, f.key, f.build, 60000ms, 5s), "replace interval owner");
require(f.lease.epoch > epoch && take(take(load_population_interval(*f.db, f.key, f.lease, 5s),
"new owner interval load"),
"retained admission") == admitted,
"replacement owner lost admitted work");
f.head = take(publish_population_batch(*f.db, *f.objects, f.lease, f.head, next.batch,
*next.state, f.id(), f.build, 5s),
"settle exact admitted interval");
recover(f);
require(f.population->snapshot() == expected &&
!take(load_population_interval(*f.db, f.key, f.lease, 5s), "settled interval load")
.has_value(),
"settlement lost work or retained spendable pending state");
require(take(admit_population_interval(*f.db, f.key, f.lease, source, source.end_tick + 64, 5s),
"settled admission retry") == admitted &&
!take(load_population_interval(*f.db, f.key, f.lease, 5s), "settled retry pending")
.has_value(),
"old admission retry reopened settled work");
require(!admit_population_interval(*f.db, f.key, f.lease, f.head.head,
f.head.head.end_tick + 65, 5s),
"oversized interval admitted");
const auto denied = f.db->query(
"SELECT * FROM evolution.publish_population_checkpoint_v44(''::bytea,''::bytea)", {}, 5s);
require(!denied && std::string(denied.error().sqlstate.data(), 5) == "42501",
"runtime bypassed interval guard");
}
void isolation_and_reverse(Fixture& f) {
auto interval = unattended(f);
const auto old_lease = f.lease;
@@ -409,6 +508,7 @@ void host_ownership_and_admission(Fixture& f) {
max_cellular_population_bytes),
"host objects");
CellularHostConfig config{f.key, 1, {}, f.build, {f.build}};
config.lease.commit_timeout = 300ms;
if (enabled)
config.population_policy = CellularTierPolicy{true, 1200, 1800};
return take(CellularRegionHost::create(
@@ -448,24 +548,83 @@ void host_ownership_and_admission(Fixture& f) {
// The fixture's old envelope may have a different membership history; biology is shared.
auto expected =
take(prepare_population_batch(*population, next.batch.biology), "host reference batch");
take(host->publish(next.batch.biology), "host population publication");
require(!take(host->advance_unattended(19), "collect bounded publication window").has_value() &&
host->state()->snapshot() == before,
"small local debt was admitted prematurely");
const auto published = take(host->advance_unattended(65), "host population publication");
require(published.has_value(), "host skipped admitted interval");
require(host->population_mode() &&
host->state()->snapshot() == expected.state->state().snapshot(),
"host population continuation differs");
const auto committed = take(host->committed(), "host continued head");
const auto continued = host->state()->snapshot();
const CellularBatchHeader recovered_header{f.key.world,
f.key.region,
1,
committed.head.commit,
committed.head.digest,
committed.head.end_tick,
committed.head.end_tick + 64};
auto recovered_expected = take(
prepare_population_batch(
*expected.state, take(unattended_population_inputs(*host->state(), recovered_header),
"recovered interval inputs")),
"recovered interval reference");
take(admit_population_interval(*f.db, f.key, take(host->authority(), "host interval authority"),
committed.head, committed.head.end_tick + 64, 5s),
"admit before host loss");
host->drain();
host = make_host(false);
require(host->step().ready && host->population_mode() &&
take(host->committed(), "host recovered head").head == committed.head &&
host->state()->snapshot() == continued,
"restarted host failed to recover population head");
require(host->status().ecological_pending_ticks == 64 && !host->set_population_mode(false),
"restarted host hid or skipped admitted work");
require(take(host->advance_unattended(0), "resume admitted work with zero new due time")
.has_value() &&
host->state()->snapshot() == recovered_expected.state->state().snapshot() &&
host->status().ecological_pending_ticks == 0,
"recovered interval did not settle exactly");
const auto recovered_head = take(host->committed(), "recovered interval head");
const auto recovered_bytes = host->state()->snapshot();
const CellularBatchHeader unknown_header{f.key.world,
f.key.region,
1,
recovered_head.head.commit,
recovered_head.head.digest,
recovered_head.head.end_tick,
recovered_head.head.end_tick + 64};
auto unknown_expected = take(
prepare_population_batch(*recovered_expected.state,
take(unattended_population_inputs(*host->state(), unknown_header),
"uncertain admission inputs")),
"uncertain admission reference");
auto locker = connection(true);
take(locker->query("BEGIN", {}, 5s), "host admission lock transaction");
take(
locker->query(
"SELECT 1 FROM evolution.region_leases WHERE world_id=30201 AND region_id=1 FOR UPDATE",
{}, 5s),
"hold host admission reply");
require(!host->advance_unattended(64) && !host->status().ready &&
host->state()->snapshot() == recovered_bytes,
"uncertain admission allowed simulation/output");
take(locker->query("ROLLBACK", {}, 5s), "release host admission reply");
for (unsigned attempt = 0; attempt < 200 && !host->step().ready; ++attempt)
std::this_thread::sleep_for(25ms);
require(host->status().ready && host->status().ecological_pending_ticks == 64,
"unknown admission failed exact reconciliation");
require(take(host->advance_unattended(0), "finish reconciled admission").has_value() &&
host->state()->snapshot() == unknown_expected.state->state().snapshot(),
"admission retry chose different work");
const auto final_bytes = host->state()->snapshot();
Uuid account{};
account.back() = 1;
// A returning player's operation completes the durable reverse transition before lookup.
[[maybe_unused]] const auto queried = host->query_primitive_life(account, 1, 1);
require(!host->population_mode() && host->state()->snapshot() == continued &&
host->state()->tick() == tick + 64,
require(!host->population_mode() && host->state()->snapshot() == final_bytes &&
host->state()->tick() == tick + 192,
"player operation did not materialize first");
host->drain();
}
@@ -474,6 +633,7 @@ void run_population_publication_tests() {
Fixture fixture;
funded_source(fixture);
conversions_and_retries(fixture);
interval_contract(fixture);
ambiguous_publication(fixture, false);
ambiguous_publication(fixture, true);
process_loss(fixture, 0);
+2 -1
View File
@@ -210,6 +210,7 @@ void upgrade_across_migration_and_barrier() {
" -v migration_sha_42=$(s 0042_cellular_v8_reproduction_bundle.sql)"
" -v migration_sha_43=$(s 0043_cellular_v9_foraging_bundle.sql)"
" -v migration_sha_44=$(s 0044_exact_population_publication.sql)"
" -v migration_sha_45=$(s 0045_owned_population_intervals.sql)"
" -v runtime_role=evo_runtime_test -f migrations/apply.sql"
" > /dev/null 2>&1") != 0);
CHECK(history(*owner_db) == "1,2,3,4,5,6,7");
@@ -226,7 +227,7 @@ void upgrade_across_migration_and_barrier() {
" EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/"
"tools/migrate-postgres.sh\" > /dev/null") == 0);
CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,"
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44");
"25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45");
CHECK(scalar(*owner_db, "SELECT (c.condeferrable AND c.condeferred AND "
"c.confrelid = 'evolution.region_commits'::regclass)::text "
"FROM pg_catalog.pg_constraint c WHERE c.conrelid = "
+5 -5
View File
@@ -129,7 +129,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha"
-f "$root/migrations/0001_region_bootstrap.sql" > /dev/null
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1
PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44
test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45
# v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2.
v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")"
v2_sha="${v2_sha%% *}"
@@ -150,7 +150,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null
PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 44:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 45:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45
test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1
# Review P1: a v4 database holding a published (possibly terminal) checkpoint has
# no durable decisions for it. The v5 upgrade must refuse before changing anything.
@@ -204,7 +204,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \
-c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \
-c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null
PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh"
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1
test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick
phase permission-audit
@@ -257,9 +257,9 @@ expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control
bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again.
phase checksum-drift
# Unknown future versions and checksum drift must not be repaired silently.
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=45 WHERE version=44'
psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=46 WHERE version=45'
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=44 WHERE version=45"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=45 WHERE version=46"
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14"
if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi
psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13"
+3
View File
@@ -100,6 +100,8 @@ digest_43="$(sha256sum "$root/migrations/0043_cellular_v9_foraging_bundle.sql")"
digest_43="${digest_43%% *}"
digest_44="$(sha256sum "$root/migrations/0044_exact_population_publication.sql")"
digest_44="${digest_44%% *}"
digest_45="$(sha256sum "$root/migrations/0045_owned_population_intervals.sql")"
digest_45="${digest_45%% *}"
psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
--set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \
--set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \
@@ -137,5 +139,6 @@ psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \
--set="migration_sha_42=$digest_42" \
--set="migration_sha_43=$digest_43" \
--set="migration_sha_44=$digest_44" \
--set="migration_sha_45=$digest_45" \
--set="runtime_role=$runtime_role" \
--file="$root/migrations/apply.sql"