M12.3: publish and recover durable exact populations
Foundation / checks (ecs) (push) Successful in 2m41s
Foundation / checks (tsan) (push) Failing after 18m12s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (network) (push) Canceled after 20m41s
Foundation / checks (postgres) (push) Canceled after 20m43s
Foundation / checks (checks) (push) Canceled after 20m49s
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (client) (push) Failing after 17m4s
Foundation / postgres-tools (push) Canceled after 0s

This commit is contained in:
dingo
2026-10-17 17:10:17 -04:00
parent d5d10a0cc7
commit 627411983f
35 changed files with 2637 additions and 102 deletions
+15
View File
@@ -1,5 +1,20 @@
# Project status and next steps
## M12.3b durable exact populations - 2026-10-17
ADR-0142 and schema 44 add dedicated population profile 5, trace 1, immutable
publication/transition evidence and exact source/target recovery. The owning
cellular host recovers either representation, materializes before player
operations and serializes mutation reservations against profile entry. The
optional `--cellular-populations on` path batches unattended work; entry is off
by default. No biological version, golden, deployment or live region changed.
[Scope and evidence](engineering/M12_DURABLE_POPULATIONS.md) distinguish portable
contracts, real PostgreSQL ambiguity/process-loss tests and host recreation from
remaining executable/network fault drills and target-host qualification. Durable
interval admission/backlog is the next M12.4 segment. The dependent ecological,
multi-day and returning-player gates remain open; M11.9 stays closed for now.
## M12.4a current-worker unattended scheduling - 2026-10-17
The cellular worker now keeps ticking for funded wildlife after players suspend,
@@ -0,0 +1,86 @@
# ADR-0142: Durable exact population profile and publication
## Status
Accepted for M12's opt-in engineering implementation, 2026-10-17. Extends
ADR-0139/0140 with a dedicated durable profile. Production enabling and target-host
qualification remain separate; no deployment or World0 operation is authorized.
## Context
The exact cohort envelope carries membership and policy beyond raw cellular
bytes. Reusing profile 4 would make retained manifests ambiguous. Portable
restoration alone cannot prove atomic publication, current authority or recovery
after a lost database reply.
## Decision
Checkpoint profile 5 contains population envelope 1 and nested cellular bundle 9,
with history profile 5 identifying exact retained membership. Profile 4 retains
its raw cellular bytes and meaning. No biological version or golden is replaced.
Use explicit fenced, no-tick profile transitions: rules-8 profile 4 to population
5 wraps the same biology with empty membership, sequence zero and a persisted
policy; population 5 to profile 4 materializes the retained individuals by
unwrapping the identical biology. Both require the exact current predecessor,
current region fence, retained build and immutable transition evidence. Recovery
checks the complete source/target pair, not just their aggregate inventory.
Conversion sequence starts at zero for each entry generation; the region commit
sequence remains monotonic and the transition records identify every generation.
Initially profile 5 admits unattended ecological batches only: controlled cells
receive frozen intentions; player births, mutation selections and session action
receipts remain profile-4 operations. A host must finish the reverse transition
before processing those operations. This restriction avoids bypassing their
existing account-control/reservation transactions. It does not freeze wildlife
or erase controlled/suspended stock. An explicit profile selector is opt-in.
The cellular worker accepts `--cellular-populations on|off` (default off). It may
enter only for unattended rules-8 wildlife, with no waiting player birth or
mutation. Session demand or a player operation completes the reverse transition
first. SQL serializes profile entry and mutation reservation under the same
region lease: entry refuses outstanding choices, and new choices require profile
4 in the current host. Historical reservation contracts remain unchanged outside
profile 5. An existing profile-5 head remains recoverable with entry disabled.
The worker collects at least 20 due ticks (one second) and services at most 64
per unattended publication, consuming local debt only on confirmation. This is
bounded publication batching; durable interval admission/restart backlog is the
next M12.4 segment, not supplied by a wall-clock inference on recovery.
Population batch format 1 records the unchanged canonical cellular intentions
and an exact membership/sequence boundary for each tick. Biological replay reads
the recorded intentions, never reruns the wildlife controller. It verifies each
recorded boundary against the persisted deterministic policy. Manual conversion
records operate on an entire wildlife genome group at the same biological tick.
No random materialization sample exists in this exact model. Restoration installs
the recorded checkpoint; validation cannot resample a different member set.
Store immutable payload and trace objects with SHA-256/read-back verification,
then commit the manifest and publication evidence in one database transaction
under the existing serialized lease/head guard. Unknown replies retain the exact
prepared request; read-only resolution never grants a stale writer permission to
continue. Dedicated evidence triggers prohibit generic publication bypasses.
Retain all old formats, histories and source checkpoints. Do not prune objects.
## Validation and limits
Require malformed/profile-confusion rejection, whole-state source/target equality,
conversion oscillation, trace/payload mismatch refusal, frozen controlled biology,
duplicate/conflicting retries, expired writers and missing/corrupt objects. Exercise
real PostgreSQL and process death before publication and after commit/receipt.
Portable codec success alone does not close the durable gate.
This profile supplies a place for bounded unattended intervals and later escrow;
it does not itself implement biological migration or environmental coupling.
Existing player actions, NPC heredity and fixed 50 ms biology retain their rules.
## Alternatives, costs and consequences
Relabeling the raw profile breaks retained recovery; persisting only aggregate
stock loses exact members. Reconstructing policy on restart can choose a different
representation. Keep explicit profiles and recorded boundaries instead.
This retains every member and full trace, so it adds durable bytes and replay
work and makes no compression claim. SQL parsers, grants and source/target
verification must evolve together. Target-host cost and executable/network fault
qualification remain risks before production enablement. All old objects and
histories remain retained; no garbage collection or compatibility reset follows.
+9
View File
@@ -31,6 +31,15 @@ the narrower environmental profile. M7.1 completes the audit, not those codecs.
## Identity, authority and watermarks
ADR-0142 adds opt-in exact population profile 5 (envelope 1, nested cellular
bundle 9, history profile 5). Raw cellular profile 4 is unchanged. Fenced no-tick
transitions retain complete biology and source/target digests; normal population
publication binds recorded intentions and exact membership boundaries. Generic
publishers cannot bypass dedicated immutable evidence. The host has one owning
state, materializes before player operations and never treats read-only commit
resolution as current authority. See [the contract](../../schemas/cellular-population-batch.md)
and [qualification limits](../engineering/M12_DURABLE_POPULATIONS.md).
World/region/frame are nonzero durable IDs. No ID is a Pod name. The host records
its unique holder ID and monotonically increasing nonzero lease epoch. Exhaustion
of IDs, epochs, commit sequences or ticks fails closed; never wrap/reuse them.
+1 -1
View File
@@ -10,7 +10,7 @@ Commit and push each validated scoped segment separately.
|---|---|---|
| 12.1 | Engineering evidence policy and explicit deferrals documented | ADR-0133, identity/time contract and worked histories; no new game rules or runtime versions. |
| 12.2 | Exact local model selected and paired qualification implemented (12.2d); production cost gate open | ADR-0139, exact canonical equality across 196,608 paired ticks per platform. Regional approximation disabled; no compression claim. Deployment-host capacity and fuller interaction coverage remain open. |
| 12.3 | Portable exact conversion closure, automatic policy and process recovery implemented | ADR-0140, population envelope 1. Dedicated PostgreSQL profile/trace/publication adapter and fault injection remain required before enabling the live selector. |
| 12.3 | Exact population profile/trace/store and opt-in host integration implemented (12.3b) | ADR-0142, schema 44; exact publication/recovery, mutation race guards, ambiguous replies and process kills. Executable/network population fault drills and target-host qualification remain open. |
| 12.4 | Current-worker unattended demand and confirmed-head pacing implemented (12.4a) | ADR-0141. Population durable intervals and biological migration escrow remain unimplemented. |
| 12.5 | Blocked on 12.2-12.4 | Finite M6 process fixtures do not qualify coupled persistent cellular/cohort atmosphere/water budgets. |
| 12.6 | Existing local archive retained; integrated population history open | World-level durable event/query/client path remains required. |
@@ -0,0 +1,68 @@
# M12.3b durable exact populations
ADR-0142 implements the persistence boundary identified in the remaining-gate
audit. This is an opt-in engineering path; no live region was migrated or enabled.
The optional cellular worker accepts `--cellular-populations on|off`, default off.
When unattended rules-8 wildlife is present, entry can wrap its exact biological
state as profile 5. A returning player, queued birth or mutation requires the
durable reverse transition before processing player operations.
## State and publication
There is exactly one owning biological store in the host. Profile 4 owns raw
CellularLives; profile 5 owns a CellularPopulation containing that store and its
exact membership tags. Common callers receive only a const biological view.
Entering, leaving and oscillating cannot heal, move, reproduce or fund a life.
Every profile-5 generation starts membership sequence zero, while the immutable
region publication sequence and source/target evidence preserve its identity.
Migration 44 retains all old parsers, histories and raw cellular formats. It adds
dedicated immutable population publication and profile-transition records, fenced
publish/resolve/load functions and generic-publication bypass guards. Entry and
new mutation reservations serialize on the same region lease. Entry refuses
outstanding player choices; profile 5 refuses new choices. Existing unrelated
reservation contracts retain their original behavior.
Payloads and traces are immutable content-addressed objects. The publisher checks
their SHA-256 and complete read-back before the atomic SQL head/evidence update.
Recovery verifies the exact predecessor and full source/target bytes. Biological
traces record intentions and exact membership boundaries; replay does not rerun
the wildlife controller. Unknown replies stop publication until exact-byte
resolution/retry succeeds under current authority.
## Evidence and qualification limits
The portable contract covers canonical codec/manifest/profile confusion, sampled
truncations, appended bytes, membership/sequence tampering, controlled-input
refusal, manual conversions and full biological replay equality. Native Release
tests run under NDEBUG; sanitizer and static-analysis evidence is recorded in
[VALIDATION](VALIDATION.md).
The private PostgreSQL fixture creates real reserved founders and waits for a
funded player-derived descendant. It exercises durable membership oscillation,
fresh-connection recovery, reverse materialization, generic bypass rejection,
mutation/profile race guards, duplicates/conflicts and stale authority. Missing
and corrupt payloads refuse recovery. Lease-row contention creates real adapter
timeouts: one backend commits after its client disconnects; another is terminated
before commit. Both resolve/retry the retained exact request.
SIGKILL tests use forked publisher processes at the pre-transaction boundary
(after immutable uploads), after SQL commit and after a pipe acknowledgement.
The parent restores through a new database connection and compares every retained
byte. These are real process kills, but the pipe acknowledgement is a fixture
signal, not an actual network client acknowledgement or an independently exec'd
worker restart. The actual CellularRegionHost is separately drained/recreated
and restores a profile-5 head with entry disabled, then materializes before a
returning-player operation. Previous raw worker process drills remain regression
coverage, not additional population-specific fault evidence.
The worker collects one second of due unattended time before publication and
services at most 64 fixed steps per batch. The portable input builder matches an
independent per-tick fixture. This removes per-tick SQL publication from the new
population path; it does not yet make local pacing debt a durable admission queue.
Still open: profile-5 executable/network fault drills, deployment-host tail costs,
durable ecological interval admission and biological migration escrow, jointly
owned environmental budgets, world-level history/extinction reconciliation and
actual multi-day/returning-player evidence. This work changes neither biological
versions nor goldens, and does not establish useful compression or M13 readiness.
+14 -21
View File
@@ -5,31 +5,24 @@ all of M12. M11.9 remains closed for now. M12.2 decisions are delegated and reco
in ADR-0139; no additional product approval is inferred as a coding prerequisite.
Completed scoped commits do not mean their entire roadmap gates have passed.
## Persistence boundary before expansion
## Persistence boundary resolved by 12.3b
The new exact population envelope cannot be labeled as the existing profile-4
CellularLives payload. The live publisher/recovery path, immutable manifest,
database validators and trace currently agree on that raw cellular format. A
population checkpoint containing ownership tags and policy needs a dedicated
profile and versioned transition/publication adapter. Reinterpreting profile 4
would violate REGION_DURABILITY and break retained recovery evidence.
Resolution: add an opt-in population profile and an explicit no-tick migration
from the accepted rules-8 cellular state, retaining identities, archive, funding
and balances. Publish its complete immutable payload, bound trace and permanent
events under the current region fence. Replay must retain recorded intentions
and exact membership samples, rather than rerunning a decision after an uncertain
commit. Verify lost replies and pre-upload/pre-transaction/post-commit/post-ACK
process death against a private PostgreSQL fixture. Keep production disabled until
these and target-host budgets pass. A portable codec is not a substitute for this
adapter. This is remaining M12.3 implementation, not a user-permission hold.
ADR-0142 adds dedicated profile 5, population trace 1 and schema 44; raw profile 4
retains its meaning. Explicit no-tick migrations preserve the complete biology.
The publisher, replay verifier, immutable SQL evidence, recovery and owning host
now support exact population membership with entry disabled by default. Private
PostgreSQL tests cover ambiguity, fencing, profile races and publisher process
kills. [Scope and limits](M12_DURABLE_POPULATIONS.md) distinguish those tests from
executable/network fault drills and deployment-host qualification still required.
This resolves the architectural profile conflict and permits the next durable
interval work. It does not close all of M12.3 or authorize production enablement.
## Dependent ecological integration
| Gate | Existing useful basis | Work/evidence still required |
|---|---|---|
| 12.2 | Exact cellular model and paired matrix; local samples through 256 lives | Deployment-host tail costs including publication; larger wildlife cohorts, predation/twins and contention coverage; useful compression remains unimplemented. |
| 12.3 | Complete population codec, exact conversion candidates, automatic policy and fresh-process continuation | Dedicated durable profile/trace/store/host integration and fault-injected atomic publication described above. |
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Current worker advances unattended wildlife and preserves running scheduling debt | Durable ecological interval admission/backlog recovery; exact biological source/destination escrow, partition/duplicate settlement and stock-in-transit audits. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained genome/organism archive with ancestry and common-ancestor queries | Append-only population census/history events in the same durable commit, bounded world history queries/indexing and client presentation; no account power or invented precise statistical ancestry. |
@@ -46,9 +39,9 @@ and photosynthesis steps B/C do not silently become accepted M12 mechanics.
The engineering constitution requires stopping expansion at this architectural
boundary and documenting the resolution. That does not prevent the independent
worker scheduling correction, which uses the existing raw cellular publication
path. The outstanding population integration is the next substantive coding
segment; none of the rows above is closed by adding a placeholder service.
worker scheduling correction or the dedicated exact population adapter above.
Durable ecological intervals are the next substantive coding segment; none of
the rows above is closed by adding a placeholder service.
Current expansion readiness: **no-go for M13 integration or World0 promotion**
while these M12 integration/evidence gates remain open. This records the present
+46
View File
@@ -1,5 +1,51 @@
# Validation report
## 2026-10-17 - M12.3b durable exact population publication
ADR-0142 affects portable lifecycle batches/checkpoint manifests, optional
PostgreSQL adapters/schema 44, the owning cellular host and opt-in worker batching.
No biological version or golden changes; no deployment or World0 operation.
Executed:
- MSVC 19.44 Release configure/build and **76/76 CTests** (final run 9.53 s).
- GCC 14.2 actual QUIC/PostgreSQL worker and integration-test build. Final private
`postgres-check.sh --population-only` **passed, exit 0, 5 s** after checked-access
refinements. Real reserved founders fund the wildlife fixture; tests exercise
exact membership oscillation, duplicate/conflicting requests, mutation/profile
race guards, missing/corrupt payloads, stale authority, host recreation and
materialization before player operations.
- Real PostgreSQL lease-row contention makes the adapter time out. One queued
request later commits and another backend is terminated before commit; both
resolve/retry the retained bytes. Forked publisher SIGKILLs occur before the
transaction, after commit and after a pipe ACK, with fresh-connection recovery.
This does not claim a population-specific network ACK or exec'd worker kill.
- Clang 19 actual server build; explicit clang-tidy 19 on changed optional store,
SQL adapter, cellular recovery, host, population integration tests and worker.
Final refinements use separately checked stable optional values; those follow-up
tidy runs pass. Analysis-preset portable checks and repository format check pass.
- Portable analysis CTests pass **7/7** (29.70 s); final batch-builder/checkpoint
checks pass **2/2** (3.73 s). ASan/UBSan population checkpoint/replay checks pass
(final 12.50 s), with no reports. Tests use explicit checks under NDEBUG.
The broad regression first exposed an over-broad mutation guard and the schema-44
upgrade expectation; both were corrected. The historical failed-migration drill
now supplies all required checksums, so it reaches its intended bad-v8-checksum
failure rather than an unrelated missing psql variable. The C++ suite, backup and
DR passed in the next run, but a concurrent local relink interrupted the upgrade
drill with `Text file busy`; this is not counted as a passing full run. Final
qualification uses a frozen copy of the executables to avoid that interference.
Final frozen-binary `postgres-check.sh --full` **passed, exit 0, 496 s**: complete
C++ suite (421 s), independent reservation connections, backup/restore, host-loss
DR, upgrade, migration history, permissions and checksum drift. The final optional
checked-access refinements additionally passed the targeted private population
suite above; all populations remain opt-in and no live region was changed.
Remaining scope: [M12_DURABLE_POPULATIONS](M12_DURABLE_POPULATIONS.md) and
[M12_REMAINING_GATES](M12_REMAINING_GATES.md). No compression, deployment-host
capacity, durable interval admission or multi-day ecological acceptance is inferred.
## 2026-10-17 - M12.4a unattended demand and confirmed-head pacing
ADR-0141 changes the existing cellular worker's demand/pacing and metrics, with