M12.6: capture consistent regional census snapshots
Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 2m20s
Foundation / checks (network) (push) Successful in 21m3s
Foundation / checks (client) (push) Successful in 22m14s
Foundation / checks (tsan) (push) Successful in 24m52s
Foundation / checks (postgres) (push) Failing after 36m19s
Foundation / checks (checks) (push) Successful in 1h10m25s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped

This commit is contained in:
dingo
2026-10-18 15:56:58 -04:00
parent 82568da9cd
commit 7631f9d2e2
17 changed files with 648 additions and 3 deletions
+16
View File
@@ -1,5 +1,21 @@
# Project status and next steps
## M12.6d consistent scoped census snapshots - 2026-10-18
Scoped census reads now select all declared regional heads from one short
read-only database snapshot, then fully replay the exact immutable cuts after
the metadata transaction ends. A concurrent atomic migration cannot split the
selected source/destination heads. Missing or failed evidence returns no partial
census; unsafe capture cleanup closes its connection.
Private raw/cohort migration races and TLS BEGIN/COMMIT request/reply-loss tests
pass in Release and ASan/UBSan, alongside existing history and migration/storage
faults. Native default Release passes **79/79**; Clang analysis and formatting
pass. [Evidence and limits](engineering/VALIDATION.md) retain durable world/branch
coverage, outside transfer/viability reconciliation, client access, target-host
query costs and later ecology/soak/product gates. A caller-declared consistent
cut cannot accept terminal extinction. No deployment or World0 action.
## M12.6c scoped multi-region census queries - 2026-10-18
The portable census now queries 1..64 explicitly declared regions at an exact
@@ -0,0 +1,66 @@
# ADR-0153: Consistent database cuts for scoped cellular census queries
## Status
Accepted for M12.6d, 2026-10-18. Backend read adapter under ADR-0151/0152;
no durable world catalog, external branch identity or extinction acceptance.
## Context and decision
Equal-tick regional observations can straddle an atomic migration. A source-after
and destination-before selection can omit stock without duplicating an organism.
The scoped portable query cannot detect that omission from detached totals alone.
Capture the current committed heads of 1..64 declared regions in one explicitly
owned PostgreSQL `REPEATABLE READ READ ONLY` transaction on an idle connection.
Requests are nonzero, distinct, ascending region IDs in one world, with object
stores and grants for the existing replay readers. Keep exact head/manifest pairs,
then commit the short metadata transaction before reading biological objects.
An atomic migration committed between head reads is either visible at both
endpoints or neither when both are in the declared set.
After capture, use ADR-0151's existing exact-cut history reader for each selected
head. Verify the returned head and manifest still match the captured cut. Complete
semantic replay, object hashes and fenced evidence checks remain mandatory; no
cached census or older-head fallback. Combine observations through ADR-0152's
coverage/boundary/genome/resident checks. Different head ticks fail rather than
advancing biology or silently selecting an older common tick. Return exact cuts
alongside the scoped census only after every step succeeds.
The adapter refuses all caller-owned transactions, including read-only ones.
It rolls back failed metadata capture; failed completion/cleanup closes the
connection. An exception during capture also closes the connection immediately,
releasing server resources when disconnect is observed. Add an explicit noexcept
disconnect method to the libpq owner for this cleanup. No automatic retry.
Per-I/O deadlines retain existing semantics, not a whole-query latency guarantee.
## Alternatives, benefits, costs and risks
Independent head reads can mix the sides of a paired publication. Holding one
transaction through full recovery is unnecessarily long, and existing fenced
evidence functions take lease row locks that a read-only transaction forbids.
Short read-only metadata capture plus immutable replay avoids that coupling.
Ordinary replay still briefly takes existing per-call lease locks; this does not
eliminate all contention or establish a deployment capacity claim.
Capture uses indexed regional head lookups. Work and retained observations remain
bounded by 64 regions; full replay/object cost is inherited from ADR-0151. A long
metadata snapshot can retain old database versions until completion. No new SQL
objects, cache/index, durable writes, biological rule/codec or dependency.
## Consequences and qualification
Require an actual paired migration on an independent connection between the first
and second head reads: both captured cuts must remain before the pair while a
later snapshot sees both after it. Recover the earlier cuts after publication,
compare exact census stock, and repeat for raw/cohort histories. Missing regions,
stale/missing/corrupt replay evidence, mismatched boundaries, invalid/over-bound
requests and caller-owned transactions must return no partial census. Confirm
transaction cleanup/connection closure, unchanged heads and no object writes.
This is one connected database's consistent committed cut for a caller-declared
region set. It neither proves that the list includes every world owner nor
reconciles branches, outside transfers, dormant/reserve viability or global
freshness. Current grants support recovery, not new observation authority. No
terminal-extinction outcome or public client permission follows. Durable
world/branch coverage and complete M12.7 reconciliation remain open.
+2
View File
@@ -1,5 +1,7 @@
# Architecture decision records
- [ADR-0153: Consistent scoped cellular census snapshots](0153-consistent-cellular-census-snapshots.md) - short read-only repeatable-read head capture followed by exact immutable replay; atomic pairs cannot straddle a declared-region cut, without claiming world/branch completeness.
- [ADR-0152: Scoped multi-region cellular census queries](0152-scoped-cellular-census-sets.md) - bounded declared coverage, exact boundaries, complete shared-genome equality and disjoint resident identities; no world-completeness or extinction authority.
- [ADR-0151: Committed cellular census history](0151-committed-cellular-census-history.md) - derives bounded, exact-cut census pages from the existing atomic immutable biological journal, with full historical replay; no duplicate census writes or world-extinction authority.
@@ -1,5 +1,14 @@
# Persistence and region authority
M12.6d captures declared regional heads in one short read-only repeatable-read
transaction, closes it, then fully replays each exact immutable cut before
constructing a scoped census. Atomic migration commits cannot split the selected
head set when both endpoints are declared. Failed metadata capture rolls back;
failed cleanup or an exception disconnects the session. Full replay remains
outside the metadata transaction, with existing per-call evidence locks.
See [ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md). The caller's
list is still not complete world coverage or external branch identity.
M12.6c's portable [scoped census set](../../schemas/ecology/scoped-cellular-census-set-v1.md)
combines a bounded declared set of regional observations, including replayed
history entries. It verifies exact common boundaries, complete shared genome
@@ -22,6 +22,13 @@ It refuses missing/stale/conflicting evidence. The list remains caller-declared;
neither a durable world catalog nor reconciled migration cuts or branch proof
is supplied, so this query cannot accept world absence or terminal extinction.
M12.6d's [consistent scoped snapshots](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md)
select current regional heads from one database snapshot, then verify each exact
immutable cut through full recovery. A pair cannot commit between selected sides
when both regions are declared. Durable required-world coverage, outside transfer
and reserve reconciliation, external branch identity and extinction acceptance
remain separate; this is no complete world evidence envelope.
## Identity and classification
An organism has a permanent identity and terminal death. An immutable genome has
+1 -1
View File
@@ -13,7 +13,7 @@ Commit and push each validated scoped segment separately.
| 12.3 | Exact population profile/trace/store and opt-in host integration implemented (12.3b) | ADR-0142, schema 44; exact publication/recovery, mutation race guards, ambiguous replies and process kills. Executable/network population fault drills and target-host qualification remain open. |
| 12.4 | Owned intervals and durable paired migration/fault qualification implemented in scoped segments through 12.4l | ADR-0143–0149/schema 49; bounded admitted work, source freeze, staged ghost, atomic pair and exact recovery, local TLS and storage faults. Multi-host/orchestrator, linked ecology and integrated stock audits remain open. |
| 12.5 | Blocked on 12.2-12.4 | Finite M6 process fixtures do not qualify coupled persistent cellular/cohort atmosphere/water budgets. |
| 12.6 | Regional census/lineage queries, replayed commit-bound history pages and scoped multi-region queries implemented (12.6a-c) | ADR-0150–0152; atomic biological commits are the durable events. Fixed regional cuts and full replay bind observations; declared sets reject missing/stale/conflicting identity or duplicate residents. Durable world/branch coverage, reconciled migration cuts, target-host query/index costs and client path remain required. |
| 12.6 | Regional queries, replayed history, scoped multi-region queries and consistent database head snapshots implemented (12.6a-d) | ADR-0150–0153; short read-only snapshot capture prevents atomic pairs from straddling declared head sets, with exact immutable replay. Durable world/branch coverage, outside transfer/viability reconciliation, target-host query/index costs and client path remain required. |
| 12.7 | Evidence policy defined; reconciler open | Complete durable world watermark and every viable/in-flight store remain required. |
| 12.8 | Not qualified | Actual multi-day integrated ecology/restart/clone evidence remains required. |
| 12.9 | Not accepted | Returning-player product evidence and expansion review remain required; World0 stays separately blocked. |
+8 -1
View File
@@ -27,7 +27,7 @@ or authorize production enablement.
| 12.3 | Dedicated population profile/trace/store/owning host; exact recovery and private PostgreSQL fault tests | Population-specific executable/network fault drills and target-host publication/capacity qualification. |
| 12.4 | Owned intervals; exact transfer/accounting; durable atomic pair/recovery; independent loopback TLS path loss and owner expiry/replacement/heal; local object sync/read-failure matrix | Multi-host/kernel-network, storage-host and orchestrator drills, linked-boundary contracts and integrated ownership/stock audits at scale. |
| 12.5 | Finite cellular contact feeding/decay and separate M6 producer/consumer resource owner | One jointly owned cellular/environment budget, explicit effective depth and source entitlements, versioned water/atmosphere species mapping and producer coefficients, plus closed/open and nutrient-exhaustion comparisons. |
| 12.6 | Retained archive, regional census/lineage queries, replayed commit-bound history and bounded scoped multi-region queries (12.6a-c) | Durable world/branch history context and required-region coverage, reconciled migration cuts, target-host query/index qualification and client permission/presentation; no account power or invented precise statistical ancestry. |
| 12.6 | Retained archive, regional census/lineage queries, replayed history, scoped multi-region queries and consistent database head snapshots (12.6a-d) | Durable world/branch history context and required-region coverage, outside transfer/viability reconciliation, target-host query/index qualification and client permission/presentation; no account power or invented precise statistical ancestry. |
| 12.7 | ADR-0133's viability/unknown semantics | Durable region-set/watermark reconciliation covering every owned representation, suspended/viable stock and migration escrow. Missing/stale regions must prevent terminal extinction. |
| 12.8 | Deterministic seed/regime drivers and separate-process exact continuation | Reproducible integrated worlds running for actual days, real restarts/clone comparisons and retained accounting/diversity/cost audits. Accelerated tests cannot supply elapsed days. |
| 12.9 | M11 feedback and current UI review retained | Returning-player samples on the integrated ecology and explicit product/3D readiness decision after soak evidence. No M12 product acceptance or World0 promotion is inferred. |
@@ -114,6 +114,13 @@ before querying. Nonresident historical records add no living stock. A required
list supplied by a caller is still not durable world coverage; equal ticks alone
cannot prove a consistent migration cut or branch. This does not close M12.7.
M12.6d's [consistent scoped snapshot](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md)
now captures all declared heads in one database read snapshot before exact replay.
An atomic pair cannot straddle that selected set when both endpoints are declared.
The short metadata transaction ends before object I/O and fenced recovery; failed
evidence still returns no partial census. Durable world/branch identity, required
owner coverage, outside transfers and viable-store reconciliation remain open.
Current expansion readiness: **no-go for M13 integration or World0 promotion**
while these M12 integration/evidence gates remain open. This records the present
engineering evidence, not a substitute for the later maintainer product review.
+12
View File
@@ -1,5 +1,17 @@
# Validation strategy
M12.6d's private PostgreSQL snapshot tests publish a real atomic migration from
an independent connection between the first and second captured head reads.
Both selected cuts must remain before the pair; the next snapshot sees both
after it. Recover the earlier cuts after publication and compare full regional
rows plus exact/descendant count/body/reserve, in both directions and raw/cohort
profiles. Verify actual repeatable-read/read-only settings, no metadata transaction
held over replay, exact cut binding and unchanged biological stock. Invalid,
missing, unretained, stale, unreadable or caller-transaction evidence returns no
partial result. Capture errors roll back; lost completion/throwing hooks close
the session. Sanitize/inspect the new adapter and existing libpq owner together.
This does not qualify complete world/branch coverage or terminal extinction.
M12.6c's scoped census-set contract compares declared-region queries with direct
resident count/body/reserve oracles. Require missing/extra/duplicate regions,
every boundary field, unknown/invalid scope, conflicting immutable successor IDs,
+68
View File
@@ -1,5 +1,73 @@
# Validation report
## 2026-10-18 - M12.6d consistent scoped census snapshots
[ADR-0153](../adr/0153-consistent-cellular-census-snapshots.md) captures the current
heads of 1..64 declared regions in one owned read-only repeatable-read database
transaction. It completes that short metadata transaction before fully replaying
each immutable selected cut through the existing history reader. Exact cut binding
and the scoped census-set checks remain mandatory. Failed capture rolls back;
failed completion/cleanup or a capture exception disconnects the session. No
partial result or older-head fallback. See
[the contract](../../schemas/ecology/consistent-cellular-census-snapshot-v1.md).
Executed locally:
- Native MSVC **19.44.35217.0** default Release build/CTest **79/79 passed**,
**13.76 s**, with optional Linux persistence disabled.
- GCC 14 Release private population/history integration passed **17 s** (15 s
population phase). Final combined snapshot/history/migration **TLS and storage
fault qualification passed, exit 0, 39 s** (38 s population phase).
- Clang 19 Linux **ASan/UBSan** private population integration passed **46 s**;
the final combined TLS/storage/snapshot matrix passed **exit 0, 70 s** (66 s
population phase), without sanitizer reports.
- Explicit clang-tidy 19 covered the libpq owner, snapshot adapter and integration
tests, then rechecked the final TLS assertions; no owned warnings. Repository
clang-format 19 and `git diff --check` passed.
Each raw/cohort migration race uses an independent publishing connection after
the first captured head read. Actual database settings are checked as repeatable
read and read-only. Both captured heads remain before the atomic pair; a later
snapshot sees both new heads. Earlier cuts fully replay after publication, retain
identical complete regional rows, and preserve exact/descendant resident counts,
body and reserve in both migration directions. The metadata transaction is closed
before the existing idle-only historical reader runs.
Tests also retain exact head/manifest binding on repeated reads, direct biology
oracles, request bounds/order/null validation, missing regions, incompatible
current ticks/owner versions, stale grants, unretained builds, missing/unreadable
payloads and unchanged current heads. Caller-owned read-only transactions and a
writer's pending publication are refused without taking over its transaction.
Capture failure rolls back; throwing hooks and failed completion close the
connection. Disconnect is idempotent and later queries return `closed`.
Actual loopback TCP with end-to-end TLS independently drops requests and replies
at both BEGIN and COMMIT. Every uncertain outcome returns no census, closes the
query session and preserves observed heads. Existing migration owner expiry,
replacement/heal, exact recovery and local object-failure checks still pass.
All database runners used fresh private temporary clusters and stopped them on
exit; no live database, deployment or World0 action.
The test allowlist initially needed an explicit nested digest-array type; analysis
also required consuming the expected result in the throwing-hook test. Both were
corrected and the final matrix rebuilt. No checks/goldens were suppressed or
rewritten. No SQL schema, biological rule/codec, dependency or ownership change.
Limits: this is a consistent cut of a caller-declared set in one connected
database, not durable world coverage or external branch identity. Outside-region
transfers, reserve viability and complete global freshness/reconciliation remain
open. SQL deadlines do not preempt local object reads or promise whole-query
latency. Existing replay evidence calls retain their per-call lease locks. No
deployment-host capacity claim, new TSan, complete SQL/backup/DR rerun, remote CI,
multi-host drill or actual multi-day soak. Prior full persistence regression is
retained in M12.6b. No terminal-extinction acceptance or client endpoint.
Logs: `build/m126d-native.log`, `build/m126d-postgres.log`,
`build/m126d-asan.log`, `build/m126d-final-postgres-build.log`,
`build/m126d-final-postgres.log`, `build/m126d-final-asan-build.log`,
`build/m126d-final-asan.log`, `build/m126d-final-tidy.log`,
`build/m126d-final-integration-tidy.log`, `build/m126d-final-format.log`.
## 2026-10-18 - M12.6c scoped multi-region census queries
[ADR-0152](../adr/0152-scoped-cellular-census-sets.md) adds a portable detached