Implement M7.4a checkpoint publication and verified recovery

ADR-0061 publishes environmental-v1 full checkpoints. Manifest v1 is a
fixed 280-byte record (identity, chain, lease, versions, build, zero
RNG/history declarations, durability policy, payload length/SHA-256)
that PostgreSQL parses and digests itself. Schema v3 adds region_commits
with a digest CHECK and a composite head FK, owner-managed retained
builds, fenced publish/load functions and owner-only genesis; it drops
the bare v2 head kernel and fails closed while v2 heads exist.

Linux evo::region_store adds OpenSSL SHA-256, an immutable
content-addressed filesystem object store (temp+fsync+link, no
overwrite/delete, O_NOFOLLOW exact-length verified reads), and
publication (validate -> upload -> read back -> fenced transaction ->
receipt check) and recovery (re-digest, allowlist, verify, full restore).
Portable evo::region_checkpoint holds the codec and payload validation.

Validated: Windows CTest 56/56; WSL Release, clang-tidy and ASan/UBSan
private-PostgreSQL suites; four SQL and one host mutation each caught.
M7.4 stays open for 7.4b (lineage archive codec and history binding).
No Docker/Gitea CI, remote store, process kill or deployment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-28 15:25:27 -04:00
co-authored by Claude Opus 5.5
parent 07116446c6
commit 5d4f59f901
32 changed files with 2253 additions and 145 deletions
+14
View File
@@ -515,3 +515,17 @@ Real-PostgreSQL tests cover the scenarios the roadmap row requires:
Timings were selected and measured locally. The director is intentionally deferred
because it has no authority and would be a placeholder. Next: M7.4 checkpoint
codecs/publication. It must also revoke runtime EXECUTE on the bare head kernel.
M7.4 split into 7.4a (environmental-v1 publication) and 7.4b (lineage archive
codec and history-capable binding). M7.4a is implemented and locally qualified
(2026-09-28) under ADR-0061:
- Manifest v1 is 280 bytes and is parsed by the database itself.
- Schema v3 adds a commit chain with digest and head constraints, retained
builds, and fenced publication. The bare v2 kernel is dropped.
- SHA-256 comes from OpenSSL, and payloads go to an immutable filesystem store.
- Publication validates, uploads, verifies, then commits. Recovery re-digests and
verifies.
- Missing or corrupt objects, upload and manifest failures, orphans, size caps,
retained-build allowlists and restored-trajectory equality are covered.
Next: M7.4b. M7.4 stays open until it passes.
+19
View File
@@ -769,3 +769,22 @@ lease check, a disabled busy check) were each detected by a failing test.
Windows ran 55/55; Linux Release, clang-tidy and ASan/UBSan PostgreSQL suites
passed in WSL. No director, host, gateway, payload publication or deployment
exists yet. The v2 head must not carry real world state before M7.4. Next: M7.4.
## M7.4a checkpoint publication (2026-09-28)
ADR-0061 and schema v3 publish environmental-v1 checkpoints. A fixed 280-byte
manifest is parsed and digested by PostgreSQL. Payloads live in an immutable,
content-addressed filesystem store and are hashed with OpenSSL SHA-256. The
database enforces the commit chain, retained builds and fixed versions/quantum,
and refuses to migrate while v2 heads exist.
Recovery re-digests the manifest, checks the build allowlist and fully restores the
payload. A restored state continues identically to the uninterrupted original.
Missing, corrupt or tampered data quarantines the region. Four SQL mutations and
one host mutation were each caught by a test.
Local fixture payloads are 23-37 KiB and publish in about 3.5-10.6 ms across
builds. Windows ran 56/56; Linux Release, clang-tidy and ASan/UBSan PostgreSQL
suites passed in WSL. M7.4 stays open for M7.4b (lineage archive codec and
history-capable binding). No real world was provisioned; there is no deployment,
host or process-kill test yet.
+3 -1
View File
@@ -35,4 +35,6 @@ Record revision/hash, owner, license, build options and removal boundary; mirror
M7.2's [ADR-0059](0059-postgresql-bootstrap.md) accepts a scoped checksum-pinned
source build for the paired PostgreSQL server/client. The general cross-platform
package-manager comparison remains open.
package-manager comparison remains open. M7.4's [ADR-0061](0061-checkpoint-publication.md)
links system OpenSSL 3 libcrypto (already required by libpq TLS) for SHA-256 in the
Linux persistence adapter only.
+2
View File
@@ -97,6 +97,8 @@ binds it to verified payloads, so v2 must not host real world state. Custom
SQLSTATEs are project conventions; they are not PostgreSQL-reserved.
## Consequences
Done in M7.4a (ADR-0061): schema v3 drops the bare kernel and publishes through a
manifest-parsing function under the same lock order.
The M7.4 migration must add the manifest/payload publication function that calls
the kernel, and revoke runtime EXECUTE on `advance_region_head`. M7.5 adds
commit-ID resolution. Until then, "head equals my commit ID" is only the primitive
+112
View File
@@ -0,0 +1,112 @@
# ADR-0061: Checkpoint publication and verified recovery
## Status
Accepted for M7.4a, 2026-09-28. Covers the `environmental-v1` profile only. The
lineage archive codec and history-capable profile binding are M7.4b; M7.4 is not
complete until both parts pass. Decisions, outbox and commit resolution remain M7.5.
## Context
ADR-0058 selects full checkpoint batches: payload bytes go to an immutable object,
then a fenced database transaction makes them the committed head. ADR-0060 fences
the head, but its v2 kernel accepts any 32-byte digest with no payload behind it.
ADR-0059 deferred the SHA-256 provider. No object-store dependency has been selected.
The M6 environmental envelope is complete for its profile and already validates
itself fully on restore.
## Decision
**Manifest v1** is a fixed 280-byte little-endian record (schemas/region-commit.md)
holding:
- identity, commit ID and sequence, predecessor ID and digest, and the tick range;
- the lease holder and epoch, and the tick quantum;
- nine pinned versions and the retained build identity;
- explicit declarations that there are zero RNG streams, no history profile, and a
single-volume development durability policy;
- the payload length and SHA-256.
The commit digest is SHA-256 of these bytes. The database parses the same bytes
itself, so no separately supplied field can disagree with the digested record.
**Schema v3** makes the following changes:
- It adds `region_commits`. A CHECK constraint ties each commit digest to its
manifest, and a composite foreign key requires the head to name an existing
commit with the same ID and digest.
- It adds owner-managed `retained_builds`.
- It adds `publish_region_checkpoint(manifest)`, in the same order as the v2 kernel:
lease row lock, database-time lease check, then head lock. It requires the
exact successor of the committed head, with unchanged versions and tick quantum,
a retained build, and a unique commit ID.
- It adds `load_region_checkpoint` and owner-only `provision_region_checkpoint`
for genesis.
- It drops the v2 kernel and v2 provisioning functions.
- It refuses to migrate a database that still has v2 heads, because those have no
payload binding. Such regions must be re-provisioned.
The runtime role can execute exactly six functions.
**SHA-256** comes from the system OpenSSL 3 libcrypto EVP interface. It is linked
only into the Linux `evo::region_store` adapter. libpq TLS already requires that
library in the same images. Locally observed: Ubuntu `libssl3t64`/`libssl-dev`
3.0.13-0ubuntu3.15, Apache-2.0. PostgreSQL's built-in `sha256()` independently
computes commit digests, and the host checks that the receipt matches.
**Object store**: a filesystem store on a trusted mounted volume (Linux only).
- Each region has its own `w<world>/r<region>` directory with mode 0700. Objects
are named by the hex SHA-256 of their content.
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
link never replaces an existing name), and then fsyncs the directory.
- Existing identical content is reused. Differing content already under a key is
reported as corrupt and never overwritten.
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
and verify the digest.
- There is no delete or GC operation: unreferenced uploads are orphans.
**Publication order**:
1. Validate the candidate the way recovery would: full envelope restore,
identity, tick and quantum checks, and a byte-identical re-encode.
2. Upload the payload.
3. Read it back and verify it.
4. Run the fenced transaction.
5. Compare the receipt with the local manifest digest.
Only step 4 has authority. **Recovery** loads the head and manifest, re-digests
the manifest, requires the head fields to match and the build to be in the host's
retained allowlist, fetches and verifies the object, and fully restores it. It
never creates genesis or selects an older head. A missing or corrupt payload
quarantines the region.
## Alternatives Considered
- **S3/MinIO clients** add a large network dependency before any measured need.
The interface boundary allows one later, with a separate ADR.
- **Payload bytes in PostgreSQL** was deferred by ADR-0058 until measured.
- **Host-supplied manifest fields as SQL parameters** could disagree with the
digested bytes.
- **A project-written SHA-256** is forbidden by the constitution.
- **rename()** can silently replace an existing object.
## Benefits
The committed head is always backed by a verified, immutable payload whose
identity, chain, lease and versions the database has checked. Tampering is
blocked by constraints and detected again on recovery. Crashes before the
transaction leave the previous head.
## Costs
- A full payload is written per batch. The local fixture's payload is about 23-37
KiB, and publication took about 3.5-6.6 ms in non-sanitizer builds.
- Orphans accumulate.
- The existing v2 regions must be re-provisioned.
- Recovery fetches whole payloads.
## Risks
- A single volume is not host-loss durability (M8).
- `fsync` semantics depend on the filesystem and mount options. The local test
volume does not prove power-loss behavior.
- The database cannot validate payload contents: a compromised worker holding a
valid lease can publish a self-consistent but illegitimate state (ADR-0058).
- The read-back verification path is not exercised by any fault injection.
## Consequences
M7.4b adds the lineage archive codec, organism-to-genome binding and a separately
versioned history-capable profile. M7.5 adds decisions, outbox and commit-ID
resolution to the same transaction. Operators insert each promoted image digest
into `retained_builds` before hosts publish with it, and hosts restore only builds
in their own allowlist.
+1
View File
@@ -71,3 +71,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0059: PostgreSQL bootstrap](0059-postgresql-bootstrap.md) - accepted for M7.2 optional adapter, pinned server/client, migrations and Secret boundary.
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
+14
View File
@@ -168,3 +168,17 @@ rejected, and the head is unchanged. A paused stale worker resuming past its lea
is fenced both locally and by the database. A writer queued on the lease lock past
expiry is also rejected. These are in-process pauses, not killed pods or network
partitions. The 7.6 parts of those rows remain open.
M7.4a (ADR-0061) executed its rows for environmental-v1 against real PostgreSQL and
a local filesystem object store:
- A partial upload leaves only an unresolvable temporary name.
- A failed upload sends no manifest.
- A verified upload whose transaction is rejected stays an unreferenced orphan,
and recovery returns the previous head.
- Missing, corrupt, symlinked or resized payloads quarantine the region.
- Unsupported builds are refused by both the database and recovery.
- Recovery never creates genesis or picks an older head.
Two parts remain open. M7.4b adds the lineage archive codec and a history-capable
profile; until then the history rows of this audit stay open. No process-kill
test was run yet (7.7).
+55
View File
@@ -2103,3 +2103,58 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
overflow (`9223372036854775807 + 2000`).
- Reran Windows CTest 55/55, the repository format check, and GCC14 Release,
clang-tidy analysis and ASan/UBSan private-PostgreSQL suites: all passed.
## M7.4a checkpoint publication - 2026-09-28
- Windows MSVC Release configure/build/CTest passed 56/56, including the new
portable `region-checkpoint-contracts`. It covers:
- golden byte offsets and round trips;
- malformed and unsupported manifests, including u32 values above 255 that
must not truncate onto supported enums;
- chain rules;
- payload identity, tick, quantum, corruption and budget checks;
- 24-tick continuation equality after restore.
The repository clang-format 19 check passed.
- WSL Ubuntu, with GCC 14.2.0, clang 19.1.1, PostgreSQL/libpq 18.6 and OpenSSL
3.0.13-0ubuntu3.15, running as uid 1000 so permission faults were really
exercised. GCC14 Release, clang19 analysis (clang-tidy warnings-as-errors) and
clang19 ASan/UBSan each passed `tools/ci/postgres-check.sh`.
- Analysis initially flagged `performance-enum-size` on the u32-based
profile/durability enums. They are now u8 with explicit decoding of values
above 255 as unknown. All three configurations were rerun.
- New real-database/filesystem coverage:
- SHA-256 vectors (NIST "", "abc" and the 448-bit message) and agreement with
PostgreSQL `sha256()`.
- Object store: rejects root symlinks, creates 0700 namespaces, puts are
idempotent, never overwrites differing content, ignores temporary leftovers,
and detects byte flips, symlinks, truncation and growth. Size bounds and
write-permission failure are also covered.
- Genesis provisioning and duplicate refusal; three batches (16/64/16 ticks);
a 65-tick batch refused before upload.
- Recovery from a new store handle is byte-identical and continues 44 ticks
identically to an uninterrupted reference.
- The build allowlist is enforced by the host and by the database.
- Upload failure leaves the head unchanged. A fenced publication leaves a
verified orphan, and recovery ignores it.
- Missing and corrupt payloads quarantine; restoring the object recovers.
- Owner edits to a manifest (23514) and head repointing (23503) are refused.
With the constraint dropped, recovery still detects the edit, and the
constraint is restored afterwards.
- Unknown regions are never created.
- Lease tests were ported to manifest publication. They additionally cover
malformed manifests, a quantum change, an unretained build, a future rules
version and a globally reused commit ID.
- Migrations: clean→v3, repeated no-op, v1→v3. v2 with existing heads fails
closed and stays at v2. The audit rejects EXECUTE on provisioning and SELECT on
commits. Tampered v2/v3 checksums and an unknown v4 are refused.
- Mutations, run on scratch copies with the source restored afterwards: removing
the predecessor-digest check, the lease check, the version/quantum continuity
check or the retained-build check in SQL, or the host manifest re-digest, each
failed at the expected assertion.
- Local fixture measurements: payloads of 22775 and 37367 bytes.
- Release publication took 3.6-4.8 ms per batch; analysis 5.5-6.7 ms; ASan
about 10.5 ms.
- Manifest publish p50 was about 0.74-0.91 ms.
These are local socket and local disk only.
- Not executed: Docker/Gitea CI, a remote object store or PVC, power-loss fsync
semantics, read-back verification fault injection, process kill, and TSan.
+33 -12
View File
@@ -68,7 +68,7 @@ every role transitively in it, and rolls back on any violation:
`pg_write_all_data`);
- any table privilege in `evolution`, including via PUBLIC;
- CREATE on the schema, the database or `public`;
- EXECUTE outside the five fenced functions.
- EXECUTE outside the six fenced functions (schema v3).
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
@@ -77,20 +77,41 @@ never creates logins or passwords, and never grants superuser.
## Fenced region ownership (schema v2, ADR-0060)
The runtime role has schema USAGE and EXECUTE on five SECURITY DEFINER functions:
`acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
`load_region_head` and `advance_region_head`. It cannot read or write tables, create
objects, call `check_region_lease` or provision regions. Each mutation locks the lease row
In schema v3 the runtime role has schema USAGE and EXECUTE on six SECURITY DEFINER
functions: `acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
`load_region_head`, `publish_region_checkpoint` and `load_region_checkpoint`. It
cannot read or write tables, create objects, call internal check/parse functions or
provision regions. (Schema v2's `advance_region_head` is dropped in v3.) Each mutation locks the lease row
and checks holder, epoch and expiry with `clock_timestamp()` after the lock wait.
Clients never send times. Rejections: EVL01 busy, EVL02 fenced, EVL03 head mismatch,
EVL04 not provisioned, EVL05 epoch/sequence exhausted, 22023 invalid arguments.
`evo::postgres` maps them to `LeaseStoreError`; a timeout or broken connection is
`unknown`.
EVL04 not provisioned, EVL05 epoch/sequence exhausted, EVL06 unsupported
profile/version/build or a changed rules/quantum, EVL07 reused commit ID, and 22023
malformed arguments or manifest. `evo::postgres` maps them to `LeaseStoreError`; a
timeout or broken connection is `unknown`.
Genesis is operator-only (owner login), never done by recovery or runtime:
`SELECT evolution.provision_region(world, region, frame, genesis_commit_uuid,
'\x<64 hex digest>');`. In schema v2 the head is an identity/digest pointer with no
payload binding. Do not provision real world state before M7.4 publication.
## Checkpoint publication (schema v3, ADR-0061)
Each commit row stores the exact 280-byte manifest, and a constraint ties the
commit digest to `sha256(manifest)`. The head has a composite foreign key to its
commit, so even the owner cannot silently repoint or edit it.
Payloads are immutable files under a trusted object root:
`<root>/w<world>/r<region>/<sha256 hex>`, mode 0700 directories and 0600 files.
Mount a dedicated volume that only the region host's user can write. Nothing is
ever deleted; orphans from failed transactions are expected.
Operator steps:
1. Record each promoted build (image digest, 32 bytes) before a host uses it:
`INSERT INTO evolution.retained_builds(build_identity) VALUES ('\x<64 hex>');`
2. Provision genesis only through `provision_environmental_region` on an owner
connection. It uploads and verifies the tick-0 payload, then calls the
owner-only `provision_region_checkpoint(manifest)`.
There is no operator CLI for provisioning yet, and no real world has been
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
have no payload.
## Host lease discipline
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
interval, 2 s safety margin, 2 s renewal deadline and 5 s commit deadline. The