Implement M7.4a checkpoint publication and verified recovery
ADR-0061 publishes environmental-v1 full checkpoints. Manifest v1 is a fixed 280-byte record (identity, chain, lease, versions, build, zero RNG/history declarations, durability policy, payload length/SHA-256) that PostgreSQL parses and digests itself. Schema v3 adds region_commits with a digest CHECK and a composite head FK, owner-managed retained builds, fenced publish/load functions and owner-only genesis; it drops the bare v2 head kernel and fails closed while v2 heads exist. Linux evo::region_store adds OpenSSL SHA-256, an immutable content-addressed filesystem object store (temp+fsync+link, no overwrite/delete, O_NOFOLLOW exact-length verified reads), and publication (validate -> upload -> read back -> fenced transaction -> receipt check) and recovery (re-digest, allowlist, verify, full restore). Portable evo::region_checkpoint holds the codec and payload validation. Validated: Windows CTest 56/56; WSL Release, clang-tidy and ASan/UBSan private-PostgreSQL suites; four SQL and one host mutation each caught. M7.4 stays open for 7.4b (lineage archive codec and history binding). No Docker/Gitea CI, remote store, process kill or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -515,3 +515,17 @@ Real-PostgreSQL tests cover the scenarios the roadmap row requires:
|
||||
Timings were selected and measured locally. The director is intentionally deferred
|
||||
because it has no authority and would be a placeholder. Next: M7.4 checkpoint
|
||||
codecs/publication. It must also revoke runtime EXECUTE on the bare head kernel.
|
||||
|
||||
M7.4 split into 7.4a (environmental-v1 publication) and 7.4b (lineage archive
|
||||
codec and history-capable binding). M7.4a is implemented and locally qualified
|
||||
(2026-09-28) under ADR-0061:
|
||||
- Manifest v1 is 280 bytes and is parsed by the database itself.
|
||||
- Schema v3 adds a commit chain with digest and head constraints, retained
|
||||
builds, and fenced publication. The bare v2 kernel is dropped.
|
||||
- SHA-256 comes from OpenSSL, and payloads go to an immutable filesystem store.
|
||||
- Publication validates, uploads, verifies, then commits. Recovery re-digests and
|
||||
verifies.
|
||||
- Missing or corrupt objects, upload and manifest failures, orphans, size caps,
|
||||
retained-build allowlists and restored-trajectory equality are covered.
|
||||
|
||||
Next: M7.4b. M7.4 stays open until it passes.
|
||||
|
||||
@@ -769,3 +769,22 @@ lease check, a disabled busy check) were each detected by a failing test.
|
||||
Windows ran 55/55; Linux Release, clang-tidy and ASan/UBSan PostgreSQL suites
|
||||
passed in WSL. No director, host, gateway, payload publication or deployment
|
||||
exists yet. The v2 head must not carry real world state before M7.4. Next: M7.4.
|
||||
|
||||
## M7.4a checkpoint publication (2026-09-28)
|
||||
|
||||
ADR-0061 and schema v3 publish environmental-v1 checkpoints. A fixed 280-byte
|
||||
manifest is parsed and digested by PostgreSQL. Payloads live in an immutable,
|
||||
content-addressed filesystem store and are hashed with OpenSSL SHA-256. The
|
||||
database enforces the commit chain, retained builds and fixed versions/quantum,
|
||||
and refuses to migrate while v2 heads exist.
|
||||
|
||||
Recovery re-digests the manifest, checks the build allowlist and fully restores the
|
||||
payload. A restored state continues identically to the uninterrupted original.
|
||||
Missing, corrupt or tampered data quarantines the region. Four SQL mutations and
|
||||
one host mutation were each caught by a test.
|
||||
|
||||
Local fixture payloads are 23-37 KiB and publish in about 3.5-10.6 ms across
|
||||
builds. Windows ran 56/56; Linux Release, clang-tidy and ASan/UBSan PostgreSQL
|
||||
suites passed in WSL. M7.4 stays open for M7.4b (lineage archive codec and
|
||||
history-capable binding). No real world was provisioned; there is no deployment,
|
||||
host or process-kill test yet.
|
||||
|
||||
@@ -35,4 +35,6 @@ Record revision/hash, owner, license, build options and removal boundary; mirror
|
||||
|
||||
M7.2's [ADR-0059](0059-postgresql-bootstrap.md) accepts a scoped checksum-pinned
|
||||
source build for the paired PostgreSQL server/client. The general cross-platform
|
||||
package-manager comparison remains open.
|
||||
package-manager comparison remains open. M7.4's [ADR-0061](0061-checkpoint-publication.md)
|
||||
links system OpenSSL 3 libcrypto (already required by libpq TLS) for SHA-256 in the
|
||||
Linux persistence adapter only.
|
||||
|
||||
@@ -97,6 +97,8 @@ binds it to verified payloads, so v2 must not host real world state. Custom
|
||||
SQLSTATEs are project conventions; they are not PostgreSQL-reserved.
|
||||
|
||||
## Consequences
|
||||
Done in M7.4a (ADR-0061): schema v3 drops the bare kernel and publishes through a
|
||||
manifest-parsing function under the same lock order.
|
||||
The M7.4 migration must add the manifest/payload publication function that calls
|
||||
the kernel, and revoke runtime EXECUTE on `advance_region_head`. M7.5 adds
|
||||
commit-ID resolution. Until then, "head equals my commit ID" is only the primitive
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# ADR-0061: Checkpoint publication and verified recovery
|
||||
|
||||
## Status
|
||||
Accepted for M7.4a, 2026-09-28. Covers the `environmental-v1` profile only. The
|
||||
lineage archive codec and history-capable profile binding are M7.4b; M7.4 is not
|
||||
complete until both parts pass. Decisions, outbox and commit resolution remain M7.5.
|
||||
|
||||
## Context
|
||||
ADR-0058 selects full checkpoint batches: payload bytes go to an immutable object,
|
||||
then a fenced database transaction makes them the committed head. ADR-0060 fences
|
||||
the head, but its v2 kernel accepts any 32-byte digest with no payload behind it.
|
||||
ADR-0059 deferred the SHA-256 provider. No object-store dependency has been selected.
|
||||
The M6 environmental envelope is complete for its profile and already validates
|
||||
itself fully on restore.
|
||||
|
||||
## Decision
|
||||
**Manifest v1** is a fixed 280-byte little-endian record (schemas/region-commit.md)
|
||||
holding:
|
||||
- identity, commit ID and sequence, predecessor ID and digest, and the tick range;
|
||||
- the lease holder and epoch, and the tick quantum;
|
||||
- nine pinned versions and the retained build identity;
|
||||
- explicit declarations that there are zero RNG streams, no history profile, and a
|
||||
single-volume development durability policy;
|
||||
- the payload length and SHA-256.
|
||||
|
||||
The commit digest is SHA-256 of these bytes. The database parses the same bytes
|
||||
itself, so no separately supplied field can disagree with the digested record.
|
||||
|
||||
**Schema v3** makes the following changes:
|
||||
- It adds `region_commits`. A CHECK constraint ties each commit digest to its
|
||||
manifest, and a composite foreign key requires the head to name an existing
|
||||
commit with the same ID and digest.
|
||||
- It adds owner-managed `retained_builds`.
|
||||
- It adds `publish_region_checkpoint(manifest)`, in the same order as the v2 kernel:
|
||||
lease row lock, database-time lease check, then head lock. It requires the
|
||||
exact successor of the committed head, with unchanged versions and tick quantum,
|
||||
a retained build, and a unique commit ID.
|
||||
- It adds `load_region_checkpoint` and owner-only `provision_region_checkpoint`
|
||||
for genesis.
|
||||
- It drops the v2 kernel and v2 provisioning functions.
|
||||
- It refuses to migrate a database that still has v2 heads, because those have no
|
||||
payload binding. Such regions must be re-provisioned.
|
||||
|
||||
The runtime role can execute exactly six functions.
|
||||
|
||||
**SHA-256** comes from the system OpenSSL 3 libcrypto EVP interface. It is linked
|
||||
only into the Linux `evo::region_store` adapter. libpq TLS already requires that
|
||||
library in the same images. Locally observed: Ubuntu `libssl3t64`/`libssl-dev`
|
||||
3.0.13-0ubuntu3.15, Apache-2.0. PostgreSQL's built-in `sha256()` independently
|
||||
computes commit digests, and the host checks that the receipt matches.
|
||||
|
||||
**Object store**: a filesystem store on a trusted mounted volume (Linux only).
|
||||
- Each region has its own `w<world>/r<region>` directory with mode 0700. Objects
|
||||
are named by the hex SHA-256 of their content.
|
||||
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
|
||||
link never replaces an existing name), and then fsyncs the directory.
|
||||
- Existing identical content is reused. Differing content already under a key is
|
||||
reported as corrupt and never overwritten.
|
||||
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
|
||||
and verify the digest.
|
||||
- There is no delete or GC operation: unreferenced uploads are orphans.
|
||||
|
||||
**Publication order**:
|
||||
1. Validate the candidate the way recovery would: full envelope restore,
|
||||
identity, tick and quantum checks, and a byte-identical re-encode.
|
||||
2. Upload the payload.
|
||||
3. Read it back and verify it.
|
||||
4. Run the fenced transaction.
|
||||
5. Compare the receipt with the local manifest digest.
|
||||
|
||||
Only step 4 has authority. **Recovery** loads the head and manifest, re-digests
|
||||
the manifest, requires the head fields to match and the build to be in the host's
|
||||
retained allowlist, fetches and verifies the object, and fully restores it. It
|
||||
never creates genesis or selects an older head. A missing or corrupt payload
|
||||
quarantines the region.
|
||||
|
||||
## Alternatives Considered
|
||||
- **S3/MinIO clients** add a large network dependency before any measured need.
|
||||
The interface boundary allows one later, with a separate ADR.
|
||||
- **Payload bytes in PostgreSQL** was deferred by ADR-0058 until measured.
|
||||
- **Host-supplied manifest fields as SQL parameters** could disagree with the
|
||||
digested bytes.
|
||||
- **A project-written SHA-256** is forbidden by the constitution.
|
||||
- **rename()** can silently replace an existing object.
|
||||
|
||||
## Benefits
|
||||
The committed head is always backed by a verified, immutable payload whose
|
||||
identity, chain, lease and versions the database has checked. Tampering is
|
||||
blocked by constraints and detected again on recovery. Crashes before the
|
||||
transaction leave the previous head.
|
||||
|
||||
## Costs
|
||||
- A full payload is written per batch. The local fixture's payload is about 23-37
|
||||
KiB, and publication took about 3.5-6.6 ms in non-sanitizer builds.
|
||||
- Orphans accumulate.
|
||||
- The existing v2 regions must be re-provisioned.
|
||||
- Recovery fetches whole payloads.
|
||||
|
||||
## Risks
|
||||
- A single volume is not host-loss durability (M8).
|
||||
- `fsync` semantics depend on the filesystem and mount options. The local test
|
||||
volume does not prove power-loss behavior.
|
||||
- The database cannot validate payload contents: a compromised worker holding a
|
||||
valid lease can publish a self-consistent but illegitimate state (ADR-0058).
|
||||
- The read-back verification path is not exercised by any fault injection.
|
||||
|
||||
## Consequences
|
||||
M7.4b adds the lineage archive codec, organism-to-genome binding and a separately
|
||||
versioned history-capable profile. M7.5 adds decisions, outbox and commit-ID
|
||||
resolution to the same transaction. Operators insert each promoted image digest
|
||||
into `retained_builds` before hosts publish with it, and hosts restore only builds
|
||||
in their own allowlist.
|
||||
@@ -71,3 +71,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
|
||||
- [ADR-0059: PostgreSQL bootstrap](0059-postgresql-bootstrap.md) - accepted for M7.2 optional adapter, pinned server/client, migrations and Secret boundary.
|
||||
- [ADR-0060: Fenced region ownership](0060-fenced-region-ownership.md) - accepted for M7.3 database-time leases, guarded head advance, runtime grants and selected timings.
|
||||
- [ADR-0061: Checkpoint publication](0061-checkpoint-publication.md) - accepted for M7.4a manifest v1, immutable filesystem objects, OpenSSL SHA-256 and verified recovery.
|
||||
|
||||
@@ -168,3 +168,17 @@ rejected, and the head is unchanged. A paused stale worker resuming past its lea
|
||||
is fenced both locally and by the database. A writer queued on the lease lock past
|
||||
expiry is also rejected. These are in-process pauses, not killed pods or network
|
||||
partitions. The 7.6 parts of those rows remain open.
|
||||
|
||||
M7.4a (ADR-0061) executed its rows for environmental-v1 against real PostgreSQL and
|
||||
a local filesystem object store:
|
||||
- A partial upload leaves only an unresolvable temporary name.
|
||||
- A failed upload sends no manifest.
|
||||
- A verified upload whose transaction is rejected stays an unreferenced orphan,
|
||||
and recovery returns the previous head.
|
||||
- Missing, corrupt, symlinked or resized payloads quarantine the region.
|
||||
- Unsupported builds are refused by both the database and recovery.
|
||||
- Recovery never creates genesis or picks an older head.
|
||||
|
||||
Two parts remain open. M7.4b adds the lineage archive codec and a history-capable
|
||||
profile; until then the history rows of this audit stay open. No process-kill
|
||||
test was run yet (7.7).
|
||||
|
||||
@@ -2103,3 +2103,58 @@ Only clean-to-v1 and exact-v1 restart are supported migration paths at this gate
|
||||
overflow (`9223372036854775807 + 2000`).
|
||||
- Reran Windows CTest 55/55, the repository format check, and GCC14 Release,
|
||||
clang-tidy analysis and ASan/UBSan private-PostgreSQL suites: all passed.
|
||||
|
||||
## M7.4a checkpoint publication - 2026-09-28
|
||||
|
||||
- Windows MSVC Release configure/build/CTest passed 56/56, including the new
|
||||
portable `region-checkpoint-contracts`. It covers:
|
||||
- golden byte offsets and round trips;
|
||||
- malformed and unsupported manifests, including u32 values above 255 that
|
||||
must not truncate onto supported enums;
|
||||
- chain rules;
|
||||
- payload identity, tick, quantum, corruption and budget checks;
|
||||
- 24-tick continuation equality after restore.
|
||||
The repository clang-format 19 check passed.
|
||||
- WSL Ubuntu, with GCC 14.2.0, clang 19.1.1, PostgreSQL/libpq 18.6 and OpenSSL
|
||||
3.0.13-0ubuntu3.15, running as uid 1000 so permission faults were really
|
||||
exercised. GCC14 Release, clang19 analysis (clang-tidy warnings-as-errors) and
|
||||
clang19 ASan/UBSan each passed `tools/ci/postgres-check.sh`.
|
||||
- Analysis initially flagged `performance-enum-size` on the u32-based
|
||||
profile/durability enums. They are now u8 with explicit decoding of values
|
||||
above 255 as unknown. All three configurations were rerun.
|
||||
- New real-database/filesystem coverage:
|
||||
- SHA-256 vectors (NIST "", "abc" and the 448-bit message) and agreement with
|
||||
PostgreSQL `sha256()`.
|
||||
- Object store: rejects root symlinks, creates 0700 namespaces, puts are
|
||||
idempotent, never overwrites differing content, ignores temporary leftovers,
|
||||
and detects byte flips, symlinks, truncation and growth. Size bounds and
|
||||
write-permission failure are also covered.
|
||||
- Genesis provisioning and duplicate refusal; three batches (16/64/16 ticks);
|
||||
a 65-tick batch refused before upload.
|
||||
- Recovery from a new store handle is byte-identical and continues 44 ticks
|
||||
identically to an uninterrupted reference.
|
||||
- The build allowlist is enforced by the host and by the database.
|
||||
- Upload failure leaves the head unchanged. A fenced publication leaves a
|
||||
verified orphan, and recovery ignores it.
|
||||
- Missing and corrupt payloads quarantine; restoring the object recovers.
|
||||
- Owner edits to a manifest (23514) and head repointing (23503) are refused.
|
||||
With the constraint dropped, recovery still detects the edit, and the
|
||||
constraint is restored afterwards.
|
||||
- Unknown regions are never created.
|
||||
- Lease tests were ported to manifest publication. They additionally cover
|
||||
malformed manifests, a quantum change, an unretained build, a future rules
|
||||
version and a globally reused commit ID.
|
||||
- Migrations: clean→v3, repeated no-op, v1→v3. v2 with existing heads fails
|
||||
closed and stays at v2. The audit rejects EXECUTE on provisioning and SELECT on
|
||||
commits. Tampered v2/v3 checksums and an unknown v4 are refused.
|
||||
- Mutations, run on scratch copies with the source restored afterwards: removing
|
||||
the predecessor-digest check, the lease check, the version/quantum continuity
|
||||
check or the retained-build check in SQL, or the host manifest re-digest, each
|
||||
failed at the expected assertion.
|
||||
- Local fixture measurements: payloads of 22775 and 37367 bytes.
|
||||
- Release publication took 3.6-4.8 ms per batch; analysis 5.5-6.7 ms; ASan
|
||||
about 10.5 ms.
|
||||
- Manifest publish p50 was about 0.74-0.91 ms.
|
||||
These are local socket and local disk only.
|
||||
- Not executed: Docker/Gitea CI, a remote object store or PVC, power-loss fsync
|
||||
semantics, read-back verification fault injection, process kill, and TSan.
|
||||
|
||||
@@ -68,7 +68,7 @@ every role transitively in it, and rolls back on any violation:
|
||||
`pg_write_all_data`);
|
||||
- any table privilege in `evolution`, including via PUBLIC;
|
||||
- CREATE on the schema, the database or `public`;
|
||||
- EXECUTE outside the five fenced functions.
|
||||
- EXECUTE outside the six fenced functions (schema v3).
|
||||
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
||||
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
||||
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
||||
@@ -77,20 +77,41 @@ never creates logins or passwords, and never grants superuser.
|
||||
|
||||
## Fenced region ownership (schema v2, ADR-0060)
|
||||
|
||||
The runtime role has schema USAGE and EXECUTE on five SECURITY DEFINER functions:
|
||||
`acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
|
||||
`load_region_head` and `advance_region_head`. It cannot read or write tables, create
|
||||
objects, call `check_region_lease` or provision regions. Each mutation locks the lease row
|
||||
In schema v3 the runtime role has schema USAGE and EXECUTE on six SECURITY DEFINER
|
||||
functions: `acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
|
||||
`load_region_head`, `publish_region_checkpoint` and `load_region_checkpoint`. It
|
||||
cannot read or write tables, create objects, call internal check/parse functions or
|
||||
provision regions. (Schema v2's `advance_region_head` is dropped in v3.) Each mutation locks the lease row
|
||||
and checks holder, epoch and expiry with `clock_timestamp()` after the lock wait.
|
||||
Clients never send times. Rejections: EVL01 busy, EVL02 fenced, EVL03 head mismatch,
|
||||
EVL04 not provisioned, EVL05 epoch/sequence exhausted, 22023 invalid arguments.
|
||||
`evo::postgres` maps them to `LeaseStoreError`; a timeout or broken connection is
|
||||
`unknown`.
|
||||
EVL04 not provisioned, EVL05 epoch/sequence exhausted, EVL06 unsupported
|
||||
profile/version/build or a changed rules/quantum, EVL07 reused commit ID, and 22023
|
||||
malformed arguments or manifest. `evo::postgres` maps them to `LeaseStoreError`; a
|
||||
timeout or broken connection is `unknown`.
|
||||
|
||||
Genesis is operator-only (owner login), never done by recovery or runtime:
|
||||
`SELECT evolution.provision_region(world, region, frame, genesis_commit_uuid,
|
||||
'\x<64 hex digest>');`. In schema v2 the head is an identity/digest pointer with no
|
||||
payload binding. Do not provision real world state before M7.4 publication.
|
||||
## Checkpoint publication (schema v3, ADR-0061)
|
||||
|
||||
Each commit row stores the exact 280-byte manifest, and a constraint ties the
|
||||
commit digest to `sha256(manifest)`. The head has a composite foreign key to its
|
||||
commit, so even the owner cannot silently repoint or edit it.
|
||||
|
||||
Payloads are immutable files under a trusted object root:
|
||||
`<root>/w<world>/r<region>/<sha256 hex>`, mode 0700 directories and 0600 files.
|
||||
Mount a dedicated volume that only the region host's user can write. Nothing is
|
||||
ever deleted; orphans from failed transactions are expected.
|
||||
|
||||
Operator steps:
|
||||
1. Record each promoted build (image digest, 32 bytes) before a host uses it:
|
||||
`INSERT INTO evolution.retained_builds(build_identity) VALUES ('\x<64 hex>');`
|
||||
2. Provision genesis only through `provision_environmental_region` on an owner
|
||||
connection. It uploads and verifies the tick-0 payload, then calls the
|
||||
owner-only `provision_region_checkpoint(manifest)`.
|
||||
|
||||
There is no operator CLI for provisioning yet, and no real world has been
|
||||
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
||||
have no payload.
|
||||
|
||||
## Host lease discipline
|
||||
|
||||
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
|
||||
interval, 2 s safety margin, 2 s renewal deadline and 5 s commit deadline. The
|
||||
|
||||
Reference in New Issue
Block a user