Files
aevolution/docs/adr/0061-checkpoint-publication.md
T
2026-10-08 12:08:53 -04:00

118 lines
6.0 KiB
Markdown

[Reviewed]
# ADR-0061: Checkpoint publication and verified recovery
## Status
Accepted for M7.4a, 2026-09-28. Covers the `environmental-v1` profile only. The
lineage archive codec and history-capable profile binding are M7.4b; M7.4 is not
complete until both parts pass. Decisions, outbox and commit resolution remain M7.5.
## Context
ADR-0058 selects full checkpoint batches: payload bytes go to an immutable object,
then a fenced database transaction makes them the committed head. ADR-0060 fences
the head, but its v2 kernel accepts any 32-byte digest with no payload behind it.
ADR-0059 deferred the SHA-256 provider. No object-store dependency has been selected.
The M6 environmental envelope is complete for its profile and already validates
itself fully on restore.
## Decision
**Manifest v1** is a fixed 280-byte little-endian record (schemas/region-commit.md)
holding:
- identity, commit ID and sequence, predecessor ID and digest, and the tick range;
- the lease holder and epoch, and the tick quantum;
- nine pinned versions and the retained build identity;
- explicit declarations that there are zero RNG streams, no history profile, and a
single-volume development durability policy;
- the payload length and SHA-256.
The commit digest is SHA-256 of these bytes. The database parses the same bytes
itself, so no separately supplied field can disagree with the digested record.
**Schema v3** makes the following changes:
- It adds `region_commits`. A CHECK constraint ties each commit digest to its
manifest, and a composite foreign key requires the head to name an existing
commit with the same ID and digest.
- It adds owner-managed `retained_builds`.
- It adds `publish_region_checkpoint(manifest)`, in the same order as the v2 kernel:
lease row lock, database-time lease check, then head lock. It requires the
exact successor of the committed head, with unchanged versions and tick quantum,
a retained build, and a unique commit ID.
- It adds `load_region_checkpoint` and owner-only `provision_region_checkpoint`
for genesis.
- It drops the v2 kernel and v2 provisioning functions.
- It refuses to migrate a database that still has v2 heads, because those have no
payload binding. Such regions must be re-provisioned.
The runtime role can execute exactly six functions.
**SHA-256** comes from the system OpenSSL 3 libcrypto EVP interface. It is linked
only into the Linux `evo::region_store` adapter. libpq TLS already requires that
library in the same images. Locally observed: Ubuntu `libssl3t64`/`libssl-dev`
3.0.13-0ubuntu3.15, Apache-2.0. PostgreSQL's built-in `sha256()` independently
computes commit digests, and the host checks that the receipt matches.
**Object store**: a filesystem store on a trusted mounted volume (Linux only).
- Each region has its own `w<world>/r<region>` directory with mode 0700. Objects
are named by the hex SHA-256 of their content.
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
link never replaces an existing name), and then fsyncs the directory.
- Existing identical content is reused only after its file and directory are
fsynced again, whether it came from an earlier attempt whose directory sync failed
or from a concurrent installer. A visible name alone is not durability (review
fix). Namespace directories sync their parent on every open, not only on
creation. Differing content already under a key is reported as corrupt and never
overwritten.
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
and verify the digest.
- There is no delete or GC operation: unreferenced uploads are orphans.
**Publication order**:
1. Validate the candidate the way recovery would: full envelope restore,
identity, tick and quantum checks, and a byte-identical re-encode.
2. Upload the payload.
3. Read it back and verify it.
4. Run the fenced transaction.
5. Compare the receipt with the local manifest digest.
Only step 4 has authority. **Recovery** loads the head and manifest, re-digests
the manifest, requires the head fields to match and the build to be in the host's
retained allowlist, fetches and verifies the object, and fully restores it. It
never creates genesis or selects an older head. A missing or corrupt payload
quarantines the region.
## Alternatives Considered
- **S3/MinIO clients** add a large network dependency before any measured need.
The interface boundary allows one later, with a separate ADR.
- **Payload bytes in PostgreSQL** was deferred by ADR-0058 until measured.
- **Host-supplied manifest fields as SQL parameters** could disagree with the
digested bytes.
- **A project-written SHA-256** is forbidden by the constitution.
- **rename()** can silently replace an existing object.
## Benefits
The committed head is always backed by a verified, immutable payload whose
identity, chain, lease and versions the database has checked. Tampering is
blocked by constraints and detected again on recovery. Crashes before the
transaction leave the previous head.
## Costs
- A full payload is written per batch. The local fixture's payload is about 23-37
KiB, and publication took about 3.5-6.6 ms in non-sanitizer builds.
- Orphans accumulate.
- The existing v2 regions must be re-provisioned.
- Recovery fetches whole payloads.
## Risks
- A single volume is not host-loss durability (M8).
- `fsync` semantics depend on the filesystem and mount options. The local test
volume does not prove power-loss behavior.
- The database cannot validate payload contents: a compromised worker holding a
valid lease can publish a self-consistent but illegitimate state (ADR-0058).
- The read-back verification path is not exercised by any fault injection.
## Consequences
M7.4b adds the lineage archive codec, organism-to-genome binding and a separately
versioned history-capable profile. M7.5 adds decisions, outbox and commit-ID
resolution to the same transaction. Operators insert each promoted image digest
into `retained_builds` before hosts publish with it, and hosts restore only builds
in their own allowlist.