118 lines
6.0 KiB
Markdown
118 lines
6.0 KiB
Markdown
[Reviewed]
|
|
# ADR-0061: Checkpoint publication and verified recovery
|
|
|
|
## Status
|
|
Accepted for M7.4a, 2026-09-28. Covers the `environmental-v1` profile only. The
|
|
lineage archive codec and history-capable profile binding are M7.4b; M7.4 is not
|
|
complete until both parts pass. Decisions, outbox and commit resolution remain M7.5.
|
|
|
|
## Context
|
|
ADR-0058 selects full checkpoint batches: payload bytes go to an immutable object,
|
|
then a fenced database transaction makes them the committed head. ADR-0060 fences
|
|
the head, but its v2 kernel accepts any 32-byte digest with no payload behind it.
|
|
ADR-0059 deferred the SHA-256 provider. No object-store dependency has been selected.
|
|
The M6 environmental envelope is complete for its profile and already validates
|
|
itself fully on restore.
|
|
|
|
## Decision
|
|
**Manifest v1** is a fixed 280-byte little-endian record (schemas/region-commit.md)
|
|
holding:
|
|
- identity, commit ID and sequence, predecessor ID and digest, and the tick range;
|
|
- the lease holder and epoch, and the tick quantum;
|
|
- nine pinned versions and the retained build identity;
|
|
- explicit declarations that there are zero RNG streams, no history profile, and a
|
|
single-volume development durability policy;
|
|
- the payload length and SHA-256.
|
|
|
|
The commit digest is SHA-256 of these bytes. The database parses the same bytes
|
|
itself, so no separately supplied field can disagree with the digested record.
|
|
|
|
**Schema v3** makes the following changes:
|
|
- It adds `region_commits`. A CHECK constraint ties each commit digest to its
|
|
manifest, and a composite foreign key requires the head to name an existing
|
|
commit with the same ID and digest.
|
|
- It adds owner-managed `retained_builds`.
|
|
- It adds `publish_region_checkpoint(manifest)`, in the same order as the v2 kernel:
|
|
lease row lock, database-time lease check, then head lock. It requires the
|
|
exact successor of the committed head, with unchanged versions and tick quantum,
|
|
a retained build, and a unique commit ID.
|
|
- It adds `load_region_checkpoint` and owner-only `provision_region_checkpoint`
|
|
for genesis.
|
|
- It drops the v2 kernel and v2 provisioning functions.
|
|
- It refuses to migrate a database that still has v2 heads, because those have no
|
|
payload binding. Such regions must be re-provisioned.
|
|
|
|
The runtime role can execute exactly six functions.
|
|
|
|
**SHA-256** comes from the system OpenSSL 3 libcrypto EVP interface. It is linked
|
|
only into the Linux `evo::region_store` adapter. libpq TLS already requires that
|
|
library in the same images. Locally observed: Ubuntu `libssl3t64`/`libssl-dev`
|
|
3.0.13-0ubuntu3.15, Apache-2.0. PostgreSQL's built-in `sha256()` independently
|
|
computes commit digests, and the host checks that the receipt matches.
|
|
|
|
**Object store**: a filesystem store on a trusted mounted volume (Linux only).
|
|
- Each region has its own `w<world>/r<region>` directory with mode 0700. Objects
|
|
are named by the hex SHA-256 of their content.
|
|
- Upload writes a private temporary file, fsyncs it, `link()`s it to the key (a
|
|
link never replaces an existing name), and then fsyncs the directory.
|
|
- Existing identical content is reused only after its file and directory are
|
|
fsynced again, whether it came from an earlier attempt whose directory sync failed
|
|
or from a concurrent installer. A visible name alone is not durability (review
|
|
fix). Namespace directories sync their parent on every open, not only on
|
|
creation. Differing content already under a key is reported as corrupt and never
|
|
overwritten.
|
|
- Reads use `O_NOFOLLOW`, check an exact-length regular file before allocating,
|
|
and verify the digest.
|
|
- There is no delete or GC operation: unreferenced uploads are orphans.
|
|
|
|
**Publication order**:
|
|
1. Validate the candidate the way recovery would: full envelope restore,
|
|
identity, tick and quantum checks, and a byte-identical re-encode.
|
|
2. Upload the payload.
|
|
3. Read it back and verify it.
|
|
4. Run the fenced transaction.
|
|
5. Compare the receipt with the local manifest digest.
|
|
|
|
Only step 4 has authority. **Recovery** loads the head and manifest, re-digests
|
|
the manifest, requires the head fields to match and the build to be in the host's
|
|
retained allowlist, fetches and verifies the object, and fully restores it. It
|
|
never creates genesis or selects an older head. A missing or corrupt payload
|
|
quarantines the region.
|
|
|
|
## Alternatives Considered
|
|
- **S3/MinIO clients** add a large network dependency before any measured need.
|
|
The interface boundary allows one later, with a separate ADR.
|
|
- **Payload bytes in PostgreSQL** was deferred by ADR-0058 until measured.
|
|
- **Host-supplied manifest fields as SQL parameters** could disagree with the
|
|
digested bytes.
|
|
- **A project-written SHA-256** is forbidden by the constitution.
|
|
- **rename()** can silently replace an existing object.
|
|
|
|
## Benefits
|
|
The committed head is always backed by a verified, immutable payload whose
|
|
identity, chain, lease and versions the database has checked. Tampering is
|
|
blocked by constraints and detected again on recovery. Crashes before the
|
|
transaction leave the previous head.
|
|
|
|
## Costs
|
|
- A full payload is written per batch. The local fixture's payload is about 23-37
|
|
KiB, and publication took about 3.5-6.6 ms in non-sanitizer builds.
|
|
- Orphans accumulate.
|
|
- The existing v2 regions must be re-provisioned.
|
|
- Recovery fetches whole payloads.
|
|
|
|
## Risks
|
|
- A single volume is not host-loss durability (M8).
|
|
- `fsync` semantics depend on the filesystem and mount options. The local test
|
|
volume does not prove power-loss behavior.
|
|
- The database cannot validate payload contents: a compromised worker holding a
|
|
valid lease can publish a self-consistent but illegitimate state (ADR-0058).
|
|
- The read-back verification path is not exercised by any fault injection.
|
|
|
|
## Consequences
|
|
M7.4b adds the lineage archive codec, organism-to-genome binding and a separately
|
|
versioned history-capable profile. M7.5 adds decisions, outbox and commit-ID
|
|
resolution to the same transaction. Operators insert each promoted image digest
|
|
into `retained_builds` before hosts publish with it, and hosts restore only builds
|
|
in their own allowlist.
|