Files
aevolution/docs/adr/0061-checkpoint-publication.md
2026-10-08 12:08:53 -04:00

6.0 KiB

[Reviewed]

ADR-0061: Checkpoint publication and verified recovery

Status

Accepted for M7.4a, 2026-09-28. Covers the environmental-v1 profile only. The lineage archive codec and history-capable profile binding are M7.4b; M7.4 is not complete until both parts pass. Decisions, outbox and commit resolution remain M7.5.

Context

ADR-0058 selects full checkpoint batches: payload bytes go to an immutable object, then a fenced database transaction makes them the committed head. ADR-0060 fences the head, but its v2 kernel accepts any 32-byte digest with no payload behind it. ADR-0059 deferred the SHA-256 provider. No object-store dependency has been selected. The M6 environmental envelope is complete for its profile and already validates itself fully on restore.

Decision

Manifest v1 is a fixed 280-byte little-endian record (schemas/region-commit.md) holding:

  • identity, commit ID and sequence, predecessor ID and digest, and the tick range;
  • the lease holder and epoch, and the tick quantum;
  • nine pinned versions and the retained build identity;
  • explicit declarations that there are zero RNG streams, no history profile, and a single-volume development durability policy;
  • the payload length and SHA-256.

The commit digest is SHA-256 of these bytes. The database parses the same bytes itself, so no separately supplied field can disagree with the digested record.

Schema v3 makes the following changes:

  • It adds region_commits. A CHECK constraint ties each commit digest to its manifest, and a composite foreign key requires the head to name an existing commit with the same ID and digest.
  • It adds owner-managed retained_builds.
  • It adds publish_region_checkpoint(manifest), in the same order as the v2 kernel: lease row lock, database-time lease check, then head lock. It requires the exact successor of the committed head, with unchanged versions and tick quantum, a retained build, and a unique commit ID.
  • It adds load_region_checkpoint and owner-only provision_region_checkpoint for genesis.
  • It drops the v2 kernel and v2 provisioning functions.
  • It refuses to migrate a database that still has v2 heads, because those have no payload binding. Such regions must be re-provisioned.

The runtime role can execute exactly six functions.

SHA-256 comes from the system OpenSSL 3 libcrypto EVP interface. It is linked only into the Linux evo::region_store adapter. libpq TLS already requires that library in the same images. Locally observed: Ubuntu libssl3t64/libssl-dev 3.0.13-0ubuntu3.15, Apache-2.0. PostgreSQL's built-in sha256() independently computes commit digests, and the host checks that the receipt matches.

Object store: a filesystem store on a trusted mounted volume (Linux only).

  • Each region has its own w<world>/r<region> directory with mode 0700. Objects are named by the hex SHA-256 of their content.
  • Upload writes a private temporary file, fsyncs it, link()s it to the key (a link never replaces an existing name), and then fsyncs the directory.
  • Existing identical content is reused only after its file and directory are fsynced again, whether it came from an earlier attempt whose directory sync failed or from a concurrent installer. A visible name alone is not durability (review fix). Namespace directories sync their parent on every open, not only on creation. Differing content already under a key is reported as corrupt and never overwritten.
  • Reads use O_NOFOLLOW, check an exact-length regular file before allocating, and verify the digest.
  • There is no delete or GC operation: unreferenced uploads are orphans.

Publication order:

  1. Validate the candidate the way recovery would: full envelope restore, identity, tick and quantum checks, and a byte-identical re-encode.
  2. Upload the payload.
  3. Read it back and verify it.
  4. Run the fenced transaction.
  5. Compare the receipt with the local manifest digest.

Only step 4 has authority. Recovery loads the head and manifest, re-digests the manifest, requires the head fields to match and the build to be in the host's retained allowlist, fetches and verifies the object, and fully restores it. It never creates genesis or selects an older head. A missing or corrupt payload quarantines the region.

Alternatives Considered

  • S3/MinIO clients add a large network dependency before any measured need. The interface boundary allows one later, with a separate ADR.
  • Payload bytes in PostgreSQL was deferred by ADR-0058 until measured.
  • Host-supplied manifest fields as SQL parameters could disagree with the digested bytes.
  • A project-written SHA-256 is forbidden by the constitution.
  • rename() can silently replace an existing object.

Benefits

The committed head is always backed by a verified, immutable payload whose identity, chain, lease and versions the database has checked. Tampering is blocked by constraints and detected again on recovery. Crashes before the transaction leave the previous head.

Costs

  • A full payload is written per batch. The local fixture's payload is about 23-37 KiB, and publication took about 3.5-6.6 ms in non-sanitizer builds.
  • Orphans accumulate.
  • The existing v2 regions must be re-provisioned.
  • Recovery fetches whole payloads.

Risks

  • A single volume is not host-loss durability (M8).
  • fsync semantics depend on the filesystem and mount options. The local test volume does not prove power-loss behavior.
  • The database cannot validate payload contents: a compromised worker holding a valid lease can publish a self-consistent but illegitimate state (ADR-0058).
  • The read-back verification path is not exercised by any fault injection.

Consequences

M7.4b adds the lineage archive codec, organism-to-genome binding and a separately versioned history-capable profile. M7.5 adds decisions, outbox and commit-ID resolution to the same transaction. Operators insert each promoted image digest into retained_builds before hosts publish with it, and hosts restore only builds in their own allowlist.