Compile the PostgreSQL stages with PG_BUILD_JOBS=6 (was 2), including PostgreSQL itself. postgres-check.sh gains --full (release), --sanitizer (ASan: C++ suite plus backup/DR/upgrade process checks that run the instrumented binaries) and --suite-only (clang-tidy build), so SQL-only checks run once. Each run prints per-phase timings, and the postgres job logs runner CPU, memory and disk headroom. Also refresh the dev REGION_SESSION_WINDOW. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
12 KiB
[Reviewed]
ADR-0074: World backup bundles and verified restore
Status
Accepted for M8.7, 2026-09-30. It builds on:
- ADR-0061 (immutable, content-addressed objects; upload verified before commit);
- ADR-0067 (clones and history audit);
- ADR-0070 (dev/stage stores);
- ADR-0073 (synthetic load).
The permanent backup destination, encryption, scheduling and WAL archiving wait on the maintainer's choice of destination (docs/BACKLOG.md). Until then, this ADR covers everything up to that destination.
Context
M8.7 requires:
- a coordinated manifest covering the database, the objects, the versions, the binaries and any in-flight state;
- a restore onto independent storage;
- retention and GC that respect clone and backup pins.
The evidence asked for is:
- rebuilding an isolated world from a backup;
- corruption and missing-object failures;
- credential and key recovery.
The named risk is a backup that exists but cannot reconstruct causal state.
docs/infrastructure/OPERATIONS.md proposes a coordinated admission barrier. Two facts make a barrier unnecessary for a single database:
- a region's durable state is one PostgreSQL database plus immutable objects;
- every object a committed row references was verified durable before that row committed.
Decision
Bundle. tools/backup/backup-world.sh --objects DIR --out DIR writes:
| File | Content |
|---|---|
MANIFEST |
Format, time and label. The server version. Every schema migration and its checksum. Every retained build (image digest) and whether it is retired. Every region head (world, region, frame, sequence, tick, commit, digest, build). Row counts, and the object count and bytes |
database.dump |
pg_dump custom format of schema evolution, with no owners. Grants are kept for an in-place restore (ADR-0075 amendment); every other restore skips them |
objects/w<world>/r<region>/<sha256> |
Every object in the store, each checked against its name while it is copied |
SHA256SUMS |
Checksums of every other file |
The cut. One psql session opens a REPEATABLE READ, READ ONLY transaction and
exports its snapshot. The session reads the heads for the MANIFEST, then runs
pg_dump --snapshot on that same snapshot while the session holds it. Objects are
copied afterwards.
- Objects are never overwritten or deleted, and no GC exists. So every object the snapshot references is present and final when it is copied.
- Objects written after the snapshot are copied as harmless orphans.
- Nothing is written to the source, and admission never stops. The backup runs online while the host commits.
In-flight state.
- Every acknowledgement made before the snapshot is in the bundle, because an acknowledgement follows its commit.
- A commit whose transaction completes after the snapshot is not in the bundle. It was not acknowledged at the cut.
- Unacknowledged inputs are never durable, by design (ADR-0064).
- Lease rows are copied, but a restored copy is never served under them (see below).
- There are no cross-region transfers yet. When they exist, they live in the same database and are covered by the same snapshot.
Restore. tools/backup/restore-world.sh --bundle B --objects DIR targets an
empty database owned by the restoring login, and fails closed in this order:
- Every file matches
SHA256SUMS, no file is unlisted, and every object's name is its digest. Nothing is touched before this passes. - The target has no
evolutionschema and runs the manifest's server version. pg_restoreruns in one transaction, then the objects are copied.- The restored heads equal the MANIFEST heads exactly.
- Every region's full history is audited with
evo-world-clone audit. The audit checks every link's manifest, payload, trace replay and outcomes, from genesis to the head.
A failure after step 2 removes the restored schema and objects, so an unverified
restore cannot be mistaken for a world. The result is a verbatim, owner-only copy
that keeps the source's world identities. It has no runtime grants and is never
served. (ADR-0075 adds --in-place for dev/stage host-loss recovery.)
Isolated rebuild. tools/backup/verify-backup.sh BUNDLE --isolate WORLD:
- Starts a private, throwaway PostgreSQL cluster on local disk.
- Restores the bundle as a fresh login that owns only the restore database.
- Creates a separate database with its own owner, runtime group and login, and
migrates it with
migrate-postgres.sh. - Clones every region of the source world into
WORLDwithevo-world-clone clone, which audits the source history first. Provenance is kept inregion_origins. - Runs
evo-serveron each clone under synthetic load until it acknowledges 20 ticks past the backup head, then drains it.
This is OPERATIONS.md steps 3–5 for a clone. Restoring in place under the source's own identity, with the old writers fenced, is host-loss recovery and belongs to M8.8.
Credentials and keys.
- A bundle carries no roles, passwords or connection details. Since ADR-0075 its
dump keeps grants (role names only), for an in-place restore; the MANIFEST says
dump privileges. - Restoring needs only a login that owns an empty database, and no source
credential.
backup-check.shandverify-backup.shprove this with fresh, unrelated roles. - Bundles are not encrypted yet, so there is no key to lose. Encryption and its key custody are chosen together with the destination. They are required before any World0 data leaves the host.
Retention and GC.
- Each bundle is a full, self-contained copy, and no bundle depends on another. Deleting one never breaks another.
- Clones copy their payloads into the target namespace (ADR-0067), so a clone does not depend on its source either.
- The object store has no delete, and history never expires.
- Any future GC must prove reachability from every live head, clone origin and retained build. Bundles hold their own copies, so GC cannot damage them.
- Retention of bundles at the destination (the OPERATIONS proposal: 30 daily, 12 weekly, 12 monthly) is applied when the destination exists.
Binaries. The MANIFEST names every build identity that wrote history. These are the region image digests that recovery and replay need. Harbor must keep those images; its retention policy is not yet verified (backlog).
In the cluster. tools/backup/cluster-backup.sh <dev|stage> --to DIR [--verify WORLD]
is an operator action, like fault-drill.sh, and refuses World0 before kubectl
runs. It works as follows:
- It runs one short-lived Pod (component
backup) with:- the environment's pinned postgres-tools image, which now includes
pg_dump,pg_restoreandbackup-world.sh; - the migrator login;
- the
aevolution-objectsvolume mounted read-only on the store node; - a 12 GiB emptyDir for the bundle.
- the environment's pinned postgres-tools image, which now includes
- The database egress policy admits the
backupcomponent. - The bundle is streamed out with
kubectl exec tarand its checksums are checked locally. Then the Pod is deleted, including on failure.
A scheduled CronJob, synced by Argo, follows once the destination exists.
Qualification. tools/ci/backup-check.sh runs inside postgres-check.sh for
the release and ASan/UBSan PostgreSQL builds (the uninstrumented clang-tidy build runs
only the C++ suite):
- It backs up a real
evo-serverwhile the server commits under synthetic load. The backup head covers every acknowledgement made before the backup, and the source then moves past it. - It runs
verify-backup.sh --isolateon a second cluster. - It restores damaged bundles, each of which must be refused, leaving nothing behind: a corrupt object, an unlisted file, a damaged dump, a missing payload that has consistent checksums, and a MANIFEST head that disagrees with the database.
- It restores a second time into a database that already holds a world, which must be refused.
Consequences
- RPO. The recovery point is the last verified bundle. Without WAL archiving, this is a logical backup and cannot recover to an arbitrary point in time. Continuous archiving on pg-forge-01 needs the destination (backlog). M8.8 decides the host-loss RPO for World0.
- Full copies. Every bundle copies the whole object store. That is fine at dev/stage sizes (the objects volume is 10 GiB). Incremental object copies, which are safe because objects are content-addressed, come with the destination.
- Off host. A bundle is off the host only if
--tois on another machine. The maintainer's workstation qualifies only if it is not the hypervisor host. - Restore test. A scheduled job or a successful upload is not a restore test.
Only
verify-backup.shoutput counts as restore evidence.
Alternatives rejected
- Physical base backups (
pg_basebackup) as the only copy. They restore the whole server, with its roles, and cannot be audited per region or restored as an isolated world without a second step. They remain the likely basis for PITR once WAL archiving exists. - An admission barrier before the cut. It is unnecessary given upload-before-commit and immutable objects, and it would pause the world.
- Restoring directly as a runnable world with fresh grants. It would serve the source's world identity from a second place, which could become competing authority. Isolation goes through the clone path, which re-identifies the world.
- A new object-storage dependency (MinIO, S3 SDK). The destination is not
chosen; a directory and
tarneed no new dependency.
Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
The first review reproduced a race. A restore checked that the target was empty, a
competing actor then created the schema, the restore's pg_restore failed, and
its cleanup dropped the other actor's schema. The first fix held advisory locks in
an auxiliary session. The follow-up review showed two remaining problems:
- losing that session released authority before verification;
- a nonzero exit was not proof of rollback.
Ownership and the serving gate are now durable state in the target itself.
-
Install is one transaction (the
pg_restoreSQL, sent to psql with an explicit COMMIT only after the whole script). It contains:- the dump;
- a claim row
evolution_restore.claims(NONCE, unpublished, mode, manifest digest, builds it held current); - every lease fenced, and every current build retired.
From that commit on, acquisition (both forms), renewal and commits refuse, whatever happens to any connection or to the server.
-
Publish runs after the head comparison, the full audit and (in place) the rights audit. It is one transaction that first locks the invocation's unpublished claim, then makes exactly the held builds current again. A restore that lost its claim cannot publish. The claim row stays as a record of the restore.
-
Resolution. When an install exits nonzero, the claim row decides:
- committed: verification continues;
- absent: rolled back, and the database is reported unchanged (confirmed);
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
is removed, and
restore-world.sh --abandon NONCE --objects DIRresolves it later.
-
Cleanup removes schemas only through a transaction that locks this NONCE's unpublished claim. It removes the objects directory only while that directory carries this NONCE's marker. A competing restore's or provisioner's state is never touched.
backup-check.sh covers:
- a competing schema created before install (rollback confirmed; the competitor survives);
- two concurrent restores (the first installer wins; the loser rolls back and leaves the winner published);
- acquisition refused after install, both before and after a database restart, with the build current again after publication;
- a lost install acknowledgement (resolved as committed, then verified);
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the right one removes everything, and a retry succeeds);
- an occupied objects path.