fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)

The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-01 06:14:34 -04:00
co-authored by Claude Opus 5.5
parent 3a34ae39bc
commit 68da5654ff
5 changed files with 386 additions and 145 deletions
+42 -16
View File
@@ -173,21 +173,47 @@ every PostgreSQL build:
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
chosen; a directory and `tar` need no new dependency.
## Amendment: restore ownership (M8 review, 2026-09-30)
The review reproduced a race. A restore checked that the target was empty, a
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
The first review reproduced a race. A restore checked that the target was empty, a
competing actor then created the schema, the restore's `pg_restore` failed, and
its cleanup dropped the other actor's schema. Restores now own what they touch:
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
refused and never touched.
- One psql session, held open for the whole restore, takes the restore claim
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
that database wait until the restore is verified or removed.
- Cleanup removes only what the invocation created: the schema only if its own
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
created it.
its cleanup dropped the other actor's schema. The first fix held advisory locks in
an auxiliary session. The follow-up review showed two remaining problems:
- losing that session released authority before verification;
- a nonzero exit was not proof of rollback.
`backup-check.sh` covers three cases:
- a competing schema created just before `pg_restore` survives;
- a concurrent restore refuses, and the first completes;
- a lease acquisition times out while the restore holds the database.
Ownership and the serving gate are now durable state in the target itself.
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
explicit COMMIT only after the whole script). It contains:
- the dump;
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
builds it held current)`;
- every lease fenced, and every current build retired.
From that commit on, acquisition (both forms), renewal and commits refuse,
whatever happens to any connection or to the server.
- **Publish** runs after the head comparison, the full audit and (in place) the
rights audit. It is one transaction that first locks the invocation's unpublished
claim, then makes exactly the held builds current again. A restore that lost its
claim cannot publish. The claim row stays as a record of the restore.
- **Resolution.** When an install exits nonzero, the claim row decides:
- committed: verification continues;
- absent: rolled back, and the database is reported unchanged (confirmed);
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
later.
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
unpublished claim. It removes the objects directory only while that directory
carries this NONCE's marker. A competing restore's or provisioner's state is
never touched.
`backup-check.sh` covers:
- a competing schema created before install (rollback confirmed; the competitor
survives);
- two concurrent restores (the first installer wins; the loser rolls back and
leaves the winner published);
- acquisition refused after install, both before and after a database restart,
with the build current again after publication;
- a lost install acknowledgement (resolved as committed, then verified);
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
right one removes everything, and a retry succeeds);
- an occupied objects path.
+3 -1
View File
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
bundle under the world's own identity into a replacement database:
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
- It fences every restored lease, so the next owner acquires a newer epoch.
- It fences every restored lease, so the next owner acquires a newer epoch. The
world stays unservable (every build retired) until verification publishes it
(ADR-0074 amendment).
- It re-runs the migration rights audit for the environment's runtime role.
- It prints a `REWIND` line per region naming the tick it was restored to.
+46
View File
@@ -3087,3 +3087,49 @@ Review: `.cache/reviews/M8_REVIEW_680e7e5-5b482ad.md` (one P1, five P2).
objects, 0 failures.
- ROADMAP corrected: M8-dev/stage is implemented and locally qualified, but **not
passed** until the live gates listed there are run.
## Review 3a34ae3 follow-up: durable restore gate - 2026-10-01
Review: `.cache/reviews/REVIEW_3a34ae3.md`. P1: losing the auxiliary claim session
released authority before verification. P2: a nonzero `pg_restore` exit was taken
as proof of rollback.
- `restore-world.sh` no longer relies on an auxiliary lock session. It works in
three steps, as the ADR-0074 amendment describes:
- an install transaction holds the dump, a claim row with the invocation's
nonce, fenced leases and retired builds;
- verification follows;
- a nonce-checked publish transaction re-enables exactly the held builds.
Ambiguous outcomes are resolved from the claim row. An unresolvable one is
OUTCOME UNKNOWN (exit 5): the target stays unservable, and
`--abandon NONCE --objects DIR` resolves it later.
- `backup-check.sh` passed in the Release, analysis and ASan/UBSan PostgreSQL
builds, with these cases:
- a competing schema before install: the rollback was confirmed and the
competitor's table survived;
- two concurrent restores: the first installer was verified and published; the
second rolled back, confirmed, and left the winner intact;
- after install, `acquire_region_lease` was refused ("build not current") both
before and after `pg_ctl restart` mid-restore; the restore then verified, and
one current build was restored;
- a lost install acknowledgement: "install outcome resolved: committed", then
verified;
- an unreachable resolution: exit 5, both schemas and the objects kept,
acquisition refused, `--abandon` with a wrong nonce removed nothing, the right
nonce removed everything, and a retry restore succeeded;
- an occupied objects path: refused and untouched;
- damaged bundles: refused, with no `evolution` or `evolution_restore` left.
- The reviewer's claim-loss reproduction, adapted to the renamed hook
(`after-install`), finds no lock to terminate. The runtime acquisition inside it
failed with `build not current` (exit 1), and the restore then published.
- `dr-check.sh` (in-place restore) and `upgrade-check.sh` passed in all three
builds.
- Mutations:
- install without retiring builds: caught ("authority was available before
publication");
- resolution ignoring a committed claim: caught (exit 5 instead of verified);
- cleanup ignoring the claim's nonce: NOT exercised (the edit pattern failed to
apply). The abandon path also checks the nonce first, so that defense is two
layers deep and only the outer layer is tested.
- Not run: a real network partition around COMMIT. The lost acknowledgement is an
injected test mode (`EVO_RESTORE_TEST_INSTALL_ACK`).