fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)
The restore no longer relies on an auxiliary lock session. One install transaction holds the dump, a nonce claim row, fenced leases and retired builds, so the world is unservable until a nonce-checked publish after verification, regardless of connection loss or restarts. A failed install is resolved from the claim (committed, rolled back, or unknown with the target kept unservable and --abandon NONCE to resolve); cleanup only removes what the nonce owns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -173,21 +173,47 @@ every PostgreSQL build:
|
||||
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
|
||||
chosen; a directory and `tar` need no new dependency.
|
||||
|
||||
## Amendment: restore ownership (M8 review, 2026-09-30)
|
||||
The review reproduced a race. A restore checked that the target was empty, a
|
||||
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
|
||||
The first review reproduced a race. A restore checked that the target was empty, a
|
||||
competing actor then created the schema, the restore's `pg_restore` failed, and
|
||||
its cleanup dropped the other actor's schema. Restores now own what they touch:
|
||||
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
|
||||
refused and never touched.
|
||||
- One psql session, held open for the whole restore, takes the restore claim
|
||||
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
|
||||
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
|
||||
that database wait until the restore is verified or removed.
|
||||
- Cleanup removes only what the invocation created: the schema only if its own
|
||||
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
|
||||
created it.
|
||||
its cleanup dropped the other actor's schema. The first fix held advisory locks in
|
||||
an auxiliary session. The follow-up review showed two remaining problems:
|
||||
- losing that session released authority before verification;
|
||||
- a nonzero exit was not proof of rollback.
|
||||
|
||||
`backup-check.sh` covers three cases:
|
||||
- a competing schema created just before `pg_restore` survives;
|
||||
- a concurrent restore refuses, and the first completes;
|
||||
- a lease acquisition times out while the restore holds the database.
|
||||
Ownership and the serving gate are now durable state in the target itself.
|
||||
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
|
||||
explicit COMMIT only after the whole script). It contains:
|
||||
- the dump;
|
||||
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
|
||||
builds it held current)`;
|
||||
- every lease fenced, and every current build retired.
|
||||
|
||||
From that commit on, acquisition (both forms), renewal and commits refuse,
|
||||
whatever happens to any connection or to the server.
|
||||
- **Publish** runs after the head comparison, the full audit and (in place) the
|
||||
rights audit. It is one transaction that first locks the invocation's unpublished
|
||||
claim, then makes exactly the held builds current again. A restore that lost its
|
||||
claim cannot publish. The claim row stays as a record of the restore.
|
||||
- **Resolution.** When an install exits nonzero, the claim row decides:
|
||||
- committed: verification continues;
|
||||
- absent: rolled back, and the database is reported unchanged (confirmed);
|
||||
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
|
||||
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
|
||||
later.
|
||||
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
|
||||
unpublished claim. It removes the objects directory only while that directory
|
||||
carries this NONCE's marker. A competing restore's or provisioner's state is
|
||||
never touched.
|
||||
|
||||
`backup-check.sh` covers:
|
||||
- a competing schema created before install (rollback confirmed; the competitor
|
||||
survives);
|
||||
- two concurrent restores (the first installer wins; the loser rolls back and
|
||||
leaves the winner published);
|
||||
- acquisition refused after install, both before and after a database restart,
|
||||
with the build current again after publication;
|
||||
- a lost install acknowledgement (resolved as committed, then verified);
|
||||
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
|
||||
right one removes everything, and a retry succeeds);
|
||||
- an occupied objects path.
|
||||
|
||||
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
|
||||
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
|
||||
bundle under the world's own identity into a replacement database:
|
||||
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch.
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch. The
|
||||
world stays unservable (every build retired) until verification publishes it
|
||||
(ADR-0074 amendment).
|
||||
- It re-runs the migration rights audit for the environment's runtime role.
|
||||
- It prints a `REWIND` line per region naming the tick it was restored to.
|
||||
|
||||
|
||||
@@ -3087,3 +3087,49 @@ Review: `.cache/reviews/M8_REVIEW_680e7e5-5b482ad.md` (one P1, five P2).
|
||||
objects, 0 failures.
|
||||
- ROADMAP corrected: M8-dev/stage is implemented and locally qualified, but **not
|
||||
passed** until the live gates listed there are run.
|
||||
|
||||
## Review 3a34ae3 follow-up: durable restore gate - 2026-10-01
|
||||
|
||||
Review: `.cache/reviews/REVIEW_3a34ae3.md`. P1: losing the auxiliary claim session
|
||||
released authority before verification. P2: a nonzero `pg_restore` exit was taken
|
||||
as proof of rollback.
|
||||
- `restore-world.sh` no longer relies on an auxiliary lock session. It works in
|
||||
three steps, as the ADR-0074 amendment describes:
|
||||
- an install transaction holds the dump, a claim row with the invocation's
|
||||
nonce, fenced leases and retired builds;
|
||||
- verification follows;
|
||||
- a nonce-checked publish transaction re-enables exactly the held builds.
|
||||
|
||||
Ambiguous outcomes are resolved from the claim row. An unresolvable one is
|
||||
OUTCOME UNKNOWN (exit 5): the target stays unservable, and
|
||||
`--abandon NONCE --objects DIR` resolves it later.
|
||||
- `backup-check.sh` passed in the Release, analysis and ASan/UBSan PostgreSQL
|
||||
builds, with these cases:
|
||||
- a competing schema before install: the rollback was confirmed and the
|
||||
competitor's table survived;
|
||||
- two concurrent restores: the first installer was verified and published; the
|
||||
second rolled back, confirmed, and left the winner intact;
|
||||
- after install, `acquire_region_lease` was refused ("build not current") both
|
||||
before and after `pg_ctl restart` mid-restore; the restore then verified, and
|
||||
one current build was restored;
|
||||
- a lost install acknowledgement: "install outcome resolved: committed", then
|
||||
verified;
|
||||
- an unreachable resolution: exit 5, both schemas and the objects kept,
|
||||
acquisition refused, `--abandon` with a wrong nonce removed nothing, the right
|
||||
nonce removed everything, and a retry restore succeeded;
|
||||
- an occupied objects path: refused and untouched;
|
||||
- damaged bundles: refused, with no `evolution` or `evolution_restore` left.
|
||||
- The reviewer's claim-loss reproduction, adapted to the renamed hook
|
||||
(`after-install`), finds no lock to terminate. The runtime acquisition inside it
|
||||
failed with `build not current` (exit 1), and the restore then published.
|
||||
- `dr-check.sh` (in-place restore) and `upgrade-check.sh` passed in all three
|
||||
builds.
|
||||
- Mutations:
|
||||
- install without retiring builds: caught ("authority was available before
|
||||
publication");
|
||||
- resolution ignoring a committed claim: caught (exit 5 instead of verified);
|
||||
- cleanup ignoring the claim's nonce: NOT exercised (the edit pattern failed to
|
||||
apply). The abandon path also checks the nonce first, so that defense is two
|
||||
layers deep and only the outer layer is tested.
|
||||
- Not run: a real network partition around COMMIT. The lost acknowledgement is an
|
||||
injected test mode (`EVO_RESTORE_TEST_INSTALL_ACK`).
|
||||
|
||||
Reference in New Issue
Block a user