fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)
The restore no longer relies on an auxiliary lock session. One install transaction holds the dump, a nonce claim row, fenced leases and retired builds, so the world is unservable until a nonce-checked publish after verification, regardless of connection loss or restarts. A failed install is resolved from the claim (committed, rolled back, or unknown with the target kept unservable and --abandon NONCE to resolve); cleanup only removes what the nonce owns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -173,21 +173,47 @@ every PostgreSQL build:
|
||||
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
|
||||
chosen; a directory and `tar` need no new dependency.
|
||||
|
||||
## Amendment: restore ownership (M8 review, 2026-09-30)
|
||||
The review reproduced a race. A restore checked that the target was empty, a
|
||||
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
|
||||
The first review reproduced a race. A restore checked that the target was empty, a
|
||||
competing actor then created the schema, the restore's `pg_restore` failed, and
|
||||
its cleanup dropped the other actor's schema. Restores now own what they touch:
|
||||
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
|
||||
refused and never touched.
|
||||
- One psql session, held open for the whole restore, takes the restore claim
|
||||
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
|
||||
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
|
||||
that database wait until the restore is verified or removed.
|
||||
- Cleanup removes only what the invocation created: the schema only if its own
|
||||
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
|
||||
created it.
|
||||
its cleanup dropped the other actor's schema. The first fix held advisory locks in
|
||||
an auxiliary session. The follow-up review showed two remaining problems:
|
||||
- losing that session released authority before verification;
|
||||
- a nonzero exit was not proof of rollback.
|
||||
|
||||
`backup-check.sh` covers three cases:
|
||||
- a competing schema created just before `pg_restore` survives;
|
||||
- a concurrent restore refuses, and the first completes;
|
||||
- a lease acquisition times out while the restore holds the database.
|
||||
Ownership and the serving gate are now durable state in the target itself.
|
||||
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
|
||||
explicit COMMIT only after the whole script). It contains:
|
||||
- the dump;
|
||||
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
|
||||
builds it held current)`;
|
||||
- every lease fenced, and every current build retired.
|
||||
|
||||
From that commit on, acquisition (both forms), renewal and commits refuse,
|
||||
whatever happens to any connection or to the server.
|
||||
- **Publish** runs after the head comparison, the full audit and (in place) the
|
||||
rights audit. It is one transaction that first locks the invocation's unpublished
|
||||
claim, then makes exactly the held builds current again. A restore that lost its
|
||||
claim cannot publish. The claim row stays as a record of the restore.
|
||||
- **Resolution.** When an install exits nonzero, the claim row decides:
|
||||
- committed: verification continues;
|
||||
- absent: rolled back, and the database is reported unchanged (confirmed);
|
||||
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
|
||||
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
|
||||
later.
|
||||
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
|
||||
unpublished claim. It removes the objects directory only while that directory
|
||||
carries this NONCE's marker. A competing restore's or provisioner's state is
|
||||
never touched.
|
||||
|
||||
`backup-check.sh` covers:
|
||||
- a competing schema created before install (rollback confirmed; the competitor
|
||||
survives);
|
||||
- two concurrent restores (the first installer wins; the loser rolls back and
|
||||
leaves the winner published);
|
||||
- acquisition refused after install, both before and after a database restart,
|
||||
with the build current again after publication;
|
||||
- a lost install acknowledgement (resolved as committed, then verified);
|
||||
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
|
||||
right one removes everything, and a retry succeeds);
|
||||
- an occupied objects path.
|
||||
|
||||
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
|
||||
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
|
||||
bundle under the world's own identity into a replacement database:
|
||||
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch.
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch. The
|
||||
world stays unservable (every build retired) until verification publishes it
|
||||
(ADR-0074 amendment).
|
||||
- It re-runs the migration rights audit for the environment's runtime role.
|
||||
- It prints a `REWIND` line per region naming the tick it was restored to.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user