fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)

The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-01 06:14:34 -04:00
co-authored by Claude Opus 5.5
parent 3a34ae39bc
commit 68da5654ff
5 changed files with 386 additions and 145 deletions
+42 -16
View File
@@ -173,21 +173,47 @@ every PostgreSQL build:
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
chosen; a directory and `tar` need no new dependency.
## Amendment: restore ownership (M8 review, 2026-09-30)
The review reproduced a race. A restore checked that the target was empty, a
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
The first review reproduced a race. A restore checked that the target was empty, a
competing actor then created the schema, the restore's `pg_restore` failed, and
its cleanup dropped the other actor's schema. Restores now own what they touch:
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
refused and never touched.
- One psql session, held open for the whole restore, takes the restore claim
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
that database wait until the restore is verified or removed.
- Cleanup removes only what the invocation created: the schema only if its own
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
created it.
its cleanup dropped the other actor's schema. The first fix held advisory locks in
an auxiliary session. The follow-up review showed two remaining problems:
- losing that session released authority before verification;
- a nonzero exit was not proof of rollback.
`backup-check.sh` covers three cases:
- a competing schema created just before `pg_restore` survives;
- a concurrent restore refuses, and the first completes;
- a lease acquisition times out while the restore holds the database.
Ownership and the serving gate are now durable state in the target itself.
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
explicit COMMIT only after the whole script). It contains:
- the dump;
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
builds it held current)`;
- every lease fenced, and every current build retired.
From that commit on, acquisition (both forms), renewal and commits refuse,
whatever happens to any connection or to the server.
- **Publish** runs after the head comparison, the full audit and (in place) the
rights audit. It is one transaction that first locks the invocation's unpublished
claim, then makes exactly the held builds current again. A restore that lost its
claim cannot publish. The claim row stays as a record of the restore.
- **Resolution.** When an install exits nonzero, the claim row decides:
- committed: verification continues;
- absent: rolled back, and the database is reported unchanged (confirmed);
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
later.
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
unpublished claim. It removes the objects directory only while that directory
carries this NONCE's marker. A competing restore's or provisioner's state is
never touched.
`backup-check.sh` covers:
- a competing schema created before install (rollback confirmed; the competitor
survives);
- two concurrent restores (the first installer wins; the loser rolls back and
leaves the winner published);
- acquisition refused after install, both before and after a database restart,
with the build current again after publication;
- a lost install acknowledgement (resolved as committed, then verified);
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
right one removes everything, and a retry succeeds);
- an occupied objects path.
+3 -1
View File
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
bundle under the world's own identity into a replacement database:
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
- It fences every restored lease, so the next owner acquires a newer epoch.
- It fences every restored lease, so the next owner acquires a newer epoch. The
world stays unservable (every build retired) until verification publishes it
(ADR-0074 amendment).
- It re-runs the migration rights audit for the environment's runtime role.
- It prints a `REWIND` line per region naming the tick it was restored to.