fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)
The restore no longer relies on an auxiliary lock session. One install transaction holds the dump, a nonce claim row, fenced leases and retired builds, so the world is unservable until a nonce-checked publish after verification, regardless of connection loss or restarts. A failed install is resolved from the claim (committed, rolled back, or unknown with the target kept unservable and --abandon NONCE to resolve); cleanup only removes what the nonce owns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -173,21 +173,47 @@ every PostgreSQL build:
|
||||
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
|
||||
chosen; a directory and `tar` need no new dependency.
|
||||
|
||||
## Amendment: restore ownership (M8 review, 2026-09-30)
|
||||
The review reproduced a race. A restore checked that the target was empty, a
|
||||
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
|
||||
The first review reproduced a race. A restore checked that the target was empty, a
|
||||
competing actor then created the schema, the restore's `pg_restore` failed, and
|
||||
its cleanup dropped the other actor's schema. Restores now own what they touch:
|
||||
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
|
||||
refused and never touched.
|
||||
- One psql session, held open for the whole restore, takes the restore claim
|
||||
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
|
||||
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
|
||||
that database wait until the restore is verified or removed.
|
||||
- Cleanup removes only what the invocation created: the schema only if its own
|
||||
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
|
||||
created it.
|
||||
its cleanup dropped the other actor's schema. The first fix held advisory locks in
|
||||
an auxiliary session. The follow-up review showed two remaining problems:
|
||||
- losing that session released authority before verification;
|
||||
- a nonzero exit was not proof of rollback.
|
||||
|
||||
`backup-check.sh` covers three cases:
|
||||
- a competing schema created just before `pg_restore` survives;
|
||||
- a concurrent restore refuses, and the first completes;
|
||||
- a lease acquisition times out while the restore holds the database.
|
||||
Ownership and the serving gate are now durable state in the target itself.
|
||||
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
|
||||
explicit COMMIT only after the whole script). It contains:
|
||||
- the dump;
|
||||
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
|
||||
builds it held current)`;
|
||||
- every lease fenced, and every current build retired.
|
||||
|
||||
From that commit on, acquisition (both forms), renewal and commits refuse,
|
||||
whatever happens to any connection or to the server.
|
||||
- **Publish** runs after the head comparison, the full audit and (in place) the
|
||||
rights audit. It is one transaction that first locks the invocation's unpublished
|
||||
claim, then makes exactly the held builds current again. A restore that lost its
|
||||
claim cannot publish. The claim row stays as a record of the restore.
|
||||
- **Resolution.** When an install exits nonzero, the claim row decides:
|
||||
- committed: verification continues;
|
||||
- absent: rolled back, and the database is reported unchanged (confirmed);
|
||||
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
|
||||
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
|
||||
later.
|
||||
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
|
||||
unpublished claim. It removes the objects directory only while that directory
|
||||
carries this NONCE's marker. A competing restore's or provisioner's state is
|
||||
never touched.
|
||||
|
||||
`backup-check.sh` covers:
|
||||
- a competing schema created before install (rollback confirmed; the competitor
|
||||
survives);
|
||||
- two concurrent restores (the first installer wins; the loser rolls back and
|
||||
leaves the winner published);
|
||||
- acquisition refused after install, both before and after a database restart,
|
||||
with the build current again after publication;
|
||||
- a lost install acknowledgement (resolved as committed, then verified);
|
||||
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
|
||||
right one removes everything, and a retry succeeds);
|
||||
- an occupied objects path.
|
||||
|
||||
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
|
||||
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
|
||||
bundle under the world's own identity into a replacement database:
|
||||
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch.
|
||||
- It fences every restored lease, so the next owner acquires a newer epoch. The
|
||||
world stays unservable (every build retired) until verification publishes it
|
||||
(ADR-0074 amendment).
|
||||
- It re-runs the migration rights audit for the environment's runtime role.
|
||||
- It prints a `REWIND` line per region naming the tick it was restored to.
|
||||
|
||||
|
||||
@@ -3087,3 +3087,49 @@ Review: `.cache/reviews/M8_REVIEW_680e7e5-5b482ad.md` (one P1, five P2).
|
||||
objects, 0 failures.
|
||||
- ROADMAP corrected: M8-dev/stage is implemented and locally qualified, but **not
|
||||
passed** until the live gates listed there are run.
|
||||
|
||||
## Review 3a34ae3 follow-up: durable restore gate - 2026-10-01
|
||||
|
||||
Review: `.cache/reviews/REVIEW_3a34ae3.md`. P1: losing the auxiliary claim session
|
||||
released authority before verification. P2: a nonzero `pg_restore` exit was taken
|
||||
as proof of rollback.
|
||||
- `restore-world.sh` no longer relies on an auxiliary lock session. It works in
|
||||
three steps, as the ADR-0074 amendment describes:
|
||||
- an install transaction holds the dump, a claim row with the invocation's
|
||||
nonce, fenced leases and retired builds;
|
||||
- verification follows;
|
||||
- a nonce-checked publish transaction re-enables exactly the held builds.
|
||||
|
||||
Ambiguous outcomes are resolved from the claim row. An unresolvable one is
|
||||
OUTCOME UNKNOWN (exit 5): the target stays unservable, and
|
||||
`--abandon NONCE --objects DIR` resolves it later.
|
||||
- `backup-check.sh` passed in the Release, analysis and ASan/UBSan PostgreSQL
|
||||
builds, with these cases:
|
||||
- a competing schema before install: the rollback was confirmed and the
|
||||
competitor's table survived;
|
||||
- two concurrent restores: the first installer was verified and published; the
|
||||
second rolled back, confirmed, and left the winner intact;
|
||||
- after install, `acquire_region_lease` was refused ("build not current") both
|
||||
before and after `pg_ctl restart` mid-restore; the restore then verified, and
|
||||
one current build was restored;
|
||||
- a lost install acknowledgement: "install outcome resolved: committed", then
|
||||
verified;
|
||||
- an unreachable resolution: exit 5, both schemas and the objects kept,
|
||||
acquisition refused, `--abandon` with a wrong nonce removed nothing, the right
|
||||
nonce removed everything, and a retry restore succeeded;
|
||||
- an occupied objects path: refused and untouched;
|
||||
- damaged bundles: refused, with no `evolution` or `evolution_restore` left.
|
||||
- The reviewer's claim-loss reproduction, adapted to the renamed hook
|
||||
(`after-install`), finds no lock to terminate. The runtime acquisition inside it
|
||||
failed with `build not current` (exit 1), and the restore then published.
|
||||
- `dr-check.sh` (in-place restore) and `upgrade-check.sh` passed in all three
|
||||
builds.
|
||||
- Mutations:
|
||||
- install without retiring builds: caught ("authority was available before
|
||||
publication");
|
||||
- resolution ignoring a committed claim: caught (exit 5 instead of verified);
|
||||
- cleanup ignoring the claim's nonce: NOT exercised (the edit pattern failed to
|
||||
apply). The abandon path also checks the nonce first, so that defense is two
|
||||
layers deep and only the outer layer is tested.
|
||||
- Not run: a real network partition around COMMIT. The lost acknowledgement is an
|
||||
injected test mode (`EVO_RESTORE_TEST_INSTALL_ACK`).
|
||||
|
||||
+216
-95
@@ -3,53 +3,67 @@
|
||||
#
|
||||
# tools/backup/restore-world.sh --bundle DIR --objects DIR
|
||||
# [--in-place --runtime-role ROLE --accept-rewind]
|
||||
# tools/backup/restore-world.sh --abandon NONCE --objects DIR
|
||||
#
|
||||
# Target connection: PG* environment naming an EMPTY database that the login owns
|
||||
# (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE).
|
||||
# The restore needs no source role, credential or key: owners and grants are not
|
||||
# restored, so the target login owns everything restored. --objects must not
|
||||
# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or
|
||||
# PATH.
|
||||
# The restore needs no source role, credential or key: owners are not restored, and
|
||||
# grants only with --in-place, so the target login owns everything restored.
|
||||
# evo-world-clone comes from EVO_WORLD_CLONE_BIN or PATH.
|
||||
#
|
||||
# Order, failing closed at each step:
|
||||
# 1. every bundle file matches SHA256SUMS, no file is unlisted, and every object's
|
||||
# name is its digest; nothing is touched before this passes;
|
||||
# 2. the targets are claimed for this invocation alone (below), then the target
|
||||
# has no `evolution` schema and runs the manifest's server version;
|
||||
# 3. pg_restore in one transaction, then objects are copied;
|
||||
# 2. --objects is claimed by creating it (mkdir, never -p: an existing path is
|
||||
# refused and never touched) and marked with this invocation's random NONCE; the
|
||||
# target must have neither `evolution` nor `evolution_restore` and must run the
|
||||
# manifest's server version;
|
||||
# 3. INSTALL, one transaction: the dump, a claim row evolution_restore.claims(NONCE,
|
||||
# unpublished), every lease fenced and every current build retired. From that
|
||||
# commit on the world is unservable: acquisition, renewal and commits refuse
|
||||
# until publication, whatever happens to any connection or the server (review
|
||||
# M8 follow-up P1). Then the objects are copied;
|
||||
# 4. the restored heads equal the MANIFEST heads exactly;
|
||||
# 5. every region's full history is audited (evo-world-clone audit: each link's
|
||||
# manifest, payload, trace replay and outcomes, from genesis to the head).
|
||||
# Ownership (review M8 P1): --objects is claimed by creating it (mkdir, never -p: an
|
||||
# existing path is refused and never touched), and the database by one psql session
|
||||
# held open for the whole restore. That session takes the restore claim (advisory
|
||||
# 71420004, try-lock: a concurrent restore of the same database refuses at once) and
|
||||
# the upgrade lock exclusively (71420002: provisioning, acquisition and lease checks
|
||||
# in that database wait until this restore is verified or removed). On failure only
|
||||
# what this invocation created is removed: the schema only if its own pg_restore
|
||||
# committed, the objects directory only if its own mkdir created it. So an
|
||||
# unverified restore can never be mistaken for a world, and a losing restore never
|
||||
# deletes a winner's state. EVO_RESTORE_TEST_HOOK (tests only) is run by bash with
|
||||
# $1 = before-restore (claimed, checked) and after-restore (installed, objects copied). The result is a
|
||||
# verbatim, owner-only copy with the source's world identities: it has no runtime
|
||||
# grants and is never served. To run it, clone it into an isolated world
|
||||
# (tools/backup/verify-backup.sh --isolate).
|
||||
# 5. every region's full history is audited (evo-world-clone audit), and with
|
||||
# --in-place migrate-postgres.sh audits the runtime role's effective rights;
|
||||
# 6. PUBLISH, one transaction that first locks this NONCE's unpublished claim:
|
||||
# exactly the builds the install retired become current again and the claim
|
||||
# becomes published. A restore that lost its claim cannot publish.
|
||||
# A failed or interrupted install is resolved, not assumed (review M8 follow-up P2):
|
||||
# the claim row says whether this invocation's transaction committed. Committed ->
|
||||
# verification continues; absent -> rolled back (the database was not changed);
|
||||
# unreachable -> OUTCOME UNKNOWN (exit 5): the target stays unservable and both the
|
||||
# schema and the objects are kept; resolve it later with --abandon NONCE.
|
||||
# Cleanup only ever removes what this NONCE owns: the schemas only through a
|
||||
# transaction that locks this NONCE's unpublished claim, the objects directory only
|
||||
# while it carries this NONCE's marker. A competing restore's or provisioner's state
|
||||
# is never touched. The default result is a verbatim, owner-only copy with the
|
||||
# source's world identities, no runtime grants and never served; to run it, clone it
|
||||
# into an isolated world (tools/backup/verify-backup.sh --isolate).
|
||||
#
|
||||
# --in-place (M8.8 host-loss recovery for dev/stage, ADR-0075) restores the world
|
||||
# under its own identity so its hosts can serve it again: the dump's grants are
|
||||
# restored (the environment's roles must already exist in the target cluster), every
|
||||
# restored lease is fenced so a new owner acquires under a newer epoch, and
|
||||
# migrate-postgres.sh audits the runtime role's effective rights. Every acknowledgement
|
||||
# after the bundle's cut is undone: deaths may reverse and resources may reappear.
|
||||
# --accept-rewind states that this is acceptable, which it is for dev and stage only;
|
||||
# World0 launch stays blocked until acknowledgements survive host loss (ADR-0075).
|
||||
# Fence the old database with fence-database.sh before any of its hosts can return.
|
||||
# under its own identity so its hosts can serve it again after publication: the
|
||||
# dump's grants are restored (the environment's roles must already exist in the
|
||||
# target cluster). Every acknowledgement after the bundle's cut is undone: deaths may
|
||||
# reverse and resources may reappear. --accept-rewind states that this is
|
||||
# acceptable, which it is for dev and stage only; World0 launch stays blocked until
|
||||
# acknowledgements survive host loss (ADR-0075). Fence the old database with
|
||||
# fence-database.sh before any of its hosts can return.
|
||||
#
|
||||
# Exit: 0 verified and published, 2 usage, 3 verification failed (this restore's
|
||||
# state removed), 4 refused or rolled back (nothing of this restore left),
|
||||
# 5 outcome unknown (target unservable; evidence kept).
|
||||
# Tests only: EVO_RESTORE_TEST_HOOK is run by bash with $1 = before-install,
|
||||
# after-install or before-publish; EVO_RESTORE_TEST_INSTALL_ACK=lost treats a committed
|
||||
# install as failed (an ambiguous completion), =unreachable also makes its resolution
|
||||
# unreachable.
|
||||
set -euo pipefail
|
||||
usage() {
|
||||
echo "usage: $0 --bundle DIR --objects DIR [--in-place --runtime-role ROLE --accept-rewind]" >&2
|
||||
echo " $0 --abandon NONCE --objects DIR" >&2
|
||||
exit 2
|
||||
}
|
||||
bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0
|
||||
bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0 abandon=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--bundle) bundle="${2:-}"; shift 2 ;;
|
||||
@@ -57,10 +71,16 @@ while [ $# -gt 0 ]; do
|
||||
--in-place) in_place=1; shift ;;
|
||||
--runtime-role) runtime_role="${2:-}"; shift 2 ;;
|
||||
--accept-rewind) accept_rewind=1; shift ;;
|
||||
--abandon) abandon="${2:-}"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$bundle" ] && [ -n "$objects" ] || usage
|
||||
[ -n "$objects" ] || usage
|
||||
if [ -n "$abandon" ]; then
|
||||
[[ "$abandon" =~ ^[0-9a-f]{32}$ ]] && [ -z "$bundle" ] && [ "$in_place" = 0 ] || usage
|
||||
else
|
||||
[ -n "$bundle" ] || usage
|
||||
fi
|
||||
if [ "$in_place" = 1 ]; then
|
||||
[ -n "$runtime_role" ] || { echo "--in-place needs --runtime-role" >&2; exit 2; }
|
||||
[ "$accept_rewind" = 1 ] ||
|
||||
@@ -69,10 +89,55 @@ elif [ -n "$runtime_role" ] || [ "$accept_rewind" = 1 ]; then
|
||||
usage
|
||||
fi
|
||||
root="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")"
|
||||
objects="$(realpath -m -- "$objects")"
|
||||
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
|
||||
umask 077
|
||||
begin_ms="$(date +%s%3N)"
|
||||
q() { psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 "$@"; }
|
||||
|
||||
# The claim row of NONCE: none, unpublished, published, or unknown (unreachable).
|
||||
claim_state() {
|
||||
local nonce="$1" exists state
|
||||
exists="$(q -c "SELECT to_regclass('evolution_restore.claims') IS NOT NULL" 2> /dev/null)" || { echo unknown; return; }
|
||||
[ "$exists" = t ] || { echo none; return; }
|
||||
state="$(q -c "SELECT state FROM evolution_restore.claims WHERE nonce = '$nonce'" 2> /dev/null)" ||
|
||||
{ echo unknown; return; }
|
||||
echo "${state:-none}"
|
||||
}
|
||||
# Removes NONCE's unpublished restore: both schemas, in one transaction that first
|
||||
# locks that claim. Anything else (another claim, a published one) is refused.
|
||||
drop_claimed() {
|
||||
q -c "DO \$\$ BEGIN
|
||||
IF to_regclass('evolution_restore.claims') IS NULL THEN
|
||||
RAISE EXCEPTION 'no restore claim';
|
||||
END IF;
|
||||
PERFORM 1 FROM evolution_restore.claims WHERE nonce = '$1' AND state = 'unpublished' FOR UPDATE;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'restore claim % is not unpublished here', '$1';
|
||||
END IF;
|
||||
DROP SCHEMA evolution CASCADE;
|
||||
DROP SCHEMA evolution_restore CASCADE;
|
||||
END \$\$" > /dev/null 2>&1
|
||||
}
|
||||
# Removes the objects directory only while it carries NONCE's marker.
|
||||
drop_objects() {
|
||||
[ -f "$objects/.restore-claim" ] && [ "$(cat "$objects/.restore-claim")" = "$1" ] && rm -rf -- "$objects"
|
||||
}
|
||||
|
||||
if [ -n "$abandon" ]; then
|
||||
state="$(claim_state "$abandon")"
|
||||
case "$state" in
|
||||
unpublished) drop_claimed "$abandon" || { echo "ABANDON FAILED: could not remove claim $abandon" >&2; exit 3; } ;;
|
||||
none) ;;
|
||||
published) echo "ABANDON REFUSED: claim $abandon is published (a verified world)" >&2; exit 4 ;;
|
||||
*) echo "ABANDON FAILED: the target database is unreachable" >&2; exit 5 ;;
|
||||
esac
|
||||
drop_objects "$abandon" || true
|
||||
echo "ABANDONED claim $abandon (database state was $state; objects $([ -e "$objects" ] && echo kept || echo removed))"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
bundle="$(realpath -- "$bundle")"
|
||||
refuse() { echo "RESTORE REFUSED: $*" >&2; exit 4; }
|
||||
|
||||
# 1. Bundle integrity, before anything else is touched.
|
||||
@@ -93,78 +158,112 @@ done <<< "$present"
|
||||
grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format"
|
||||
expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)"
|
||||
[ -n "$expected_heads" ] || refuse "manifest names no region"
|
||||
privileges=(--no-privileges)
|
||||
privileges=(--no-privileges) mode=isolated
|
||||
if [ "$in_place" = 1 ]; then
|
||||
grep -qx 'dump privileges' "$bundle/MANIFEST" || refuse "bundle predates in-place restore (no grants in its dump)"
|
||||
privileges=()
|
||||
privileges=() mode=in-place
|
||||
fi
|
||||
manifest_sha="$(sha256sum "$bundle/MANIFEST")"; manifest_sha="${manifest_sha%% *}"
|
||||
label="$(sed -n 's/^label \([A-Za-z0-9._:-]*\)$/\1/p' "$bundle/MANIFEST" | head -1)"
|
||||
hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; }
|
||||
|
||||
target="${PGDATABASE:-}"
|
||||
[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE"
|
||||
[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; }
|
||||
|
||||
# 2. Claim the targets, then check them under the claim.
|
||||
created_objects=0 installed=0 restored=0 lock_pid="" claim=""
|
||||
# 2. Claim the objects path, then check the target.
|
||||
nonce="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"
|
||||
mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects"
|
||||
echo "$nonce" > "$objects/.restore-claim"
|
||||
work="$(mktemp -d /tmp/evo-restore.XXXXXXXX)"
|
||||
installed=unknown_before published=0
|
||||
cleanup() {
|
||||
local code=$?
|
||||
if [ "$restored" != 1 ]; then
|
||||
if [ "$installed" = 1 ]; then
|
||||
psql -X --no-password -q -v ON_ERROR_STOP=1 -c 'DROP SCHEMA evolution CASCADE' > /dev/null 2>&1 || true
|
||||
fi
|
||||
[ "$created_objects" = 1 ] && rm -rf -- "$objects"
|
||||
if [ "$installed" = 1 ] || [ "$created_objects" = 1 ]; then
|
||||
echo "RESTORE FAILED: removed only what this restore created" \
|
||||
"(schema=$([ "$installed" = 1 ] && echo yes || echo no), objects=$([ "$created_objects" = 1 ] && echo yes || echo no))" >&2
|
||||
fi
|
||||
if [ "$published" != 1 ]; then
|
||||
case "$installed" in
|
||||
no) drop_objects "$nonce" || true ;;
|
||||
yes)
|
||||
if drop_claimed "$nonce"; then
|
||||
drop_objects "$nonce" || true
|
||||
echo "RESTORE FAILED: removed this restore's schema and objects (claim $nonce)" >&2
|
||||
else
|
||||
echo "RESTORE FAILED: claim $nonce could not be removed; the target stays unservable." \
|
||||
"Resolve with: $0 --abandon $nonce --objects $objects" >&2
|
||||
code=5
|
||||
fi
|
||||
;;
|
||||
unknown_before) drop_objects "$nonce" || true ;;
|
||||
*)
|
||||
echo "RESTORE OUTCOME UNKNOWN: claim $nonce. If installed, the target is unservable" \
|
||||
"(every build retired). Resolve with: $0 --abandon $nonce --objects $objects" >&2
|
||||
code=5
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
[ -n "$claim" ] && exec 7>&- # Ends the claim session: its locks are released.
|
||||
[ -n "$lock_pid" ] && { wait "$lock_pid" 2> /dev/null || true; }
|
||||
[ -n "$claim" ] && rm -rf -- "$claim"
|
||||
return "$code"
|
||||
rm -rf -- "$work"
|
||||
exit "$code"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects"
|
||||
created_objects=1
|
||||
claim="$(mktemp -d /tmp/evo-restore-claim.XXXXXXXX)"
|
||||
mkfifo "$claim/in"
|
||||
psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 < "$claim/in" > "$claim/out" 2>&1 &
|
||||
lock_pid=$!
|
||||
exec 7> "$claim/in"
|
||||
cat >&7 <<SQL
|
||||
SELECT pg_catalog.pg_try_advisory_lock(71420004) AS claimed \gset
|
||||
\if :claimed
|
||||
SELECT pg_catalog.pg_advisory_lock(71420002);
|
||||
\! touch '$claim/held'
|
||||
\else
|
||||
\! touch '$claim/busy'
|
||||
\endif
|
||||
SQL
|
||||
for _ in $(seq 1 600); do
|
||||
[ -e "$claim/held" ] || [ -e "$claim/busy" ] && break
|
||||
kill -0 "$lock_pid" 2> /dev/null || break
|
||||
sleep 0.1
|
||||
done
|
||||
[ -e "$claim/busy" ] && refuse "another restore holds this database"
|
||||
[ -e "$claim/held" ] || refuse "could not claim the target database ($(head -c 200 "$claim/out"))"
|
||||
version="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version')")"
|
||||
echo "restore claim $nonce"
|
||||
version="$(q -c "SELECT current_setting('server_version')")"
|
||||
grep -qx "database $version" "$bundle/MANIFEST" || refuse "target server $version differs from the manifest"
|
||||
existing="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")"
|
||||
existing="$(q -c "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")"
|
||||
[ "$existing" = 0 ] || refuse "target database already has an evolution schema"
|
||||
|
||||
# 3. Database in one transaction, then objects. A failed pg_restore installed
|
||||
# nothing (one transaction), so nothing of the database is removed after it.
|
||||
hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; }
|
||||
hook before-restore
|
||||
pg_restore --no-password --exit-on-error --single-transaction --no-owner "${privileges[@]}" \
|
||||
--dbname="$target" "$bundle/database.dump" || refuse "pg_restore failed; the database was not changed"
|
||||
installed=1
|
||||
# 3. INSTALL: dump, claim and the serving gate commit together or not at all. The
|
||||
# COMMIT is only sent after the complete script: a truncated stream rolls back.
|
||||
pg_restore --no-owner "${privileges[@]}" --file="$work/restore.sql" "$bundle/database.dump" ||
|
||||
refuse "the dump could not be read"
|
||||
{
|
||||
cat <<SQL
|
||||
\\set ON_ERROR_STOP on
|
||||
BEGIN;
|
||||
CREATE SCHEMA evolution_restore;
|
||||
REVOKE ALL ON SCHEMA evolution_restore FROM PUBLIC;
|
||||
CREATE TABLE evolution_restore.claims (
|
||||
nonce text PRIMARY KEY CHECK (nonce ~ '^[0-9a-f]{32}\$'),
|
||||
state text NOT NULL CHECK (state IN ('unpublished', 'published')),
|
||||
mode text NOT NULL, manifest_sha256 text NOT NULL, label text,
|
||||
held_builds bytea[] NOT NULL,
|
||||
installed_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at timestamptz);
|
||||
REVOKE ALL ON evolution_restore.claims FROM PUBLIC;
|
||||
SQL
|
||||
cat "$work/restore.sql"
|
||||
cat <<SQL
|
||||
INSERT INTO evolution_restore.claims(nonce, state, mode, manifest_sha256, label, held_builds)
|
||||
SELECT '$nonce', 'unpublished', '$mode', '$manifest_sha', '$label',
|
||||
coalesce(pg_catalog.array_agg(b.build_identity ORDER BY b.build_identity), '{}')
|
||||
FROM evolution.retained_builds b WHERE b.retired_at IS NULL;
|
||||
UPDATE evolution.retained_builds SET retired_at = pg_catalog.clock_timestamp() WHERE retired_at IS NULL;
|
||||
UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL;
|
||||
COMMIT;
|
||||
SQL
|
||||
} > "$work/install.sql"
|
||||
hook before-install
|
||||
status=0
|
||||
psql -X --no-password -q -f "$work/install.sql" > "$work/install.log" 2>&1 || status=$?
|
||||
case "${EVO_RESTORE_TEST_INSTALL_ACK:-}" in
|
||||
lost) status=1 ;;
|
||||
unreachable) status=1; export PGPORT=1 PGHOST=/nonexistent ;;
|
||||
esac
|
||||
if [ "$status" = 0 ]; then
|
||||
installed=yes
|
||||
else
|
||||
state="$(claim_state "$nonce")"
|
||||
case "$state" in
|
||||
unpublished) installed=yes; echo "install outcome resolved: committed (claim $nonce)" ;;
|
||||
none)
|
||||
installed=no
|
||||
sed -n 's/^psql:[^:]*:[0-9]*: //p' "$work/install.log" | head -3 >&2
|
||||
refuse "the restore transaction rolled back; the database was not changed (confirmed)"
|
||||
;;
|
||||
*) installed=unknown; exit 5 ;;
|
||||
esac
|
||||
fi
|
||||
(cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' |
|
||||
xargs -0 -r -I{} mkdir -p -m 700 -- "$objects/{}")
|
||||
(cd "$bundle/objects" && find . -type f -printf '%P\0' | xargs -0 -r -I{} cp -- {} "$objects/{}")
|
||||
hook after-restore
|
||||
hook after-install
|
||||
|
||||
# 4. Heads exactly as recorded.
|
||||
actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
|
||||
actual_heads="$(q -F ' ' -c "
|
||||
SELECT h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick, h.commit_id,
|
||||
encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
|
||||
FROM evolution.region_heads h
|
||||
@@ -173,7 +272,8 @@ actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
|
||||
AND c.commit_sequence = h.commit_sequence" | sort)"
|
||||
[ "$actual_heads" = "$expected_heads" ] || { echo "restored heads differ from the manifest" >&2; exit 3; }
|
||||
|
||||
# 5. Full history audit of every region.
|
||||
# 5. Full history audit of every region (the CLI's retained list, not the database's
|
||||
# temporarily retired state), then the runtime rights audit for --in-place.
|
||||
pg=()
|
||||
if [ -n "${PGSERVICEFILE:-}" ]; then
|
||||
pg=(--pg-service "$PGSERVICEFILE")
|
||||
@@ -195,17 +295,38 @@ while read -r world region _ _ tick _; do
|
||||
echo "region $world/$region $line"
|
||||
done <<< "$expected_heads"
|
||||
if [ "$in_place" = 1 ]; then
|
||||
# No lease from before the loss survives: the next owner acquires a newer epoch.
|
||||
psql -X --no-password -q -v ON_ERROR_STOP=1 -c \
|
||||
'UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL' > /dev/null
|
||||
EVO_PG_RUNTIME_ROLE="$runtime_role" bash "$root/tools/migrate-postgres.sh" > /dev/null ||
|
||||
{ echo "runtime role $runtime_role failed the rights audit" >&2; exit 3; }
|
||||
fi
|
||||
|
||||
# 6. PUBLISH under this NONCE's claim; an ambiguous completion is resolved likewise.
|
||||
hook before-publish
|
||||
status=0
|
||||
q -c "DO \$\$ DECLARE held bytea[]; BEGIN
|
||||
SELECT c.held_builds INTO held FROM evolution_restore.claims c
|
||||
WHERE c.nonce = '$nonce' AND c.state = 'unpublished' FOR UPDATE;
|
||||
IF NOT FOUND THEN
|
||||
RAISE EXCEPTION 'restore claim % lost before publication', '$nonce';
|
||||
END IF;
|
||||
UPDATE evolution.retained_builds SET retired_at = NULL WHERE build_identity = ANY (held);
|
||||
UPDATE evolution_restore.claims SET state = 'published', published_at = clock_timestamp()
|
||||
WHERE nonce = '$nonce';
|
||||
END \$\$" > /dev/null 2>&1 || status=$?
|
||||
if [ "$status" != 0 ]; then
|
||||
case "$(claim_state "$nonce")" in
|
||||
published) ;;
|
||||
unpublished) echo "publication failed" >&2; exit 3 ;;
|
||||
*) installed=unknown; exit 5 ;;
|
||||
esac
|
||||
fi
|
||||
published=1
|
||||
rm -f -- "$objects/.restore-claim"
|
||||
if [ "$in_place" = 1 ]; then
|
||||
created="$(sed -n 's/^created //p' "$bundle/MANIFEST")"
|
||||
while read -r world region _ _ tick _; do
|
||||
echo "REWIND region $world/$region restored at tick $tick (backup $created): every acknowledgement after this tick is undone"
|
||||
done <<< "$expected_heads"
|
||||
fi
|
||||
restored=1
|
||||
what="RESTORE VERIFIED"
|
||||
[ "$in_place" = 1 ] && what="RESTORE IN PLACE VERIFIED"
|
||||
echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
|
||||
echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits claim=$nonce duration_ms=$(( $(date +%s%3N) - begin_ms ))"
|
||||
|
||||
+79
-33
@@ -60,7 +60,7 @@ expect_refusal() { # expect_refusal <exit code> <bundle> <what>
|
||||
--bundle "$2" --objects "$work/restored-$case_n" > "$work/case-$case_n.log" 2>&1 || code=$?
|
||||
[ "$code" = "$1" ] || { cat "$work/case-$case_n.log" >&2; fail "$3: exit $code, expected $1"; }
|
||||
[ ! -e "$work/restored-$case_n" ] || fail "$3: restored objects were left behind"
|
||||
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" = 0 ] ||
|
||||
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")" = 0 ] ||
|
||||
fail "$3: a restored schema was left behind"
|
||||
}
|
||||
damaged() { rm -rf "$work/damaged"; cp -a "$work/bundle" "$work/damaged"; }
|
||||
@@ -90,46 +90,92 @@ PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle
|
||||
[ "$code" = 4 ] && grep -q 'already has an evolution schema' "$work/twice.log" || fail "a second restore into one database"
|
||||
[ "$(PGDATABASE=evo_restore_twice psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
|
||||
fail "the refused second restore changed the first"
|
||||
# Ownership races (review M8 P1). A competing actor creates the schema between this
|
||||
# restore's checks and its pg_restore: the restore fails and leaves that schema alone.
|
||||
# Ownership and failure paths (M8 review P1; follow-up P1/P2). Each case names the
|
||||
# boundary it interrupts; only this invocation's state may ever be removed.
|
||||
restore() { # restore <database> <objects> [env...]: runs restore-world.sh, sets rc.
|
||||
local db="$1" dir="$2"; shift 2
|
||||
rc=0
|
||||
env PGDATABASE="$db" "$@" bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
|
||||
--objects "$dir" > "$work/$(basename "$dir").log" 2>&1 || rc=$?
|
||||
}
|
||||
schemas() { PGDATABASE="$1" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')"; }
|
||||
acquire_refused() { # A lease acquisition in <database> is refused (the world is unservable).
|
||||
! PGDATABASE="$1" psql -XAt -v ON_ERROR_STOP=1 \
|
||||
-c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('b1', 32), 'hex'))" \
|
||||
> "$work/acquire-$1.log" 2>&1 && grep -q 'build not current' "$work/acquire-$1.log"
|
||||
}
|
||||
log_of() { cat "$work/$(basename "$1").log" >&2; }
|
||||
|
||||
# A competing actor creates the schema after this restore's checks: its install
|
||||
# transaction rolls back (confirmed), and the competitor's schema survives.
|
||||
createdb evo_restore_race
|
||||
code=0
|
||||
PGDATABASE=evo_restore_race EVO_RESTORE_TEST_HOOK='[ "$1" = before-restore ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true' \
|
||||
bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/race-a" \
|
||||
> "$work/race-a.log" 2>&1 || code=$?
|
||||
[ "$code" = 4 ] && grep -q 'pg_restore failed; the database was not changed' "$work/race-a.log" ||
|
||||
{ cat "$work/race-a.log" >&2; fail "a pre-empted restore: exit $code"; }
|
||||
restore evo_restore_race "$work/race-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true'
|
||||
[ "$rc" = 4 ] && grep -q 'rolled back; the database was not changed (confirmed)' "$work/race-a.log" ||
|
||||
{ log_of race-a; fail "a pre-empted restore: exit $rc"; }
|
||||
[ "$(PGDATABASE=evo_restore_race psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'winner'")" = 1 ] ||
|
||||
fail "a pre-empted restore deleted the competing schema"
|
||||
[ ! -e "$work/race-a" ] || fail "a pre-empted restore left its own objects directory"
|
||||
# Two restores of one database: the second refuses at once and touches nothing of
|
||||
# the first, which completes. While the first holds the database (after its install),
|
||||
# a lease acquisition there waits instead of running against unverified state.
|
||||
|
||||
# Two restores of one database: whichever installs first wins; the other rolls back
|
||||
# and leaves the winner published and intact.
|
||||
createdb evo_restore_pair
|
||||
PGDATABASE=evo_restore_pair EVO_RESTORE_TEST_HOOK='
|
||||
if [ "$1" = before-restore ]; then touch "'"$work"'/pair-claimed"; sleep 3; fi
|
||||
if [ "$1" = after-restore ]; then
|
||||
psql -XAt -c "SET lock_timeout = '"'"'1s'"'"'" -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/pair-acquire.log" 2>&1 || true
|
||||
fi' bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/pair-a" \
|
||||
> "$work/pair-a.log" 2>&1 &
|
||||
restore evo_restore_pair "$work/pair-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && sleep 4 || true' &
|
||||
pair_pid=$!
|
||||
for _ in $(seq 1 100); do [ -e "$work/pair-claimed" ] && break; sleep 0.1; done
|
||||
code=0
|
||||
PGDATABASE=evo_restore_pair bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
|
||||
--objects "$work/pair-b" > "$work/pair-b.log" 2>&1 || code=$?
|
||||
[ "$code" = 4 ] && grep -q 'another restore holds this database' "$work/pair-b.log" ||
|
||||
{ cat "$work/pair-b.log" >&2; fail "a concurrent restore: exit $code"; }
|
||||
[ ! -e "$work/pair-b" ] || fail "the refused concurrent restore left its objects directory"
|
||||
sleep 1
|
||||
restore evo_restore_pair "$work/pair-b"
|
||||
[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-b.log" || { log_of pair-b; fail "the first installer: exit $rc"; }
|
||||
code=0; wait "$pair_pid" || code=$?
|
||||
[ "$code" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-a.log" ||
|
||||
{ cat "$work/pair-a.log" >&2; fail "the first of two restores: exit $code"; }
|
||||
grep -q 'lock timeout' "$work/pair-acquire.log" ||
|
||||
{ cat "$work/pair-acquire.log" >&2; fail "a lease acquisition ran during the restore"; }
|
||||
grep -q 'rolled back; the database was not changed (confirmed)' "$work/pair-a.log" && [ ! -e "$work/pair-a" ] ||
|
||||
{ log_of pair-a; fail "the second installer: exit $code"; }
|
||||
[ "$(PGDATABASE=evo_restore_pair psql -XAtc "SELECT state FROM evolution_restore.claims")" = published ] &&
|
||||
[ "$(PGDATABASE=evo_restore_pair psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
|
||||
fail "the losing restore disturbed the winner"
|
||||
|
||||
# After install and before publication the world is unservable, durably: across a
|
||||
# database restart (every connection and lock lost) acquisition is still refused, and
|
||||
# the restore then verifies and publishes; afterwards the build is current again.
|
||||
createdb evo_restore_gate
|
||||
restore evo_restore_gate "$work/gate" EVO_RESTORE_TEST_HOOK='
|
||||
if [ "$1" = after-install ]; then
|
||||
psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-before.log" 2>&1 || true
|
||||
pg_ctl -D "$PGHOST/data" -m fast -w restart > /dev/null
|
||||
psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-after.log" 2>&1 || true
|
||||
fi'
|
||||
[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/gate.log" || { log_of gate; fail "a restore across a restart: exit $rc"; }
|
||||
grep -q 'build not current' "$work/gate-before.log" && grep -q 'build not current' "$work/gate-after.log" ||
|
||||
{ cat "$work/gate-before.log" "$work/gate-after.log" >&2; fail "authority was available before publication"; }
|
||||
[ "$(PGDATABASE=evo_restore_gate psql -XAtc 'SELECT count(*) FROM evolution.retained_builds WHERE retired_at IS NULL')" = 1 ] ||
|
||||
fail "publication did not restore the current build"
|
||||
|
||||
# A committed install whose completion is lost is resolved from the claim and
|
||||
# verified, not reported as unchanged.
|
||||
createdb evo_restore_lost
|
||||
restore evo_restore_lost "$work/lost" EVO_RESTORE_TEST_INSTALL_ACK=lost
|
||||
[ "$rc" = 0 ] && grep -q '^install outcome resolved: committed' "$work/lost.log" &&
|
||||
grep -q '^RESTORE VERIFIED regions=1 ' "$work/lost.log" || { log_of lost; fail "a lost install acknowledgement: exit $rc"; }
|
||||
|
||||
# An install whose outcome cannot be resolved keeps everything, stays unservable,
|
||||
# and is removed only by --abandon with its claim.
|
||||
createdb evo_restore_unknown
|
||||
restore evo_restore_unknown "$work/unknown" EVO_RESTORE_TEST_INSTALL_ACK=unreachable
|
||||
nonce="$(sed -n 's/^restore claim \([0-9a-f]\{32\}\)$/\1/p' "$work/unknown.log")"
|
||||
[ "$rc" = 5 ] && grep -q '^RESTORE OUTCOME UNKNOWN' "$work/unknown.log" && [ -d "$work/unknown" ] && [ -n "$nonce" ] ||
|
||||
{ log_of unknown; fail "an unresolvable install: exit $rc"; }
|
||||
[ "$(schemas evo_restore_unknown)" = 2 ] && acquire_refused evo_restore_unknown || fail "an unresolved restore is servable"
|
||||
code=0
|
||||
PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon 00000000000000000000000000000000 \
|
||||
--objects "$work/unknown" > "$work/abandon-wrong.log" 2>&1 || code=$?
|
||||
[ "$(schemas evo_restore_unknown)" = 2 ] && [ -d "$work/unknown" ] || fail "--abandon with another claim removed state"
|
||||
PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon "$nonce" \
|
||||
--objects "$work/unknown" > "$work/abandon.log" 2>&1 || { cat "$work/abandon.log" >&2; fail "--abandon"; }
|
||||
[ "$(schemas evo_restore_unknown)" = 0 ] && [ ! -e "$work/unknown" ] || fail "--abandon left state behind"
|
||||
restore evo_restore_unknown "$work/unknown-retry"
|
||||
[ "$rc" = 0 ] || { log_of unknown-retry; fail "a restore after --abandon: exit $rc"; }
|
||||
|
||||
# An existing objects path is never claimed or removed.
|
||||
mkdir "$work/occupied"; echo keep > "$work/occupied/mine"
|
||||
createdb evo_restore_occupied
|
||||
code=0
|
||||
PGDATABASE=evo_restore_occupied bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
|
||||
--objects "$work/occupied" > "$work/occupied.log" 2>&1 || code=$?
|
||||
[ "$code" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] || fail "an occupied objects path: exit $code"
|
||||
restore evo_restore_occupied "$work/occupied"
|
||||
[ "$rc" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] && [ "$(schemas evo_restore_occupied)" = 0 ] ||
|
||||
fail "an occupied objects path: exit $rc"
|
||||
echo "backup and restore qualification passed (backup tick $backup_tick, acknowledged $acked_before, source $source_tick)"
|
||||
|
||||
Reference in New Issue
Block a user