fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3)

The restore no longer relies on an auxiliary lock session. One install
transaction holds the dump, a nonce claim row, fenced leases and retired builds,
so the world is unservable until a nonce-checked publish after verification,
regardless of connection loss or restarts. A failed install is resolved from the
claim (committed, rolled back, or unknown with the target kept unservable and
--abandon NONCE to resolve); cleanup only removes what the nonce owns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-10-01 06:14:34 -04:00
co-authored by Claude Opus 5.5
parent 3a34ae39bc
commit 68da5654ff
5 changed files with 386 additions and 145 deletions
+42 -16
View File
@@ -173,21 +173,47 @@ every PostgreSQL build:
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
chosen; a directory and `tar` need no new dependency.
## Amendment: restore ownership (M8 review, 2026-09-30)
The review reproduced a race. A restore checked that the target was empty, a
## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01)
The first review reproduced a race. A restore checked that the target was empty, a
competing actor then created the schema, the restore's `pg_restore` failed, and
its cleanup dropped the other actor's schema. Restores now own what they touch:
- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is
refused and never touched.
- One psql session, held open for the whole restore, takes the restore claim
(advisory 71420004, try-lock, so a concurrent restore refuses at once) and the
upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in
that database wait until the restore is verified or removed.
- Cleanup removes only what the invocation created: the schema only if its own
one-transaction `pg_restore` committed, and the directory only if its own `mkdir`
created it.
its cleanup dropped the other actor's schema. The first fix held advisory locks in
an auxiliary session. The follow-up review showed two remaining problems:
- losing that session released authority before verification;
- a nonzero exit was not proof of rollback.
`backup-check.sh` covers three cases:
- a competing schema created just before `pg_restore` survives;
- a concurrent restore refuses, and the first completes;
- a lease acquisition times out while the restore holds the database.
Ownership and the serving gate are now durable state in the target itself.
- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an
explicit COMMIT only after the whole script). It contains:
- the dump;
- a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest,
builds it held current)`;
- every lease fenced, and every current build retired.
From that commit on, acquisition (both forms), renewal and commits refuse,
whatever happens to any connection or to the server.
- **Publish** runs after the head comparison, the full audit and (in place) the
rights audit. It is one transaction that first locks the invocation's unpublished
claim, then makes exactly the held builds current again. A restore that lost its
claim cannot publish. The claim row stays as a record of the restore.
- **Resolution.** When an install exits nonzero, the claim row decides:
- committed: verification continues;
- absent: rolled back, and the database is reported unchanged (confirmed);
- unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing
is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it
later.
- **Cleanup** removes schemas only through a transaction that locks this NONCE's
unpublished claim. It removes the objects directory only while that directory
carries this NONCE's marker. A competing restore's or provisioner's state is
never touched.
`backup-check.sh` covers:
- a competing schema created before install (rollback confirmed; the competitor
survives);
- two concurrent restores (the first installer wins; the loser rolls back and
leaves the winner published);
- acquisition refused after install, both before and after a database restart,
with the build current again after publication;
- a lost install acknowledgement (resolved as committed, then verified);
- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the
right one removes everything, and a retry succeeds);
- an occupied objects path.
+3 -1
View File
@@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows.
`restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a
bundle under the world's own identity into a replacement database:
- It restores the dump's grants (bundles now keep them; ADR-0074 is amended).
- It fences every restored lease, so the next owner acquires a newer epoch.
- It fences every restored lease, so the next owner acquires a newer epoch. The
world stays unservable (every build retired) until verification publishes it
(ADR-0074 amendment).
- It re-runs the migration rights audit for the environment's runtime role.
- It prints a `REWIND` line per region naming the tick it was restored to.
+46
View File
@@ -3087,3 +3087,49 @@ Review: `.cache/reviews/M8_REVIEW_680e7e5-5b482ad.md` (one P1, five P2).
objects, 0 failures.
- ROADMAP corrected: M8-dev/stage is implemented and locally qualified, but **not
passed** until the live gates listed there are run.
## Review 3a34ae3 follow-up: durable restore gate - 2026-10-01
Review: `.cache/reviews/REVIEW_3a34ae3.md`. P1: losing the auxiliary claim session
released authority before verification. P2: a nonzero `pg_restore` exit was taken
as proof of rollback.
- `restore-world.sh` no longer relies on an auxiliary lock session. It works in
three steps, as the ADR-0074 amendment describes:
- an install transaction holds the dump, a claim row with the invocation's
nonce, fenced leases and retired builds;
- verification follows;
- a nonce-checked publish transaction re-enables exactly the held builds.
Ambiguous outcomes are resolved from the claim row. An unresolvable one is
OUTCOME UNKNOWN (exit 5): the target stays unservable, and
`--abandon NONCE --objects DIR` resolves it later.
- `backup-check.sh` passed in the Release, analysis and ASan/UBSan PostgreSQL
builds, with these cases:
- a competing schema before install: the rollback was confirmed and the
competitor's table survived;
- two concurrent restores: the first installer was verified and published; the
second rolled back, confirmed, and left the winner intact;
- after install, `acquire_region_lease` was refused ("build not current") both
before and after `pg_ctl restart` mid-restore; the restore then verified, and
one current build was restored;
- a lost install acknowledgement: "install outcome resolved: committed", then
verified;
- an unreachable resolution: exit 5, both schemas and the objects kept,
acquisition refused, `--abandon` with a wrong nonce removed nothing, the right
nonce removed everything, and a retry restore succeeded;
- an occupied objects path: refused and untouched;
- damaged bundles: refused, with no `evolution` or `evolution_restore` left.
- The reviewer's claim-loss reproduction, adapted to the renamed hook
(`after-install`), finds no lock to terminate. The runtime acquisition inside it
failed with `build not current` (exit 1), and the restore then published.
- `dr-check.sh` (in-place restore) and `upgrade-check.sh` passed in all three
builds.
- Mutations:
- install without retiring builds: caught ("authority was available before
publication");
- resolution ignoring a committed claim: caught (exit 5 instead of verified);
- cleanup ignoring the claim's nonce: NOT exercised (the edit pattern failed to
apply). The abandon path also checks the nonce first, so that defense is two
layers deep and only the outer layer is tested.
- Not run: a real network partition around COMMIT. The lost acknowledgement is an
injected test mode (`EVO_RESTORE_TEST_INSTALL_ACK`).
+216 -95
View File
@@ -3,53 +3,67 @@
#
# tools/backup/restore-world.sh --bundle DIR --objects DIR
# [--in-place --runtime-role ROLE --accept-rewind]
# tools/backup/restore-world.sh --abandon NONCE --objects DIR
#
# Target connection: PG* environment naming an EMPTY database that the login owns
# (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE).
# The restore needs no source role, credential or key: owners and grants are not
# restored, so the target login owns everything restored. --objects must not
# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or
# PATH.
# The restore needs no source role, credential or key: owners are not restored, and
# grants only with --in-place, so the target login owns everything restored.
# evo-world-clone comes from EVO_WORLD_CLONE_BIN or PATH.
#
# Order, failing closed at each step:
# 1. every bundle file matches SHA256SUMS, no file is unlisted, and every object's
# name is its digest; nothing is touched before this passes;
# 2. the targets are claimed for this invocation alone (below), then the target
# has no `evolution` schema and runs the manifest's server version;
# 3. pg_restore in one transaction, then objects are copied;
# 2. --objects is claimed by creating it (mkdir, never -p: an existing path is
# refused and never touched) and marked with this invocation's random NONCE; the
# target must have neither `evolution` nor `evolution_restore` and must run the
# manifest's server version;
# 3. INSTALL, one transaction: the dump, a claim row evolution_restore.claims(NONCE,
# unpublished), every lease fenced and every current build retired. From that
# commit on the world is unservable: acquisition, renewal and commits refuse
# until publication, whatever happens to any connection or the server (review
# M8 follow-up P1). Then the objects are copied;
# 4. the restored heads equal the MANIFEST heads exactly;
# 5. every region's full history is audited (evo-world-clone audit: each link's
# manifest, payload, trace replay and outcomes, from genesis to the head).
# Ownership (review M8 P1): --objects is claimed by creating it (mkdir, never -p: an
# existing path is refused and never touched), and the database by one psql session
# held open for the whole restore. That session takes the restore claim (advisory
# 71420004, try-lock: a concurrent restore of the same database refuses at once) and
# the upgrade lock exclusively (71420002: provisioning, acquisition and lease checks
# in that database wait until this restore is verified or removed). On failure only
# what this invocation created is removed: the schema only if its own pg_restore
# committed, the objects directory only if its own mkdir created it. So an
# unverified restore can never be mistaken for a world, and a losing restore never
# deletes a winner's state. EVO_RESTORE_TEST_HOOK (tests only) is run by bash with
# $1 = before-restore (claimed, checked) and after-restore (installed, objects copied). The result is a
# verbatim, owner-only copy with the source's world identities: it has no runtime
# grants and is never served. To run it, clone it into an isolated world
# (tools/backup/verify-backup.sh --isolate).
# 5. every region's full history is audited (evo-world-clone audit), and with
# --in-place migrate-postgres.sh audits the runtime role's effective rights;
# 6. PUBLISH, one transaction that first locks this NONCE's unpublished claim:
# exactly the builds the install retired become current again and the claim
# becomes published. A restore that lost its claim cannot publish.
# A failed or interrupted install is resolved, not assumed (review M8 follow-up P2):
# the claim row says whether this invocation's transaction committed. Committed ->
# verification continues; absent -> rolled back (the database was not changed);
# unreachable -> OUTCOME UNKNOWN (exit 5): the target stays unservable and both the
# schema and the objects are kept; resolve it later with --abandon NONCE.
# Cleanup only ever removes what this NONCE owns: the schemas only through a
# transaction that locks this NONCE's unpublished claim, the objects directory only
# while it carries this NONCE's marker. A competing restore's or provisioner's state
# is never touched. The default result is a verbatim, owner-only copy with the
# source's world identities, no runtime grants and never served; to run it, clone it
# into an isolated world (tools/backup/verify-backup.sh --isolate).
#
# --in-place (M8.8 host-loss recovery for dev/stage, ADR-0075) restores the world
# under its own identity so its hosts can serve it again: the dump's grants are
# restored (the environment's roles must already exist in the target cluster), every
# restored lease is fenced so a new owner acquires under a newer epoch, and
# migrate-postgres.sh audits the runtime role's effective rights. Every acknowledgement
# after the bundle's cut is undone: deaths may reverse and resources may reappear.
# --accept-rewind states that this is acceptable, which it is for dev and stage only;
# World0 launch stays blocked until acknowledgements survive host loss (ADR-0075).
# Fence the old database with fence-database.sh before any of its hosts can return.
# under its own identity so its hosts can serve it again after publication: the
# dump's grants are restored (the environment's roles must already exist in the
# target cluster). Every acknowledgement after the bundle's cut is undone: deaths may
# reverse and resources may reappear. --accept-rewind states that this is
# acceptable, which it is for dev and stage only; World0 launch stays blocked until
# acknowledgements survive host loss (ADR-0075). Fence the old database with
# fence-database.sh before any of its hosts can return.
#
# Exit: 0 verified and published, 2 usage, 3 verification failed (this restore's
# state removed), 4 refused or rolled back (nothing of this restore left),
# 5 outcome unknown (target unservable; evidence kept).
# Tests only: EVO_RESTORE_TEST_HOOK is run by bash with $1 = before-install,
# after-install or before-publish; EVO_RESTORE_TEST_INSTALL_ACK=lost treats a committed
# install as failed (an ambiguous completion), =unreachable also makes its resolution
# unreachable.
set -euo pipefail
usage() {
echo "usage: $0 --bundle DIR --objects DIR [--in-place --runtime-role ROLE --accept-rewind]" >&2
echo " $0 --abandon NONCE --objects DIR" >&2
exit 2
}
bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0
bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0 abandon=""
while [ $# -gt 0 ]; do
case "$1" in
--bundle) bundle="${2:-}"; shift 2 ;;
@@ -57,10 +71,16 @@ while [ $# -gt 0 ]; do
--in-place) in_place=1; shift ;;
--runtime-role) runtime_role="${2:-}"; shift 2 ;;
--accept-rewind) accept_rewind=1; shift ;;
--abandon) abandon="${2:-}"; shift 2 ;;
*) usage ;;
esac
done
[ -n "$bundle" ] && [ -n "$objects" ] || usage
[ -n "$objects" ] || usage
if [ -n "$abandon" ]; then
[[ "$abandon" =~ ^[0-9a-f]{32}$ ]] && [ -z "$bundle" ] && [ "$in_place" = 0 ] || usage
else
[ -n "$bundle" ] || usage
fi
if [ "$in_place" = 1 ]; then
[ -n "$runtime_role" ] || { echo "--in-place needs --runtime-role" >&2; exit 2; }
[ "$accept_rewind" = 1 ] ||
@@ -69,10 +89,55 @@ elif [ -n "$runtime_role" ] || [ "$accept_rewind" = 1 ]; then
usage
fi
root="$(cd "$(dirname "$0")/../.." && pwd)"
bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")"
objects="$(realpath -m -- "$objects")"
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
umask 077
begin_ms="$(date +%s%3N)"
q() { psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 "$@"; }
# The claim row of NONCE: none, unpublished, published, or unknown (unreachable).
claim_state() {
local nonce="$1" exists state
exists="$(q -c "SELECT to_regclass('evolution_restore.claims') IS NOT NULL" 2> /dev/null)" || { echo unknown; return; }
[ "$exists" = t ] || { echo none; return; }
state="$(q -c "SELECT state FROM evolution_restore.claims WHERE nonce = '$nonce'" 2> /dev/null)" ||
{ echo unknown; return; }
echo "${state:-none}"
}
# Removes NONCE's unpublished restore: both schemas, in one transaction that first
# locks that claim. Anything else (another claim, a published one) is refused.
drop_claimed() {
q -c "DO \$\$ BEGIN
IF to_regclass('evolution_restore.claims') IS NULL THEN
RAISE EXCEPTION 'no restore claim';
END IF;
PERFORM 1 FROM evolution_restore.claims WHERE nonce = '$1' AND state = 'unpublished' FOR UPDATE;
IF NOT FOUND THEN
RAISE EXCEPTION 'restore claim % is not unpublished here', '$1';
END IF;
DROP SCHEMA evolution CASCADE;
DROP SCHEMA evolution_restore CASCADE;
END \$\$" > /dev/null 2>&1
}
# Removes the objects directory only while it carries NONCE's marker.
drop_objects() {
[ -f "$objects/.restore-claim" ] && [ "$(cat "$objects/.restore-claim")" = "$1" ] && rm -rf -- "$objects"
}
if [ -n "$abandon" ]; then
state="$(claim_state "$abandon")"
case "$state" in
unpublished) drop_claimed "$abandon" || { echo "ABANDON FAILED: could not remove claim $abandon" >&2; exit 3; } ;;
none) ;;
published) echo "ABANDON REFUSED: claim $abandon is published (a verified world)" >&2; exit 4 ;;
*) echo "ABANDON FAILED: the target database is unreachable" >&2; exit 5 ;;
esac
drop_objects "$abandon" || true
echo "ABANDONED claim $abandon (database state was $state; objects $([ -e "$objects" ] && echo kept || echo removed))"
exit 0
fi
bundle="$(realpath -- "$bundle")"
refuse() { echo "RESTORE REFUSED: $*" >&2; exit 4; }
# 1. Bundle integrity, before anything else is touched.
@@ -93,78 +158,112 @@ done <<< "$present"
grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format"
expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)"
[ -n "$expected_heads" ] || refuse "manifest names no region"
privileges=(--no-privileges)
privileges=(--no-privileges) mode=isolated
if [ "$in_place" = 1 ]; then
grep -qx 'dump privileges' "$bundle/MANIFEST" || refuse "bundle predates in-place restore (no grants in its dump)"
privileges=()
privileges=() mode=in-place
fi
manifest_sha="$(sha256sum "$bundle/MANIFEST")"; manifest_sha="${manifest_sha%% *}"
label="$(sed -n 's/^label \([A-Za-z0-9._:-]*\)$/\1/p' "$bundle/MANIFEST" | head -1)"
hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; }
target="${PGDATABASE:-}"
[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE"
[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; }
# 2. Claim the targets, then check them under the claim.
created_objects=0 installed=0 restored=0 lock_pid="" claim=""
# 2. Claim the objects path, then check the target.
nonce="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')"
mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects"
echo "$nonce" > "$objects/.restore-claim"
work="$(mktemp -d /tmp/evo-restore.XXXXXXXX)"
installed=unknown_before published=0
cleanup() {
local code=$?
if [ "$restored" != 1 ]; then
if [ "$installed" = 1 ]; then
psql -X --no-password -q -v ON_ERROR_STOP=1 -c 'DROP SCHEMA evolution CASCADE' > /dev/null 2>&1 || true
fi
[ "$created_objects" = 1 ] && rm -rf -- "$objects"
if [ "$installed" = 1 ] || [ "$created_objects" = 1 ]; then
echo "RESTORE FAILED: removed only what this restore created" \
"(schema=$([ "$installed" = 1 ] && echo yes || echo no), objects=$([ "$created_objects" = 1 ] && echo yes || echo no))" >&2
fi
if [ "$published" != 1 ]; then
case "$installed" in
no) drop_objects "$nonce" || true ;;
yes)
if drop_claimed "$nonce"; then
drop_objects "$nonce" || true
echo "RESTORE FAILED: removed this restore's schema and objects (claim $nonce)" >&2
else
echo "RESTORE FAILED: claim $nonce could not be removed; the target stays unservable." \
"Resolve with: $0 --abandon $nonce --objects $objects" >&2
code=5
fi
;;
unknown_before) drop_objects "$nonce" || true ;;
*)
echo "RESTORE OUTCOME UNKNOWN: claim $nonce. If installed, the target is unservable" \
"(every build retired). Resolve with: $0 --abandon $nonce --objects $objects" >&2
code=5
;;
esac
fi
[ -n "$claim" ] && exec 7>&- # Ends the claim session: its locks are released.
[ -n "$lock_pid" ] && { wait "$lock_pid" 2> /dev/null || true; }
[ -n "$claim" ] && rm -rf -- "$claim"
return "$code"
rm -rf -- "$work"
exit "$code"
}
trap cleanup EXIT
mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects"
created_objects=1
claim="$(mktemp -d /tmp/evo-restore-claim.XXXXXXXX)"
mkfifo "$claim/in"
psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 < "$claim/in" > "$claim/out" 2>&1 &
lock_pid=$!
exec 7> "$claim/in"
cat >&7 <<SQL
SELECT pg_catalog.pg_try_advisory_lock(71420004) AS claimed \gset
\if :claimed
SELECT pg_catalog.pg_advisory_lock(71420002);
\! touch '$claim/held'
\else
\! touch '$claim/busy'
\endif
SQL
for _ in $(seq 1 600); do
[ -e "$claim/held" ] || [ -e "$claim/busy" ] && break
kill -0 "$lock_pid" 2> /dev/null || break
sleep 0.1
done
[ -e "$claim/busy" ] && refuse "another restore holds this database"
[ -e "$claim/held" ] || refuse "could not claim the target database ($(head -c 200 "$claim/out"))"
version="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version')")"
echo "restore claim $nonce"
version="$(q -c "SELECT current_setting('server_version')")"
grep -qx "database $version" "$bundle/MANIFEST" || refuse "target server $version differs from the manifest"
existing="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")"
existing="$(q -c "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")"
[ "$existing" = 0 ] || refuse "target database already has an evolution schema"
# 3. Database in one transaction, then objects. A failed pg_restore installed
# nothing (one transaction), so nothing of the database is removed after it.
hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; }
hook before-restore
pg_restore --no-password --exit-on-error --single-transaction --no-owner "${privileges[@]}" \
--dbname="$target" "$bundle/database.dump" || refuse "pg_restore failed; the database was not changed"
installed=1
# 3. INSTALL: dump, claim and the serving gate commit together or not at all. The
# COMMIT is only sent after the complete script: a truncated stream rolls back.
pg_restore --no-owner "${privileges[@]}" --file="$work/restore.sql" "$bundle/database.dump" ||
refuse "the dump could not be read"
{
cat <<SQL
\\set ON_ERROR_STOP on
BEGIN;
CREATE SCHEMA evolution_restore;
REVOKE ALL ON SCHEMA evolution_restore FROM PUBLIC;
CREATE TABLE evolution_restore.claims (
nonce text PRIMARY KEY CHECK (nonce ~ '^[0-9a-f]{32}\$'),
state text NOT NULL CHECK (state IN ('unpublished', 'published')),
mode text NOT NULL, manifest_sha256 text NOT NULL, label text,
held_builds bytea[] NOT NULL,
installed_at timestamptz NOT NULL DEFAULT clock_timestamp(),
published_at timestamptz);
REVOKE ALL ON evolution_restore.claims FROM PUBLIC;
SQL
cat "$work/restore.sql"
cat <<SQL
INSERT INTO evolution_restore.claims(nonce, state, mode, manifest_sha256, label, held_builds)
SELECT '$nonce', 'unpublished', '$mode', '$manifest_sha', '$label',
coalesce(pg_catalog.array_agg(b.build_identity ORDER BY b.build_identity), '{}')
FROM evolution.retained_builds b WHERE b.retired_at IS NULL;
UPDATE evolution.retained_builds SET retired_at = pg_catalog.clock_timestamp() WHERE retired_at IS NULL;
UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL;
COMMIT;
SQL
} > "$work/install.sql"
hook before-install
status=0
psql -X --no-password -q -f "$work/install.sql" > "$work/install.log" 2>&1 || status=$?
case "${EVO_RESTORE_TEST_INSTALL_ACK:-}" in
lost) status=1 ;;
unreachable) status=1; export PGPORT=1 PGHOST=/nonexistent ;;
esac
if [ "$status" = 0 ]; then
installed=yes
else
state="$(claim_state "$nonce")"
case "$state" in
unpublished) installed=yes; echo "install outcome resolved: committed (claim $nonce)" ;;
none)
installed=no
sed -n 's/^psql:[^:]*:[0-9]*: //p' "$work/install.log" | head -3 >&2
refuse "the restore transaction rolled back; the database was not changed (confirmed)"
;;
*) installed=unknown; exit 5 ;;
esac
fi
(cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' |
xargs -0 -r -I{} mkdir -p -m 700 -- "$objects/{}")
(cd "$bundle/objects" && find . -type f -printf '%P\0' | xargs -0 -r -I{} cp -- {} "$objects/{}")
hook after-restore
hook after-install
# 4. Heads exactly as recorded.
actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
actual_heads="$(q -F ' ' -c "
SELECT h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick, h.commit_id,
encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
FROM evolution.region_heads h
@@ -173,7 +272,8 @@ actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
AND c.commit_sequence = h.commit_sequence" | sort)"
[ "$actual_heads" = "$expected_heads" ] || { echo "restored heads differ from the manifest" >&2; exit 3; }
# 5. Full history audit of every region.
# 5. Full history audit of every region (the CLI's retained list, not the database's
# temporarily retired state), then the runtime rights audit for --in-place.
pg=()
if [ -n "${PGSERVICEFILE:-}" ]; then
pg=(--pg-service "$PGSERVICEFILE")
@@ -195,17 +295,38 @@ while read -r world region _ _ tick _; do
echo "region $world/$region $line"
done <<< "$expected_heads"
if [ "$in_place" = 1 ]; then
# No lease from before the loss survives: the next owner acquires a newer epoch.
psql -X --no-password -q -v ON_ERROR_STOP=1 -c \
'UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL' > /dev/null
EVO_PG_RUNTIME_ROLE="$runtime_role" bash "$root/tools/migrate-postgres.sh" > /dev/null ||
{ echo "runtime role $runtime_role failed the rights audit" >&2; exit 3; }
fi
# 6. PUBLISH under this NONCE's claim; an ambiguous completion is resolved likewise.
hook before-publish
status=0
q -c "DO \$\$ DECLARE held bytea[]; BEGIN
SELECT c.held_builds INTO held FROM evolution_restore.claims c
WHERE c.nonce = '$nonce' AND c.state = 'unpublished' FOR UPDATE;
IF NOT FOUND THEN
RAISE EXCEPTION 'restore claim % lost before publication', '$nonce';
END IF;
UPDATE evolution.retained_builds SET retired_at = NULL WHERE build_identity = ANY (held);
UPDATE evolution_restore.claims SET state = 'published', published_at = clock_timestamp()
WHERE nonce = '$nonce';
END \$\$" > /dev/null 2>&1 || status=$?
if [ "$status" != 0 ]; then
case "$(claim_state "$nonce")" in
published) ;;
unpublished) echo "publication failed" >&2; exit 3 ;;
*) installed=unknown; exit 5 ;;
esac
fi
published=1
rm -f -- "$objects/.restore-claim"
if [ "$in_place" = 1 ]; then
created="$(sed -n 's/^created //p' "$bundle/MANIFEST")"
while read -r world region _ _ tick _; do
echo "REWIND region $world/$region restored at tick $tick (backup $created): every acknowledgement after this tick is undone"
done <<< "$expected_heads"
fi
restored=1
what="RESTORE VERIFIED"
[ "$in_place" = 1 ] && what="RESTORE IN PLACE VERIFIED"
echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits claim=$nonce duration_ms=$(( $(date +%s%3N) - begin_ms ))"
+79 -33
View File
@@ -60,7 +60,7 @@ expect_refusal() { # expect_refusal <exit code> <bundle> <what>
--bundle "$2" --objects "$work/restored-$case_n" > "$work/case-$case_n.log" 2>&1 || code=$?
[ "$code" = "$1" ] || { cat "$work/case-$case_n.log" >&2; fail "$3: exit $code, expected $1"; }
[ ! -e "$work/restored-$case_n" ] || fail "$3: restored objects were left behind"
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" = 0 ] ||
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")" = 0 ] ||
fail "$3: a restored schema was left behind"
}
damaged() { rm -rf "$work/damaged"; cp -a "$work/bundle" "$work/damaged"; }
@@ -90,46 +90,92 @@ PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle
[ "$code" = 4 ] && grep -q 'already has an evolution schema' "$work/twice.log" || fail "a second restore into one database"
[ "$(PGDATABASE=evo_restore_twice psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
fail "the refused second restore changed the first"
# Ownership races (review M8 P1). A competing actor creates the schema between this
# restore's checks and its pg_restore: the restore fails and leaves that schema alone.
# Ownership and failure paths (M8 review P1; follow-up P1/P2). Each case names the
# boundary it interrupts; only this invocation's state may ever be removed.
restore() { # restore <database> <objects> [env...]: runs restore-world.sh, sets rc.
local db="$1" dir="$2"; shift 2
rc=0
env PGDATABASE="$db" "$@" bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$dir" > "$work/$(basename "$dir").log" 2>&1 || rc=$?
}
schemas() { PGDATABASE="$1" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')"; }
acquire_refused() { # A lease acquisition in <database> is refused (the world is unservable).
! PGDATABASE="$1" psql -XAt -v ON_ERROR_STOP=1 \
-c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('b1', 32), 'hex'))" \
> "$work/acquire-$1.log" 2>&1 && grep -q 'build not current' "$work/acquire-$1.log"
}
log_of() { cat "$work/$(basename "$1").log" >&2; }
# A competing actor creates the schema after this restore's checks: its install
# transaction rolls back (confirmed), and the competitor's schema survives.
createdb evo_restore_race
code=0
PGDATABASE=evo_restore_race EVO_RESTORE_TEST_HOOK='[ "$1" = before-restore ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true' \
bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/race-a" \
> "$work/race-a.log" 2>&1 || code=$?
[ "$code" = 4 ] && grep -q 'pg_restore failed; the database was not changed' "$work/race-a.log" ||
{ cat "$work/race-a.log" >&2; fail "a pre-empted restore: exit $code"; }
restore evo_restore_race "$work/race-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true'
[ "$rc" = 4 ] && grep -q 'rolled back; the database was not changed (confirmed)' "$work/race-a.log" ||
{ log_of race-a; fail "a pre-empted restore: exit $rc"; }
[ "$(PGDATABASE=evo_restore_race psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'winner'")" = 1 ] ||
fail "a pre-empted restore deleted the competing schema"
[ ! -e "$work/race-a" ] || fail "a pre-empted restore left its own objects directory"
# Two restores of one database: the second refuses at once and touches nothing of
# the first, which completes. While the first holds the database (after its install),
# a lease acquisition there waits instead of running against unverified state.
# Two restores of one database: whichever installs first wins; the other rolls back
# and leaves the winner published and intact.
createdb evo_restore_pair
PGDATABASE=evo_restore_pair EVO_RESTORE_TEST_HOOK='
if [ "$1" = before-restore ]; then touch "'"$work"'/pair-claimed"; sleep 3; fi
if [ "$1" = after-restore ]; then
psql -XAt -c "SET lock_timeout = '"'"'1s'"'"'" -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/pair-acquire.log" 2>&1 || true
fi' bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/pair-a" \
> "$work/pair-a.log" 2>&1 &
restore evo_restore_pair "$work/pair-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && sleep 4 || true' &
pair_pid=$!
for _ in $(seq 1 100); do [ -e "$work/pair-claimed" ] && break; sleep 0.1; done
code=0
PGDATABASE=evo_restore_pair bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$work/pair-b" > "$work/pair-b.log" 2>&1 || code=$?
[ "$code" = 4 ] && grep -q 'another restore holds this database' "$work/pair-b.log" ||
{ cat "$work/pair-b.log" >&2; fail "a concurrent restore: exit $code"; }
[ ! -e "$work/pair-b" ] || fail "the refused concurrent restore left its objects directory"
sleep 1
restore evo_restore_pair "$work/pair-b"
[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-b.log" || { log_of pair-b; fail "the first installer: exit $rc"; }
code=0; wait "$pair_pid" || code=$?
[ "$code" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-a.log" ||
{ cat "$work/pair-a.log" >&2; fail "the first of two restores: exit $code"; }
grep -q 'lock timeout' "$work/pair-acquire.log" ||
{ cat "$work/pair-acquire.log" >&2; fail "a lease acquisition ran during the restore"; }
grep -q 'rolled back; the database was not changed (confirmed)' "$work/pair-a.log" && [ ! -e "$work/pair-a" ] ||
{ log_of pair-a; fail "the second installer: exit $code"; }
[ "$(PGDATABASE=evo_restore_pair psql -XAtc "SELECT state FROM evolution_restore.claims")" = published ] &&
[ "$(PGDATABASE=evo_restore_pair psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
fail "the losing restore disturbed the winner"
# After install and before publication the world is unservable, durably: across a
# database restart (every connection and lock lost) acquisition is still refused, and
# the restore then verifies and publishes; afterwards the build is current again.
createdb evo_restore_gate
restore evo_restore_gate "$work/gate" EVO_RESTORE_TEST_HOOK='
if [ "$1" = after-install ]; then
psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-before.log" 2>&1 || true
pg_ctl -D "$PGHOST/data" -m fast -w restart > /dev/null
psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-after.log" 2>&1 || true
fi'
[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/gate.log" || { log_of gate; fail "a restore across a restart: exit $rc"; }
grep -q 'build not current' "$work/gate-before.log" && grep -q 'build not current' "$work/gate-after.log" ||
{ cat "$work/gate-before.log" "$work/gate-after.log" >&2; fail "authority was available before publication"; }
[ "$(PGDATABASE=evo_restore_gate psql -XAtc 'SELECT count(*) FROM evolution.retained_builds WHERE retired_at IS NULL')" = 1 ] ||
fail "publication did not restore the current build"
# A committed install whose completion is lost is resolved from the claim and
# verified, not reported as unchanged.
createdb evo_restore_lost
restore evo_restore_lost "$work/lost" EVO_RESTORE_TEST_INSTALL_ACK=lost
[ "$rc" = 0 ] && grep -q '^install outcome resolved: committed' "$work/lost.log" &&
grep -q '^RESTORE VERIFIED regions=1 ' "$work/lost.log" || { log_of lost; fail "a lost install acknowledgement: exit $rc"; }
# An install whose outcome cannot be resolved keeps everything, stays unservable,
# and is removed only by --abandon with its claim.
createdb evo_restore_unknown
restore evo_restore_unknown "$work/unknown" EVO_RESTORE_TEST_INSTALL_ACK=unreachable
nonce="$(sed -n 's/^restore claim \([0-9a-f]\{32\}\)$/\1/p' "$work/unknown.log")"
[ "$rc" = 5 ] && grep -q '^RESTORE OUTCOME UNKNOWN' "$work/unknown.log" && [ -d "$work/unknown" ] && [ -n "$nonce" ] ||
{ log_of unknown; fail "an unresolvable install: exit $rc"; }
[ "$(schemas evo_restore_unknown)" = 2 ] && acquire_refused evo_restore_unknown || fail "an unresolved restore is servable"
code=0
PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon 00000000000000000000000000000000 \
--objects "$work/unknown" > "$work/abandon-wrong.log" 2>&1 || code=$?
[ "$(schemas evo_restore_unknown)" = 2 ] && [ -d "$work/unknown" ] || fail "--abandon with another claim removed state"
PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon "$nonce" \
--objects "$work/unknown" > "$work/abandon.log" 2>&1 || { cat "$work/abandon.log" >&2; fail "--abandon"; }
[ "$(schemas evo_restore_unknown)" = 0 ] && [ ! -e "$work/unknown" ] || fail "--abandon left state behind"
restore evo_restore_unknown "$work/unknown-retry"
[ "$rc" = 0 ] || { log_of unknown-retry; fail "a restore after --abandon: exit $rc"; }
# An existing objects path is never claimed or removed.
mkdir "$work/occupied"; echo keep > "$work/occupied/mine"
createdb evo_restore_occupied
code=0
PGDATABASE=evo_restore_occupied bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$work/occupied" > "$work/occupied.log" 2>&1 || code=$?
[ "$code" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] || fail "an occupied objects path: exit $code"
restore evo_restore_occupied "$work/occupied"
[ "$rc" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] && [ "$(schemas evo_restore_occupied)" = 0 ] ||
fail "an occupied objects path: exit $rc"
echo "backup and restore qualification passed (backup tick $backup_tick, acknowledged $acked_before, source $source_tick)"