From 68da5654ff339d670b40e6cde720535a65cbf5fd Mon Sep 17 00:00:00 2001 From: dingo Date: Thu, 1 Oct 2026 06:14:34 -0400 Subject: [PATCH] fix(backup): durable restore gate and resolved install outcomes (review 3a34ae3) The restore no longer relies on an auxiliary lock session. One install transaction holds the dump, a nonce claim row, fenced leases and retired builds, so the world is unservable until a nonce-checked publish after verification, regardless of connection loss or restarts. A failed install is resolved from the claim (committed, rolled back, or unknown with the target kept unservable and --abandon NONCE to resolve); cleanup only removes what the nonce owns. Co-Authored-By: Claude Opus 5.5 --- docs/adr/0074-world-backup-and-restore.md | 58 +++- docs/adr/0075-world0-acknowledgment-gate.md | 4 +- docs/engineering/VALIDATION.md | 46 +++ tools/backup/restore-world.sh | 311 ++++++++++++++------ tools/ci/backup-check.sh | 112 ++++--- 5 files changed, 386 insertions(+), 145 deletions(-) diff --git a/docs/adr/0074-world-backup-and-restore.md b/docs/adr/0074-world-backup-and-restore.md index 2e7b926..b601624 100644 --- a/docs/adr/0074-world-backup-and-restore.md +++ b/docs/adr/0074-world-backup-and-restore.md @@ -173,21 +173,47 @@ every PostgreSQL build: - **A new object-storage dependency (MinIO, S3 SDK).** The destination is not chosen; a directory and `tar` need no new dependency. -## Amendment: restore ownership (M8 review, 2026-09-30) -The review reproduced a race. A restore checked that the target was empty, a +## Amendment: restore ownership (M8 reviews, 2026-09-30 and 2026-10-01) +The first review reproduced a race. A restore checked that the target was empty, a competing actor then created the schema, the restore's `pg_restore` failed, and -its cleanup dropped the other actor's schema. Restores now own what they touch: -- `--objects` is claimed by creating it (`mkdir`, never `-p`). An existing path is - refused and never touched. -- One psql session, held open for the whole restore, takes the restore claim - (advisory 71420004, try-lock, so a concurrent restore refuses at once) and the - upgrade lock (71420002, exclusive). Provisioning, acquisition and lease checks in - that database wait until the restore is verified or removed. -- Cleanup removes only what the invocation created: the schema only if its own - one-transaction `pg_restore` committed, and the directory only if its own `mkdir` - created it. +its cleanup dropped the other actor's schema. The first fix held advisory locks in +an auxiliary session. The follow-up review showed two remaining problems: +- losing that session released authority before verification; +- a nonzero exit was not proof of rollback. -`backup-check.sh` covers three cases: -- a competing schema created just before `pg_restore` survives; -- a concurrent restore refuses, and the first completes; -- a lease acquisition times out while the restore holds the database. +Ownership and the serving gate are now durable state in the target itself. +- **Install** is one transaction (the `pg_restore` SQL, sent to psql with an + explicit COMMIT only after the whole script). It contains: + - the dump; + - a claim row `evolution_restore.claims(NONCE, unpublished, mode, manifest digest, + builds it held current)`; + - every lease fenced, and every current build retired. + + From that commit on, acquisition (both forms), renewal and commits refuse, + whatever happens to any connection or to the server. +- **Publish** runs after the head comparison, the full audit and (in place) the + rights audit. It is one transaction that first locks the invocation's unpublished + claim, then makes exactly the held builds current again. A restore that lost its + claim cannot publish. The claim row stays as a record of the restore. +- **Resolution.** When an install exits nonzero, the claim row decides: + - committed: verification continues; + - absent: rolled back, and the database is reported unchanged (confirmed); + - unreachable: OUTCOME UNKNOWN (exit 5). The target stays unservable, nothing + is removed, and `restore-world.sh --abandon NONCE --objects DIR` resolves it + later. +- **Cleanup** removes schemas only through a transaction that locks this NONCE's + unpublished claim. It removes the objects directory only while that directory + carries this NONCE's marker. A competing restore's or provisioner's state is + never touched. + +`backup-check.sh` covers: +- a competing schema created before install (rollback confirmed; the competitor + survives); +- two concurrent restores (the first installer wins; the loser rolls back and + leaves the winner published); +- acquisition refused after install, both before and after a database restart, + with the build current again after publication; +- a lost install acknowledgement (resolved as committed, then verified); +- an unresolvable install (kept and unservable; a wrong NONCE abandons nothing; the + right one removes everything, and a retry succeeds); +- an occupied objects path. diff --git a/docs/adr/0075-world0-acknowledgment-gate.md b/docs/adr/0075-world0-acknowledgment-gate.md index da524df..fe7c183 100644 --- a/docs/adr/0075-world0-acknowledgment-gate.md +++ b/docs/adr/0075-world0-acknowledgment-gate.md @@ -67,7 +67,9 @@ evidence. Dev and stage proceed, as the roadmap allows. `restore-world.sh --in-place --runtime-role ROLE --accept-rewind` restores a bundle under the world's own identity into a replacement database: - It restores the dump's grants (bundles now keep them; ADR-0074 is amended). -- It fences every restored lease, so the next owner acquires a newer epoch. +- It fences every restored lease, so the next owner acquires a newer epoch. The + world stays unservable (every build retired) until verification publishes it + (ADR-0074 amendment). - It re-runs the migration rights audit for the environment's runtime role. - It prints a `REWIND` line per region naming the tick it was restored to. diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 444e6a4..c84fa0c 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -3087,3 +3087,49 @@ Review: `.cache/reviews/M8_REVIEW_680e7e5-5b482ad.md` (one P1, five P2). objects, 0 failures. - ROADMAP corrected: M8-dev/stage is implemented and locally qualified, but **not passed** until the live gates listed there are run. + +## Review 3a34ae3 follow-up: durable restore gate - 2026-10-01 + +Review: `.cache/reviews/REVIEW_3a34ae3.md`. P1: losing the auxiliary claim session +released authority before verification. P2: a nonzero `pg_restore` exit was taken +as proof of rollback. +- `restore-world.sh` no longer relies on an auxiliary lock session. It works in + three steps, as the ADR-0074 amendment describes: + - an install transaction holds the dump, a claim row with the invocation's + nonce, fenced leases and retired builds; + - verification follows; + - a nonce-checked publish transaction re-enables exactly the held builds. + + Ambiguous outcomes are resolved from the claim row. An unresolvable one is + OUTCOME UNKNOWN (exit 5): the target stays unservable, and + `--abandon NONCE --objects DIR` resolves it later. +- `backup-check.sh` passed in the Release, analysis and ASan/UBSan PostgreSQL + builds, with these cases: + - a competing schema before install: the rollback was confirmed and the + competitor's table survived; + - two concurrent restores: the first installer was verified and published; the + second rolled back, confirmed, and left the winner intact; + - after install, `acquire_region_lease` was refused ("build not current") both + before and after `pg_ctl restart` mid-restore; the restore then verified, and + one current build was restored; + - a lost install acknowledgement: "install outcome resolved: committed", then + verified; + - an unreachable resolution: exit 5, both schemas and the objects kept, + acquisition refused, `--abandon` with a wrong nonce removed nothing, the right + nonce removed everything, and a retry restore succeeded; + - an occupied objects path: refused and untouched; + - damaged bundles: refused, with no `evolution` or `evolution_restore` left. +- The reviewer's claim-loss reproduction, adapted to the renamed hook + (`after-install`), finds no lock to terminate. The runtime acquisition inside it + failed with `build not current` (exit 1), and the restore then published. +- `dr-check.sh` (in-place restore) and `upgrade-check.sh` passed in all three + builds. +- Mutations: + - install without retiring builds: caught ("authority was available before + publication"); + - resolution ignoring a committed claim: caught (exit 5 instead of verified); + - cleanup ignoring the claim's nonce: NOT exercised (the edit pattern failed to + apply). The abandon path also checks the nonce first, so that defense is two + layers deep and only the outer layer is tested. +- Not run: a real network partition around COMMIT. The lost acknowledgement is an + injected test mode (`EVO_RESTORE_TEST_INSTALL_ACK`). diff --git a/tools/backup/restore-world.sh b/tools/backup/restore-world.sh index 12beb34..4baf20f 100644 --- a/tools/backup/restore-world.sh +++ b/tools/backup/restore-world.sh @@ -3,53 +3,67 @@ # # tools/backup/restore-world.sh --bundle DIR --objects DIR # [--in-place --runtime-role ROLE --accept-rewind] +# tools/backup/restore-world.sh --abandon NONCE --objects DIR # # Target connection: PG* environment naming an EMPTY database that the login owns # (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE). -# The restore needs no source role, credential or key: owners and grants are not -# restored, so the target login owns everything restored. --objects must not -# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or -# PATH. +# The restore needs no source role, credential or key: owners are not restored, and +# grants only with --in-place, so the target login owns everything restored. +# evo-world-clone comes from EVO_WORLD_CLONE_BIN or PATH. # # Order, failing closed at each step: # 1. every bundle file matches SHA256SUMS, no file is unlisted, and every object's # name is its digest; nothing is touched before this passes; -# 2. the targets are claimed for this invocation alone (below), then the target -# has no `evolution` schema and runs the manifest's server version; -# 3. pg_restore in one transaction, then objects are copied; +# 2. --objects is claimed by creating it (mkdir, never -p: an existing path is +# refused and never touched) and marked with this invocation's random NONCE; the +# target must have neither `evolution` nor `evolution_restore` and must run the +# manifest's server version; +# 3. INSTALL, one transaction: the dump, a claim row evolution_restore.claims(NONCE, +# unpublished), every lease fenced and every current build retired. From that +# commit on the world is unservable: acquisition, renewal and commits refuse +# until publication, whatever happens to any connection or the server (review +# M8 follow-up P1). Then the objects are copied; # 4. the restored heads equal the MANIFEST heads exactly; -# 5. every region's full history is audited (evo-world-clone audit: each link's -# manifest, payload, trace replay and outcomes, from genesis to the head). -# Ownership (review M8 P1): --objects is claimed by creating it (mkdir, never -p: an -# existing path is refused and never touched), and the database by one psql session -# held open for the whole restore. That session takes the restore claim (advisory -# 71420004, try-lock: a concurrent restore of the same database refuses at once) and -# the upgrade lock exclusively (71420002: provisioning, acquisition and lease checks -# in that database wait until this restore is verified or removed). On failure only -# what this invocation created is removed: the schema only if its own pg_restore -# committed, the objects directory only if its own mkdir created it. So an -# unverified restore can never be mistaken for a world, and a losing restore never -# deletes a winner's state. EVO_RESTORE_TEST_HOOK (tests only) is run by bash with -# $1 = before-restore (claimed, checked) and after-restore (installed, objects copied). The result is a -# verbatim, owner-only copy with the source's world identities: it has no runtime -# grants and is never served. To run it, clone it into an isolated world -# (tools/backup/verify-backup.sh --isolate). +# 5. every region's full history is audited (evo-world-clone audit), and with +# --in-place migrate-postgres.sh audits the runtime role's effective rights; +# 6. PUBLISH, one transaction that first locks this NONCE's unpublished claim: +# exactly the builds the install retired become current again and the claim +# becomes published. A restore that lost its claim cannot publish. +# A failed or interrupted install is resolved, not assumed (review M8 follow-up P2): +# the claim row says whether this invocation's transaction committed. Committed -> +# verification continues; absent -> rolled back (the database was not changed); +# unreachable -> OUTCOME UNKNOWN (exit 5): the target stays unservable and both the +# schema and the objects are kept; resolve it later with --abandon NONCE. +# Cleanup only ever removes what this NONCE owns: the schemas only through a +# transaction that locks this NONCE's unpublished claim, the objects directory only +# while it carries this NONCE's marker. A competing restore's or provisioner's state +# is never touched. The default result is a verbatim, owner-only copy with the +# source's world identities, no runtime grants and never served; to run it, clone it +# into an isolated world (tools/backup/verify-backup.sh --isolate). # # --in-place (M8.8 host-loss recovery for dev/stage, ADR-0075) restores the world -# under its own identity so its hosts can serve it again: the dump's grants are -# restored (the environment's roles must already exist in the target cluster), every -# restored lease is fenced so a new owner acquires under a newer epoch, and -# migrate-postgres.sh audits the runtime role's effective rights. Every acknowledgement -# after the bundle's cut is undone: deaths may reverse and resources may reappear. -# --accept-rewind states that this is acceptable, which it is for dev and stage only; -# World0 launch stays blocked until acknowledgements survive host loss (ADR-0075). -# Fence the old database with fence-database.sh before any of its hosts can return. +# under its own identity so its hosts can serve it again after publication: the +# dump's grants are restored (the environment's roles must already exist in the +# target cluster). Every acknowledgement after the bundle's cut is undone: deaths may +# reverse and resources may reappear. --accept-rewind states that this is +# acceptable, which it is for dev and stage only; World0 launch stays blocked until +# acknowledgements survive host loss (ADR-0075). Fence the old database with +# fence-database.sh before any of its hosts can return. +# +# Exit: 0 verified and published, 2 usage, 3 verification failed (this restore's +# state removed), 4 refused or rolled back (nothing of this restore left), +# 5 outcome unknown (target unservable; evidence kept). +# Tests only: EVO_RESTORE_TEST_HOOK is run by bash with $1 = before-install, +# after-install or before-publish; EVO_RESTORE_TEST_INSTALL_ACK=lost treats a committed +# install as failed (an ambiguous completion), =unreachable also makes its resolution +# unreachable. set -euo pipefail usage() { echo "usage: $0 --bundle DIR --objects DIR [--in-place --runtime-role ROLE --accept-rewind]" >&2 + echo " $0 --abandon NONCE --objects DIR" >&2 exit 2 } -bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0 +bundle="" objects="" in_place=0 runtime_role="" accept_rewind=0 abandon="" while [ $# -gt 0 ]; do case "$1" in --bundle) bundle="${2:-}"; shift 2 ;; @@ -57,10 +71,16 @@ while [ $# -gt 0 ]; do --in-place) in_place=1; shift ;; --runtime-role) runtime_role="${2:-}"; shift 2 ;; --accept-rewind) accept_rewind=1; shift ;; + --abandon) abandon="${2:-}"; shift 2 ;; *) usage ;; esac done -[ -n "$bundle" ] && [ -n "$objects" ] || usage +[ -n "$objects" ] || usage +if [ -n "$abandon" ]; then + [[ "$abandon" =~ ^[0-9a-f]{32}$ ]] && [ -z "$bundle" ] && [ "$in_place" = 0 ] || usage +else + [ -n "$bundle" ] || usage +fi if [ "$in_place" = 1 ]; then [ -n "$runtime_role" ] || { echo "--in-place needs --runtime-role" >&2; exit 2; } [ "$accept_rewind" = 1 ] || @@ -69,10 +89,55 @@ elif [ -n "$runtime_role" ] || [ "$accept_rewind" = 1 ]; then usage fi root="$(cd "$(dirname "$0")/../.." && pwd)" -bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")" +objects="$(realpath -m -- "$objects")" clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}" umask 077 begin_ms="$(date +%s%3N)" +q() { psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 "$@"; } + +# The claim row of NONCE: none, unpublished, published, or unknown (unreachable). +claim_state() { + local nonce="$1" exists state + exists="$(q -c "SELECT to_regclass('evolution_restore.claims') IS NOT NULL" 2> /dev/null)" || { echo unknown; return; } + [ "$exists" = t ] || { echo none; return; } + state="$(q -c "SELECT state FROM evolution_restore.claims WHERE nonce = '$nonce'" 2> /dev/null)" || + { echo unknown; return; } + echo "${state:-none}" +} +# Removes NONCE's unpublished restore: both schemas, in one transaction that first +# locks that claim. Anything else (another claim, a published one) is refused. +drop_claimed() { + q -c "DO \$\$ BEGIN + IF to_regclass('evolution_restore.claims') IS NULL THEN + RAISE EXCEPTION 'no restore claim'; + END IF; + PERFORM 1 FROM evolution_restore.claims WHERE nonce = '$1' AND state = 'unpublished' FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'restore claim % is not unpublished here', '$1'; + END IF; + DROP SCHEMA evolution CASCADE; + DROP SCHEMA evolution_restore CASCADE; + END \$\$" > /dev/null 2>&1 +} +# Removes the objects directory only while it carries NONCE's marker. +drop_objects() { + [ -f "$objects/.restore-claim" ] && [ "$(cat "$objects/.restore-claim")" = "$1" ] && rm -rf -- "$objects" +} + +if [ -n "$abandon" ]; then + state="$(claim_state "$abandon")" + case "$state" in + unpublished) drop_claimed "$abandon" || { echo "ABANDON FAILED: could not remove claim $abandon" >&2; exit 3; } ;; + none) ;; + published) echo "ABANDON REFUSED: claim $abandon is published (a verified world)" >&2; exit 4 ;; + *) echo "ABANDON FAILED: the target database is unreachable" >&2; exit 5 ;; + esac + drop_objects "$abandon" || true + echo "ABANDONED claim $abandon (database state was $state; objects $([ -e "$objects" ] && echo kept || echo removed))" + exit 0 +fi + +bundle="$(realpath -- "$bundle")" refuse() { echo "RESTORE REFUSED: $*" >&2; exit 4; } # 1. Bundle integrity, before anything else is touched. @@ -93,78 +158,112 @@ done <<< "$present" grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format" expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)" [ -n "$expected_heads" ] || refuse "manifest names no region" -privileges=(--no-privileges) +privileges=(--no-privileges) mode=isolated if [ "$in_place" = 1 ]; then grep -qx 'dump privileges' "$bundle/MANIFEST" || refuse "bundle predates in-place restore (no grants in its dump)" - privileges=() + privileges=() mode=in-place fi +manifest_sha="$(sha256sum "$bundle/MANIFEST")"; manifest_sha="${manifest_sha%% *}" +label="$(sed -n 's/^label \([A-Za-z0-9._:-]*\)$/\1/p' "$bundle/MANIFEST" | head -1)" +hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; } -target="${PGDATABASE:-}" -[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE" -[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; } - -# 2. Claim the targets, then check them under the claim. -created_objects=0 installed=0 restored=0 lock_pid="" claim="" +# 2. Claim the objects path, then check the target. +nonce="$(od -An -N16 -tx1 /dev/urandom | tr -d ' \n')" +mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects" +echo "$nonce" > "$objects/.restore-claim" +work="$(mktemp -d /tmp/evo-restore.XXXXXXXX)" +installed=unknown_before published=0 cleanup() { local code=$? - if [ "$restored" != 1 ]; then - if [ "$installed" = 1 ]; then - psql -X --no-password -q -v ON_ERROR_STOP=1 -c 'DROP SCHEMA evolution CASCADE' > /dev/null 2>&1 || true - fi - [ "$created_objects" = 1 ] && rm -rf -- "$objects" - if [ "$installed" = 1 ] || [ "$created_objects" = 1 ]; then - echo "RESTORE FAILED: removed only what this restore created" \ - "(schema=$([ "$installed" = 1 ] && echo yes || echo no), objects=$([ "$created_objects" = 1 ] && echo yes || echo no))" >&2 - fi + if [ "$published" != 1 ]; then + case "$installed" in + no) drop_objects "$nonce" || true ;; + yes) + if drop_claimed "$nonce"; then + drop_objects "$nonce" || true + echo "RESTORE FAILED: removed this restore's schema and objects (claim $nonce)" >&2 + else + echo "RESTORE FAILED: claim $nonce could not be removed; the target stays unservable." \ + "Resolve with: $0 --abandon $nonce --objects $objects" >&2 + code=5 + fi + ;; + unknown_before) drop_objects "$nonce" || true ;; + *) + echo "RESTORE OUTCOME UNKNOWN: claim $nonce. If installed, the target is unservable" \ + "(every build retired). Resolve with: $0 --abandon $nonce --objects $objects" >&2 + code=5 + ;; + esac fi - [ -n "$claim" ] && exec 7>&- # Ends the claim session: its locks are released. - [ -n "$lock_pid" ] && { wait "$lock_pid" 2> /dev/null || true; } - [ -n "$claim" ] && rm -rf -- "$claim" - return "$code" + rm -rf -- "$work" + exit "$code" } trap cleanup EXIT -mkdir -m 700 -- "$objects" 2> /dev/null || refuse "objects target already exists or cannot be created: $objects" -created_objects=1 -claim="$(mktemp -d /tmp/evo-restore-claim.XXXXXXXX)" -mkfifo "$claim/in" -psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 < "$claim/in" > "$claim/out" 2>&1 & -lock_pid=$! -exec 7> "$claim/in" -cat >&7 < /dev/null || break - sleep 0.1 -done -[ -e "$claim/busy" ] && refuse "another restore holds this database" -[ -e "$claim/held" ] || refuse "could not claim the target database ($(head -c 200 "$claim/out"))" -version="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version')")" +echo "restore claim $nonce" +version="$(q -c "SELECT current_setting('server_version')")" grep -qx "database $version" "$bundle/MANIFEST" || refuse "target server $version differs from the manifest" -existing="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" +existing="$(q -c "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")" [ "$existing" = 0 ] || refuse "target database already has an evolution schema" -# 3. Database in one transaction, then objects. A failed pg_restore installed -# nothing (one transaction), so nothing of the database is removed after it. -hook() { [ -z "${EVO_RESTORE_TEST_HOOK:-}" ] || bash -c "$EVO_RESTORE_TEST_HOOK" hook "$1"; } -hook before-restore -pg_restore --no-password --exit-on-error --single-transaction --no-owner "${privileges[@]}" \ - --dbname="$target" "$bundle/database.dump" || refuse "pg_restore failed; the database was not changed" -installed=1 +# 3. INSTALL: dump, claim and the serving gate commit together or not at all. The +# COMMIT is only sent after the complete script: a truncated stream rolls back. +pg_restore --no-owner "${privileges[@]}" --file="$work/restore.sql" "$bundle/database.dump" || + refuse "the dump could not be read" +{ + cat < "$work/install.sql" +hook before-install +status=0 +psql -X --no-password -q -f "$work/install.sql" > "$work/install.log" 2>&1 || status=$? +case "${EVO_RESTORE_TEST_INSTALL_ACK:-}" in + lost) status=1 ;; + unreachable) status=1; export PGPORT=1 PGHOST=/nonexistent ;; +esac +if [ "$status" = 0 ]; then + installed=yes +else + state="$(claim_state "$nonce")" + case "$state" in + unpublished) installed=yes; echo "install outcome resolved: committed (claim $nonce)" ;; + none) + installed=no + sed -n 's/^psql:[^:]*:[0-9]*: //p' "$work/install.log" | head -3 >&2 + refuse "the restore transaction rolled back; the database was not changed (confirmed)" + ;; + *) installed=unknown; exit 5 ;; + esac +fi (cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' | xargs -0 -r -I{} mkdir -p -m 700 -- "$objects/{}") (cd "$bundle/objects" && find . -type f -printf '%P\0' | xargs -0 -r -I{} cp -- {} "$objects/{}") -hook after-restore +hook after-install # 4. Heads exactly as recorded. -actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c " +actual_heads="$(q -F ' ' -c " SELECT h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick, h.commit_id, encode(h.digest, 'hex'), encode(c.build_identity, 'hex') FROM evolution.region_heads h @@ -173,7 +272,8 @@ actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c " AND c.commit_sequence = h.commit_sequence" | sort)" [ "$actual_heads" = "$expected_heads" ] || { echo "restored heads differ from the manifest" >&2; exit 3; } -# 5. Full history audit of every region. +# 5. Full history audit of every region (the CLI's retained list, not the database's +# temporarily retired state), then the runtime rights audit for --in-place. pg=() if [ -n "${PGSERVICEFILE:-}" ]; then pg=(--pg-service "$PGSERVICEFILE") @@ -195,17 +295,38 @@ while read -r world region _ _ tick _; do echo "region $world/$region $line" done <<< "$expected_heads" if [ "$in_place" = 1 ]; then - # No lease from before the loss survives: the next owner acquires a newer epoch. - psql -X --no-password -q -v ON_ERROR_STOP=1 -c \ - 'UPDATE evolution.region_leases SET holder = NULL, expires_at = NULL WHERE holder IS NOT NULL' > /dev/null EVO_PG_RUNTIME_ROLE="$runtime_role" bash "$root/tools/migrate-postgres.sh" > /dev/null || { echo "runtime role $runtime_role failed the rights audit" >&2; exit 3; } +fi + +# 6. PUBLISH under this NONCE's claim; an ambiguous completion is resolved likewise. +hook before-publish +status=0 +q -c "DO \$\$ DECLARE held bytea[]; BEGIN + SELECT c.held_builds INTO held FROM evolution_restore.claims c + WHERE c.nonce = '$nonce' AND c.state = 'unpublished' FOR UPDATE; + IF NOT FOUND THEN + RAISE EXCEPTION 'restore claim % lost before publication', '$nonce'; + END IF; + UPDATE evolution.retained_builds SET retired_at = NULL WHERE build_identity = ANY (held); + UPDATE evolution_restore.claims SET state = 'published', published_at = clock_timestamp() + WHERE nonce = '$nonce'; +END \$\$" > /dev/null 2>&1 || status=$? +if [ "$status" != 0 ]; then + case "$(claim_state "$nonce")" in + published) ;; + unpublished) echo "publication failed" >&2; exit 3 ;; + *) installed=unknown; exit 5 ;; + esac +fi +published=1 +rm -f -- "$objects/.restore-claim" +if [ "$in_place" = 1 ]; then created="$(sed -n 's/^created //p' "$bundle/MANIFEST")" while read -r world region _ _ tick _; do echo "REWIND region $world/$region restored at tick $tick (backup $created): every acknowledgement after this tick is undone" done <<< "$expected_heads" fi -restored=1 what="RESTORE VERIFIED" [ "$in_place" = 1 ] && what="RESTORE IN PLACE VERIFIED" -echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))" +echo "$what regions=$(wc -l <<< "$expected_heads") commits=$commits claim=$nonce duration_ms=$(( $(date +%s%3N) - begin_ms ))" diff --git a/tools/ci/backup-check.sh b/tools/ci/backup-check.sh index 3de1db0..f5c9227 100644 --- a/tools/ci/backup-check.sh +++ b/tools/ci/backup-check.sh @@ -60,7 +60,7 @@ expect_refusal() { # expect_refusal --bundle "$2" --objects "$work/restored-$case_n" > "$work/case-$case_n.log" 2>&1 || code=$? [ "$code" = "$1" ] || { cat "$work/case-$case_n.log" >&2; fail "$3: exit $code, expected $1"; } [ ! -e "$work/restored-$case_n" ] || fail "$3: restored objects were left behind" - [ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" = 0 ] || + [ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')")" = 0 ] || fail "$3: a restored schema was left behind" } damaged() { rm -rf "$work/damaged"; cp -a "$work/bundle" "$work/damaged"; } @@ -90,46 +90,92 @@ PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle [ "$code" = 4 ] && grep -q 'already has an evolution schema' "$work/twice.log" || fail "a second restore into one database" [ "$(PGDATABASE=evo_restore_twice psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] || fail "the refused second restore changed the first" -# Ownership races (review M8 P1). A competing actor creates the schema between this -# restore's checks and its pg_restore: the restore fails and leaves that schema alone. +# Ownership and failure paths (M8 review P1; follow-up P1/P2). Each case names the +# boundary it interrupts; only this invocation's state may ever be removed. +restore() { # restore [env...]: runs restore-world.sh, sets rc. + local db="$1" dir="$2"; shift 2 + rc=0 + env PGDATABASE="$db" "$@" bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \ + --objects "$dir" > "$work/$(basename "$dir").log" 2>&1 || rc=$? +} +schemas() { PGDATABASE="$1" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname IN ('evolution', 'evolution_restore')"; } +acquire_refused() { # A lease acquisition in is refused (the world is unservable). + ! PGDATABASE="$1" psql -XAt -v ON_ERROR_STOP=1 \ + -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('b1', 32), 'hex'))" \ + > "$work/acquire-$1.log" 2>&1 && grep -q 'build not current' "$work/acquire-$1.log" +} +log_of() { cat "$work/$(basename "$1").log" >&2; } + +# A competing actor creates the schema after this restore's checks: its install +# transaction rolls back (confirmed), and the competitor's schema survives. createdb evo_restore_race -code=0 -PGDATABASE=evo_restore_race EVO_RESTORE_TEST_HOOK='[ "$1" = before-restore ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true' \ - bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/race-a" \ - > "$work/race-a.log" 2>&1 || code=$? -[ "$code" = 4 ] && grep -q 'pg_restore failed; the database was not changed' "$work/race-a.log" || - { cat "$work/race-a.log" >&2; fail "a pre-empted restore: exit $code"; } +restore evo_restore_race "$work/race-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && psql -XAtq -c "CREATE SCHEMA evolution" -c "CREATE TABLE evolution.winner(x int)" || true' +[ "$rc" = 4 ] && grep -q 'rolled back; the database was not changed (confirmed)' "$work/race-a.log" || + { log_of race-a; fail "a pre-empted restore: exit $rc"; } [ "$(PGDATABASE=evo_restore_race psql -XAtc "SELECT count(*) FROM pg_tables WHERE schemaname = 'evolution' AND tablename = 'winner'")" = 1 ] || fail "a pre-empted restore deleted the competing schema" [ ! -e "$work/race-a" ] || fail "a pre-empted restore left its own objects directory" -# Two restores of one database: the second refuses at once and touches nothing of -# the first, which completes. While the first holds the database (after its install), -# a lease acquisition there waits instead of running against unverified state. + +# Two restores of one database: whichever installs first wins; the other rolls back +# and leaves the winner published and intact. createdb evo_restore_pair -PGDATABASE=evo_restore_pair EVO_RESTORE_TEST_HOOK=' - if [ "$1" = before-restore ]; then touch "'"$work"'/pair-claimed"; sleep 3; fi - if [ "$1" = after-restore ]; then - psql -XAt -c "SET lock_timeout = '"'"'1s'"'"'" -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/pair-acquire.log" 2>&1 || true - fi' bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" --objects "$work/pair-a" \ - > "$work/pair-a.log" 2>&1 & +restore evo_restore_pair "$work/pair-a" EVO_RESTORE_TEST_HOOK='[ "$1" = before-install ] && sleep 4 || true' & pair_pid=$! -for _ in $(seq 1 100); do [ -e "$work/pair-claimed" ] && break; sleep 0.1; done -code=0 -PGDATABASE=evo_restore_pair bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \ - --objects "$work/pair-b" > "$work/pair-b.log" 2>&1 || code=$? -[ "$code" = 4 ] && grep -q 'another restore holds this database' "$work/pair-b.log" || - { cat "$work/pair-b.log" >&2; fail "a concurrent restore: exit $code"; } -[ ! -e "$work/pair-b" ] || fail "the refused concurrent restore left its objects directory" +sleep 1 +restore evo_restore_pair "$work/pair-b" +[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-b.log" || { log_of pair-b; fail "the first installer: exit $rc"; } code=0; wait "$pair_pid" || code=$? -[ "$code" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/pair-a.log" || - { cat "$work/pair-a.log" >&2; fail "the first of two restores: exit $code"; } -grep -q 'lock timeout' "$work/pair-acquire.log" || - { cat "$work/pair-acquire.log" >&2; fail "a lease acquisition ran during the restore"; } +grep -q 'rolled back; the database was not changed (confirmed)' "$work/pair-a.log" && [ ! -e "$work/pair-a" ] || + { log_of pair-a; fail "the second installer: exit $code"; } +[ "$(PGDATABASE=evo_restore_pair psql -XAtc "SELECT state FROM evolution_restore.claims")" = published ] && + [ "$(PGDATABASE=evo_restore_pair psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] || + fail "the losing restore disturbed the winner" + +# After install and before publication the world is unservable, durably: across a +# database restart (every connection and lock lost) acquisition is still refused, and +# the restore then verifies and publishes; afterwards the build is current again. +createdb evo_restore_gate +restore evo_restore_gate "$work/gate" EVO_RESTORE_TEST_HOOK=' + if [ "$1" = after-install ]; then + psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-before.log" 2>&1 || true + pg_ctl -D "$PGHOST/data" -m fast -w restart > /dev/null + psql -XAt -c "SELECT * FROM evolution.acquire_region_lease(300, 1, 20000, decode(repeat('"'"'b1'"'"', 32), '"'"'hex'"'"'))" > "'"$work"'/gate-after.log" 2>&1 || true + fi' +[ "$rc" = 0 ] && grep -q '^RESTORE VERIFIED regions=1 ' "$work/gate.log" || { log_of gate; fail "a restore across a restart: exit $rc"; } +grep -q 'build not current' "$work/gate-before.log" && grep -q 'build not current' "$work/gate-after.log" || + { cat "$work/gate-before.log" "$work/gate-after.log" >&2; fail "authority was available before publication"; } +[ "$(PGDATABASE=evo_restore_gate psql -XAtc 'SELECT count(*) FROM evolution.retained_builds WHERE retired_at IS NULL')" = 1 ] || + fail "publication did not restore the current build" + +# A committed install whose completion is lost is resolved from the claim and +# verified, not reported as unchanged. +createdb evo_restore_lost +restore evo_restore_lost "$work/lost" EVO_RESTORE_TEST_INSTALL_ACK=lost +[ "$rc" = 0 ] && grep -q '^install outcome resolved: committed' "$work/lost.log" && + grep -q '^RESTORE VERIFIED regions=1 ' "$work/lost.log" || { log_of lost; fail "a lost install acknowledgement: exit $rc"; } + +# An install whose outcome cannot be resolved keeps everything, stays unservable, +# and is removed only by --abandon with its claim. +createdb evo_restore_unknown +restore evo_restore_unknown "$work/unknown" EVO_RESTORE_TEST_INSTALL_ACK=unreachable +nonce="$(sed -n 's/^restore claim \([0-9a-f]\{32\}\)$/\1/p' "$work/unknown.log")" +[ "$rc" = 5 ] && grep -q '^RESTORE OUTCOME UNKNOWN' "$work/unknown.log" && [ -d "$work/unknown" ] && [ -n "$nonce" ] || + { log_of unknown; fail "an unresolvable install: exit $rc"; } +[ "$(schemas evo_restore_unknown)" = 2 ] && acquire_refused evo_restore_unknown || fail "an unresolved restore is servable" +code=0 +PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon 00000000000000000000000000000000 \ + --objects "$work/unknown" > "$work/abandon-wrong.log" 2>&1 || code=$? +[ "$(schemas evo_restore_unknown)" = 2 ] && [ -d "$work/unknown" ] || fail "--abandon with another claim removed state" +PGDATABASE=evo_restore_unknown bash "$root/tools/backup/restore-world.sh" --abandon "$nonce" \ + --objects "$work/unknown" > "$work/abandon.log" 2>&1 || { cat "$work/abandon.log" >&2; fail "--abandon"; } +[ "$(schemas evo_restore_unknown)" = 0 ] && [ ! -e "$work/unknown" ] || fail "--abandon left state behind" +restore evo_restore_unknown "$work/unknown-retry" +[ "$rc" = 0 ] || { log_of unknown-retry; fail "a restore after --abandon: exit $rc"; } + # An existing objects path is never claimed or removed. mkdir "$work/occupied"; echo keep > "$work/occupied/mine" createdb evo_restore_occupied -code=0 -PGDATABASE=evo_restore_occupied bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \ - --objects "$work/occupied" > "$work/occupied.log" 2>&1 || code=$? -[ "$code" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] || fail "an occupied objects path: exit $code" +restore evo_restore_occupied "$work/occupied" +[ "$rc" = 4 ] && [ "$(cat "$work/occupied/mine")" = keep ] && [ "$(schemas evo_restore_occupied)" = 0 ] || + fail "an occupied objects path: exit $rc" echo "backup and restore qualification passed (backup tick $backup_tick, acknowledged $acked_before, source $source_tick)"