feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)

Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable
objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a
fail-closed restore that audits every region's full history on independent
storage; an isolated rebuild through clones that serves under load; an
operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in
every PostgreSQL build, with corruption, missing-object, tampered-manifest and
non-empty-target refusals. Destination, encryption, schedule and WAL archiving
remain on the backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 15:51:04 -04:00
co-authored by Claude Opus 5.5
parent 4a900b0bd3
commit 5512fdec54
19 changed files with 869 additions and 13 deletions
+2
View File
@@ -86,6 +86,8 @@ jobs:
bash tools/ci/docker-build.sh postgres-tools "$image" aevolution-postgres-cache
docker run --rm --network none --read-only --cap-drop ALL \
--entrypoint /opt/postgres/bin/psql "$image" --version
docker run --rm --network none --read-only --cap-drop ALL \
--entrypoint /opt/postgres/bin/pg_dump "$image" --version
- name: Publish configured migration tools
if: ${{ gitea.ref == 'refs/heads/main' && gitea.event_name != 'pull_request' && vars.HARBOR_POSTGRES_REPOSITORY != '' }}
env:
+5
View File
@@ -59,6 +59,7 @@ WORKDIR /source
COPY deploy/ deploy/
COPY tools/ci/deploy-policy-check.py tools/ci/
COPY tools/ops/ tools/ops/
COPY tools/backup/ tools/backup/
RUN python3 tools/ci/deploy-policy-check.py
# M8.5: monitoring configuration and alert tests with the pinned images' own tools.
COPY --from=quay.io/prometheus/prometheus@sha256:6976aa8a60fec930796ce5772b8d12da7a318a5daa8d40d69c5c7819a05eeed7 /bin/promtool /usr/local/bin/promtool
@@ -109,9 +110,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends libssl3t64 zlib
&& rm -rf /var/lib/apt/lists/* \
&& groupadd --gid 10001 evo && useradd --uid 10001 --gid 10001 --no-create-home evo
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/psql /opt/postgres/bin/psql
# M8.7 backups (ADR-0074): the same pinned PostgreSQL build's dump tools.
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/pg_dump /opt/postgres/bin/pg_dump
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/pg_restore /opt/postgres/bin/pg_restore
COPY --from=postgres-toolchain /source/.cache/postgres/install/lib/libpq.so* /opt/postgres/lib/
COPY --from=postgres-toolchain /source/.cache/postgres/postgresql-18.6/COPYRIGHT /usr/share/doc/evo-postgres/COPYRIGHT
COPY tools/migrate-postgres.sh /opt/evo/tools/migrate-postgres.sh
COPY tools/backup/backup-world.sh /opt/evo/tools/backup-world.sh
COPY migrations/ /opt/evo/migrations/
ENV PATH="/opt/postgres/bin:${PATH}" LD_LIBRARY_PATH=/opt/postgres/lib
USER 10001:10001
@@ -1,5 +1,5 @@
# Egress from migrate, store-check and region pods to exactly the AEvolution
# PostgreSQL VM (pg-forge-01) and nothing else (ADR-0070).
# Egress from migrate, store-check, region and operator backup pods to exactly the
# AEvolution PostgreSQL VM (pg-forge-01) and nothing else (ADR-0070, ADR-0074).
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
@@ -13,7 +13,7 @@ spec:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values: [migrate, store-check, region]
values: [migrate, store-check, region, backup]
policyTypes: [Egress]
egress:
- to:
+3 -1
View File
@@ -7,7 +7,9 @@ names it. Remove an item only when it is done, and record where it was done.
| Item | Raised | Why deferred | Needed before |
|---|---|---|---|
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
| **Backup destination for dev/stage stores.** There are no backups yet (maintainer, 2026-09-29). The plan is to build backup tooling for the AEvolution PostgreSQL VM and each `aevolution-objects` volume up to a pluggable destination, then add the destination. It must be off the host for anything beyond disposable environments. | M8.3, 2026-09-29 | The maintainer will supply a destination later | M8.7 backup and restore; any World0 data |
| **Backup destination for dev/stage stores.** The tooling now exists up to the destination (ADR-0074): `tools/backup/cluster-backup.sh <env> --to DIR` writes a verified bundle to any directory, and `verify-backup.sh` restore-tests it. Still to choose: the permanent off-host destination, and with it (1) bundle encryption and key custody, (2) a scheduled CronJob synced by Argo, (3) retention (proposal: 30 daily, 12 weekly, 12 monthly), (4) incremental object copies. | M8.3, 2026-09-29; M8.7, 2026-09-30 | The maintainer will supply a destination later | Scheduled backups; any World0 data; M8.8 |
| **WAL archiving on pg-forge-01.** Bundles are logical backups: the recovery point is the last bundle. Continuous archiving (`archive_command` or `pg_receivewal`) plus periodic base backups would allow point-in-time recovery. It needs the backup destination. | M8.7, 2026-09-30 | Needs the destination and VM configuration | M8.8 host-loss RPO decision |
| **Harbor retention of build images.** A bundle's MANIFEST names every build (region image digest) that wrote history; recovery and replay need those images. Confirm Harbor's retention/tag-immutability policy keeps them (or pin them in a protected project). | M8.7, 2026-09-30 | Not yet checked | M8.8; any upgrade barrier that retires a build |
| **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 |
| **Store Job values.** Supply the published registry digests for `aevolution-postgres-tools` and `aevolution-server` (from each CI Publish step's `Digest:` line), and the NOLOGIN runtime group role for dev (`aevolution`) and for stage (`aevolution_stage`). Then include `stores/<env>` in each environment's path. | M8.3, 2026-09-29 | Values pending | M8.3 in-cluster evidence |
| **Alert destination.** Alertmanager routes every alert to a null receiver (ADR-0072). Choose where alerts go (Discord, email, a webhook), and optionally a heartbeat receiver for the always-firing Watchdog. Receiver credentials go in a Secret. | M8.5, 2026-09-30 | The maintainer will decide later | Any claim that alerts reach a person; M8.6 alert evidence; World0 |
+10
View File
@@ -650,3 +650,13 @@ M8.6 implemented and locally qualified (2026-09-30), under ADR-0073:
and an operator-driven database outage;
- RTO and zero-lost-acknowledgement checks.
The in-cluster drill results wait on the maintainer.
M8.7 implemented and locally qualified (2026-09-30), under ADR-0074:
- online, snapshot-consistent world bundles (database dump, immutable objects,
MANIFEST of versions, builds and heads, checksums);
- a fail-closed restore that audits every region's full history on independent
storage, and an isolated rebuild through clones that serves under load;
- corruption, missing-object, tampered-manifest and non-empty-target refusals;
- an operator-run in-cluster backup to a directory off the cluster.
Pending the maintainer's destination: encryption, schedule, retention and WAL
archiving (docs/BACKLOG.md). In-cluster bundle evidence waits on a run.
+20
View File
@@ -1105,3 +1105,23 @@ It restores the Git configuration on exit.
Earlier (M8.4 follow-up): `REGION_RETAINED` keeps earlier builds recoverable
across image upgrades.
## M8.7 backup and restore (2026-09-30)
ADR-0074 adds world backup bundles and a verified restore:
- `tools/backup/backup-world.sh`: an online backup cut by one exported database
snapshot (MANIFEST heads and `pg_dump` read the same snapshot), then the
immutable objects, each checked against its digest; `SHA256SUMS` over all of it.
- `tools/backup/restore-world.sh`: checksums first, then an owner-only restore into
an empty database, exact head comparison and a full history audit of every region;
any failure removes what it restored.
- `tools/backup/verify-backup.sh`: the restore on a private throwaway cluster with
fresh roles, then (`--isolate WORLD`) a clone into an isolated world that serves
under synthetic load and drains.
- `tools/backup/cluster-backup.sh <dev|stage> --to DIR`: an operator Pod (read-only
objects volume, migrator login) whose bundle is streamed off the cluster.
- The postgres-tools image now carries `pg_dump` and `pg_restore` from the pinned
18.6 build; the database egress policy admits the `backup` component.
The destination, encryption, schedule, retention and WAL archiving wait on the
maintainer (docs/BACKLOG.md).
+172
View File
@@ -0,0 +1,172 @@
# ADR-0074: World backup bundles and verified restore
## Status
Accepted for M8.7, 2026-09-30. It builds on:
- ADR-0061 (immutable, content-addressed objects; upload verified before commit);
- ADR-0067 (clones and history audit);
- ADR-0070 (dev/stage stores);
- ADR-0073 (synthetic load).
The permanent backup destination, encryption, scheduling and WAL archiving wait
on the maintainer's choice of destination (docs/BACKLOG.md). Until then, this ADR
covers everything up to that destination.
## Context
M8.7 requires:
- a coordinated manifest covering the database, the objects, the versions, the
binaries and any in-flight state;
- a restore onto independent storage;
- retention and GC that respect clone and backup pins.
The evidence asked for is:
- rebuilding an isolated world from a backup;
- corruption and missing-object failures;
- credential and key recovery.
The named risk is a backup that exists but cannot reconstruct causal state.
docs/infrastructure/OPERATIONS.md proposes a coordinated admission barrier. Two
facts make a barrier unnecessary for a single database:
- a region's durable state is one PostgreSQL database plus immutable objects;
- every object a committed row references was verified durable before that row
committed.
## Decision
**Bundle.** `tools/backup/backup-world.sh --objects DIR --out DIR` writes:
| File | Content |
|---|---|
| `MANIFEST` | Format, time and label. The server version. Every schema migration and its checksum. Every retained build (image digest) and whether it is retired. Every region head (world, region, frame, sequence, tick, commit, digest, build). Row counts, and the object count and bytes |
| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners and no grants |
| `objects/w<world>/r<region>/<sha256>` | Every object in the store, each checked against its name while it is copied |
| `SHA256SUMS` | Checksums of every other file |
**The cut.** One psql session opens a `REPEATABLE READ, READ ONLY` transaction and
exports its snapshot. The session reads the heads for the MANIFEST, then runs
`pg_dump --snapshot` on that same snapshot while the session holds it. Objects are
copied afterwards.
- Objects are never overwritten or deleted, and no GC exists. So every object the
snapshot references is present and final when it is copied.
- Objects written after the snapshot are copied as harmless orphans.
- Nothing is written to the source, and admission never stops. The backup runs
online while the host commits.
**In-flight state.**
- Every acknowledgement made before the snapshot is in the bundle, because an
acknowledgement follows its commit.
- A commit whose transaction completes after the snapshot is not in the bundle.
It was not acknowledged at the cut.
- Unacknowledged inputs are never durable, by design (ADR-0064).
- Lease rows are copied, but a restored copy is never served under them (see
below).
- There are no cross-region transfers yet. When they exist, they live in the same
database and are covered by the same snapshot.
**Restore.** `tools/backup/restore-world.sh --bundle B --objects DIR` targets an
empty database owned by the restoring login, and fails closed in this order:
1. Every file matches `SHA256SUMS`, no file is unlisted, and every object's name
is its digest. Nothing is touched before this passes.
2. The target has no `evolution` schema and runs the manifest's server version.
3. `pg_restore` runs in one transaction, then the objects are copied.
4. The restored heads equal the MANIFEST heads exactly.
5. Every region's full history is audited with `evo-world-clone audit`. The audit
checks every link's manifest, payload, trace replay and outcomes, from genesis
to the head.
A failure after step 2 removes the restored schema and objects, so an unverified
restore cannot be mistaken for a world. The result is a verbatim, owner-only copy
that keeps the source's world identities. It has no runtime grants and is never
served.
**Isolated rebuild.** `tools/backup/verify-backup.sh BUNDLE --isolate WORLD`:
1. Starts a private, throwaway PostgreSQL cluster on local disk.
2. Restores the bundle as a fresh login that owns only the restore database.
3. Creates a separate database with its own owner, runtime group and login, and
migrates it with `migrate-postgres.sh`.
4. Clones every region of the source world into `WORLD` with
`evo-world-clone clone`, which audits the source history first. Provenance is
kept in `region_origins`.
5. Runs `evo-server` on each clone under synthetic load until it acknowledges 20
ticks past the backup head, then drains it.
This is OPERATIONS.md steps 3–5 for a clone. Restoring in place under the source's
own identity, with the old writers fenced, is host-loss recovery and belongs to
M8.8.
**Credentials and keys.**
- A bundle carries no roles, grants, passwords or connection details.
- Restoring needs only a login that owns an empty database, and no source
credential. `backup-check.sh` and `verify-backup.sh` prove this with fresh,
unrelated roles.
- Bundles are not encrypted yet, so there is no key to lose. Encryption and its key
custody are chosen together with the destination. They are required before any
World0 data leaves the host.
**Retention and GC.**
- Each bundle is a full, self-contained copy, and no bundle depends on another.
Deleting one never breaks another.
- Clones copy their payloads into the target namespace (ADR-0067), so a clone does
not depend on its source either.
- The object store has no delete, and history never expires.
- Any future GC must prove reachability from every live head, clone origin and
retained build. Bundles hold their own copies, so GC cannot damage them.
- Retention of bundles at the destination (the OPERATIONS proposal: 30 daily, 12
weekly, 12 monthly) is applied when the destination exists.
**Binaries.** The MANIFEST names every build identity that wrote history. These
are the region image digests that recovery and replay need. Harbor must keep those
images; its retention policy is not yet verified (backlog).
**In the cluster.** `tools/backup/cluster-backup.sh <dev|stage> --to DIR [--verify WORLD]`
is an operator action, like `fault-drill.sh`, and refuses World0 before kubectl
runs. It works as follows:
- It runs one short-lived Pod (component `backup`) with:
- the environment's pinned postgres-tools image, which now includes `pg_dump`,
`pg_restore` and `backup-world.sh`;
- the migrator login;
- the `aevolution-objects` volume mounted read-only on the store node;
- a 12 GiB emptyDir for the bundle.
- The database egress policy admits the `backup` component.
- The bundle is streamed out with `kubectl exec tar` and its checksums are checked
locally. Then the Pod is deleted, including on failure.
A scheduled CronJob, synced by Argo, follows once the destination exists.
**Qualification.** `tools/ci/backup-check.sh` runs inside `postgres-check.sh` for
every PostgreSQL build:
- It backs up a real `evo-server` while the server commits under synthetic load.
The backup head covers every acknowledgement made before the backup, and the
source then moves past it.
- It runs `verify-backup.sh --isolate` on a second cluster.
- It restores damaged bundles, each of which must be refused, leaving nothing
behind: a corrupt object, an unlisted file, a damaged dump, a missing payload
that has consistent checksums, and a MANIFEST head that disagrees with the
database.
- It restores a second time into a database that already holds a world, which
must be refused.
## Consequences
- **RPO.** The recovery point is the last verified bundle. Without WAL archiving,
this is a logical backup and cannot recover to an arbitrary point in time.
Continuous archiving on pg-forge-01 needs the destination (backlog). M8.8 decides
the host-loss RPO for World0.
- **Full copies.** Every bundle copies the whole object store. That is fine at
dev/stage sizes (the objects volume is 10 GiB). Incremental object copies, which
are safe because objects are content-addressed, come with the destination.
- **Off host.** A bundle is off the host only if `--to` is on another machine. The
maintainer's workstation qualifies only if it is not the hypervisor host.
- **Restore test.** A scheduled job or a successful upload is not a restore test.
Only `verify-backup.sh` output counts as restore evidence.
## Alternatives rejected
- **Physical base backups (`pg_basebackup`) as the only copy.** They restore the
whole server, with its roles, and cannot be audited per region or restored as an
isolated world without a second step. They remain the likely basis for PITR once
WAL archiving exists.
- **An admission barrier before the cut.** It is unnecessary given
upload-before-commit and immutable objects, and it would pause the world.
- **Restoring directly as a runnable world with fresh grants.** It would serve the
source's world identity from a second place, which could become competing
authority. Isolation goes through the clone path, which re-identifies the world.
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
chosen; a directory and `tar` need no new dependency.
+1
View File
@@ -84,3 +84,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
- [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending.
+36
View File
@@ -2916,3 +2916,39 @@ Fixes (this segment):
MSVC CTest 59/59 (the server is not part of the Windows build).
`fault-drill.sh` passed `bash -n`.
- Pending: a new image, then re-running the full, empty and db-outage scenarios.
## M8.7 backup and restore - 2026-09-30
- `tools/ci/backup-check.sh` (inside `postgres-check.sh`; real PostgreSQL 18.6, a
real `evo-server` under synthetic load) passed in the Release, analysis and
ASan/UBSan PostgreSQL builds:
- An online backup taken while the server committed recorded head tick 60, at or
past every acknowledgement before it (60). The source then moved on to tick 101.
- `verify-backup.sh --isolate 301` on a second, throwaway cluster, as fresh
roles: the restore audited 13 commits to tick 60 and matched the MANIFEST. The
clone into world 301 resumed at `epoch=1 committed=60`, acknowledged to tick
82, and drained with exit 0.
- Refused bundles, each leaving no schema and no objects behind:
- corrupt object, unlisted file, damaged dump: exit 4 at the checksum step;
- missing payload object with consistent checksums: exit 3,
`audit failed missing_payload sequence=0`;
- MANIFEST head changed: exit 3, heads differ.
- A second restore into a database that already holds a world was refused
(exit 4), and the first restore was left unchanged.
- The MANIFEST names no credentials.
- Mutations, each caught by `backup-check.sh`:
- restore without the history audit: the missing object was accepted (exit 0);
- a failed restore that keeps its files: restored objects were left behind;
- `pg_dump` outside the exported snapshot (2 s later): the restored heads
differed from the MANIFEST, and verification failed.
- The PostgreSQL migration, rollback and permission suites passed in all three
builds. `deploy-policy-check.py`: 67 objects, 0 failures.
`cluster-backup.sh` refuses World0 before kubectl (policy rule).
- `bash -n` passed on every new script. shellcheck is not installed in WSL, so it
did not run.
- Not executed:
- the postgres-tools image build with `pg_dump` (CI builds it on push);
- `cluster-backup.sh` against dev or stage;
- any restore of in-cluster data.
- The permanent destination, encryption, schedule and WAL archiving are not
implemented; they are pending the maintainer's choice of destination.
+27 -1
View File
@@ -192,7 +192,7 @@ To delete World0 data or identity on purpose:
1. Record the operation and its approval.
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
example a short-lived client certificate with that organization.
3. Take and verify a backup first (M8.7).
3. Take and verify a backup first (M8.7; see "Backups" below).
Removing the admission policy itself is the break-glass of last resort. It needs
cluster-admin and must be recorded the same way. Never test by deleting World0: the
@@ -206,3 +206,29 @@ verify script uses server-side dry runs.
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
node-local until M8.3.
## Backups (M8.7, ADR-0074)
Dev and stage only; the tools refuse World0. From a machine with kubectl access
that is not the hypervisor host:
```bash
# 1. Sync the environment's application once so the egress policy admits the
# backup component and the store overlay pins a postgres-tools image that
# contains pg_dump (any image built after ADR-0074).
# 2. Back up to a local directory (the destination until one is chosen):
tools/backup/cluster-backup.sh dev --to /path/to/backups
# 3. Restore-test it on independent storage and rebuild it as isolated world 900
# (WSL: the pinned PostgreSQL build and the Linux PostgreSQL build of the tools).
PATH="$PWD/.cache/postgres/install/bin:$PATH" \
LD_LIBRARY_PATH="$PWD/.cache/postgres/install/lib" \
EVO_WORLD_CLONE_BIN="$PWD/build/postgres-linux/evo-world-clone" \
EVO_SERVER_BIN="$PWD/build/postgres-linux/evo-server" \
tools/backup/verify-backup.sh /path/to/backups/dev-<time> --isolate 900
```
Only `verify-backup.sh` output that ends in `PASS restore verified and world ...
rebuilt as isolated world ...` counts as restore evidence. The restore uses fresh
roles on a throwaway cluster; nothing reaches the source database. A restored copy
keeps the source world identity and is never served; restoring in place after host
loss is M8.8.
+9 -4
View File
@@ -1,7 +1,10 @@
# Operations and disaster recovery
Status: proposal requiring measured acceptance before World0. No deployment,
backup or distributed restore has been exercised in this foundation.
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
implements snapshot-consistent world backup bundles and a verified restore on
independent storage with an isolated rebuild; the destination, encryption,
schedule and WAL archiving are pending, and no host-loss restore in place has been
exercised (M8.8).
| Failure | Proposed objective / gate |
|---|---|
@@ -36,8 +39,10 @@ pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
history does not expire with backup retention. GC respects all clone/backup pins
and complete replay chains.
1. First implementation uses a coordinated admission/transfer barrier, avoiding
an unfinished distributed-snapshot algorithm.
1. First implementation (ADR-0074) needs no admission barrier: one exported
database snapshot is the cut, and objects are immutable and verified before
commit, so copying them after the snapshot is consistent. A barrier returns
only if state ever spans more than one database.
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
transfer state, region watermarks and versions into the manifest.
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
+105
View File
@@ -0,0 +1,105 @@
#!/usr/bin/env bash
# M8.7 world backup (ADR-0074): one consistent, self-verifying bundle of an
# AEvolution database and its object store.
#
# tools/backup/backup-world.sh --objects DIR --out DIR [--label TEXT]
#
# Connection: PG* environment (PGSERVICE/PGSERVICEFILE, or PGHOST/PGDATABASE/...),
# optionally EVO_PGPASS_SOURCE (copied to a private file, as migrate does). The
# login must read the whole `evolution` schema (the migrator/owner). Never print
# the connection or enable shell tracing here.
#
# Cut: one exported REPEATABLE READ snapshot names the region heads recorded in the
# MANIFEST and is the snapshot pg_dump reads. Objects are copied afterwards: every
# object a committed row references was verified durable before its commit, and
# objects are immutable and never deleted (no GC), so each one the snapshot
# references is present and final when copied. Objects written after the snapshot
# are copied too and are harmless orphans in the bundle. Nothing is written to the
# source database or the object volume, and no admission barrier is needed.
#
# Bundle (DIR): MANIFEST, database.dump (pg_dump custom format, schema `evolution`,
# no owners or grants: a restore never needs the source's roles or credentials),
# objects/w<world>/r<region>/<sha256>, SHA256SUMS over every other file. A source
# object whose bytes differ from its name fails the backup.
set -euo pipefail
usage() { echo "usage: $0 --objects DIR --out DIR [--label TEXT]" >&2; exit 2; }
objects="" out="" label=""
while [ $# -gt 0 ]; do
case "$1" in
--objects) objects="${2:-}"; shift 2 ;;
--out) out="${2:-}"; shift 2 ;;
--label) label="${2:-}"; shift 2 ;;
*) usage ;;
esac
done
[ -n "$objects" ] && [ -n "$out" ] || usage
objects="${objects%/}"
[[ "$label" =~ ^[A-Za-z0-9._:-]*$ ]] || { echo "label: letters, digits and ._:- only" >&2; exit 2; }
[ -d "$objects" ] && [ ! -L "$objects" ] || { echo "objects: not a real directory: $objects" >&2; exit 2; }
if [ -e "$out" ] && [ -n "$(ls -A "$out")" ]; then echo "out: not empty: $out" >&2; exit 2; fi
umask 077
if [ -n "${EVO_PGPASS_SOURCE:-}" ]; then
export PGPASSFILE
PGPASSFILE="$(mktemp /tmp/evo-pgpass.XXXXXXXX)"
trap 'rm -f -- "$PGPASSFILE"' EXIT
cat "$EVO_PGPASS_SOURCE" > "$PGPASSFILE"
fi
mkdir -p "$out"
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
begin_ms="$(date +%s%3N)"
# The snapshot is held by this psql session while pg_dump reads it (\!).
export EVO_BACKUP_DUMP="$out/database.dump"
psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 > "$out/.cut" <<'SQL'
\pset fieldsep ' '
BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY;
SELECT pg_export_snapshot() AS snapshot \gset
SELECT 'database', current_setting('server_version');
SELECT 'schema', version, sha256 FROM evolution.schema_migrations ORDER BY version;
SELECT 'build', encode(build_identity, 'hex'),
CASE WHEN retired_at IS NULL THEN 'retained' ELSE 'retired' END
FROM evolution.retained_builds ORDER BY 2;
SELECT 'head', h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick,
h.commit_id, encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
FROM evolution.region_heads h
JOIN evolution.region_leases l USING (world_id, region_id)
JOIN evolution.region_commits c ON c.world_id = h.world_id AND c.region_id = h.region_id
AND c.commit_sequence = h.commit_sequence
ORDER BY h.world_id, h.region_id;
SELECT 'rows', (SELECT count(*) FROM evolution.region_commits),
(SELECT count(*) FROM evolution.region_decisions),
(SELECT count(*) FROM evolution.region_outbox),
(SELECT count(*) FROM evolution.region_origins);
\setenv EVO_BACKUP_SNAPSHOT :snapshot
\! pg_dump --no-password --format=custom --schema=evolution --no-owner --no-privileges --snapshot="$EVO_BACKUP_SNAPSHOT" --file="$EVO_BACKUP_DUMP" && echo 'dumped' || echo 'dump-failed'
COMMIT;
SQL
grep -qx 'dumped' "$out/.cut" || { echo "backup failed: pg_dump did not complete" >&2; exit 3; }
pg_restore --list "$out/database.dump" > /dev/null || { echo "backup failed: unreadable dump" >&2; exit 3; }
# Objects, verified by name while copying; private temporaries are skipped.
count=0 bytes=0
while IFS= read -r -d '' file; do
rel="${file#"$objects"/}"
name="${rel##*/}"
[[ "$rel" =~ ^w[0-9]+/r[0-9]+/[0-9a-f]{64}$ ]] || continue
mkdir -p "$out/objects/${rel%/*}"
cp -- "$file" "$out/objects/$rel"
sum="$(sha256sum -- "$out/objects/$rel")"
[ "${sum%% *}" = "$name" ] || { echo "backup failed: source object $rel is corrupt" >&2; exit 3; }
count=$((count + 1))
bytes=$((bytes + $(stat -c %s -- "$out/objects/$rel")))
done < <(find "$objects" -mindepth 3 -maxdepth 3 -type f ! -name '.tmp-*' -print0)
{
echo "format aevolution-backup-v1"
echo "created $started"
echo "label ${label:--}"
grep -E '^(database|schema|build|head|rows) ' "$out/.cut"
echo "objects $count $bytes"
} > "$out/MANIFEST"
rm -f "$out/.cut"
heads="$(grep -c '^head ' "$out/MANIFEST" || true)"
[ "$heads" -gt 0 ] || { echo "backup failed: no provisioned regions" >&2; exit 3; }
(cd "$out" && find . -type f ! -name SHA256SUMS -printf '%P\0' | sort -z | xargs -0 sha256sum -- > SHA256SUMS)
echo "backup complete regions=$heads objects=$count object_bytes=$bytes dump_bytes=$(stat -c %s "$out/database.dump") duration_ms=$(( $(date +%s%3N) - begin_ms )) out=$out"
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env bash
# M8.7 operator-run backup of a disposable environment to storage off the cluster
# (ADR-0074).
#
# tools/backup/cluster-backup.sh <dev|stage> --to DIR [--verify WORLD] [--context NAME]
#
# Runs one short-lived Pod in evolution-<env> (image: the postgres-tools digest
# pinned in deploy/kubernetes/stores/<env>; migrator login; the aevolution-objects
# volume mounted read-only on the store node) that runs backup-world.sh into an
# emptyDir. The bundle is then streamed out with `kubectl exec tar` to
# DIR/<env>-<UTC time>/, its checksums are verified here, and the Pod is deleted
# (also on failure). DIR is the destination: until the maintainer chooses the
# permanent one (docs/BACKLOG.md), it is any directory on a machine outside the
# hypervisor host. --verify WORLD then runs verify-backup.sh on this machine
# (restore on a private cluster, rebuilt and served as isolated world WORLD); it
# needs the PostgreSQL 18.6 binaries and evo-server/evo-world-clone.
# World0 is refused before anything touches the cluster. The Pod is an operator
# action, like fault-drill.sh; it is not part of any Argo application. A scheduled
# backup (CronJob in Git, synced by Argo) follows once the destination exists.
set -euo pipefail
usage() { echo "usage: $0 <dev|stage> --to DIR [--verify WORLD] [--context NAME]" >&2; exit 2; }
[ $# -ge 1 ] || usage
env="$1"; shift
case "$env" in
dev|stage) ;;
*) echo "refused: '$env' is not a disposable environment (only dev or stage; never world0)" >&2
exit 2 ;;
esac
to="" verify="" k=(kubectl --request-timeout=30s)
while [ $# -gt 0 ]; do
case "$1" in
--to) to="${2:-}"; shift 2 ;;
--verify) verify="${2:-}"; shift 2 ;;
--context) k+=(--context "${2:?}"); shift 2 ;;
*) usage ;;
esac
done
[ -n "$to" ] && [ -d "$to" ] || { echo "--to must be an existing directory" >&2; exit 2; }
root="$(cd "$(dirname "$0")/../.." && pwd)"
ns="evolution-$env"
image="$(awk '/name: aevolution-postgres-tools/ {f = 1} f && /newName:/ {n = $2} f && /digest:/ {print n "@" $2; exit}' \
"$root/deploy/kubernetes/stores/$env/kustomization.yaml")"
[[ "$image" =~ ^[a-z0-9./-]+@sha256:[0-9a-f]{64}$ ]] || { echo "no pinned postgres-tools image for $env" >&2; exit 2; }
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
pod="aevolution-backup-$(date -u +%Y%m%d%H%M%S)"
dest="$to/$env-$stamp"
say() { echo "$(date -u +%H:%M:%S) $*"; }
cleanup() { "${k[@]}" delete pod -n "$ns" "$pod" --wait=false > /dev/null 2>&1 || true; }
trap cleanup EXIT
say "starting $pod in $ns ($image)"
"${k[@]}" apply -n "$ns" -f - > /dev/null <<YAML
apiVersion: v1
kind: Pod
metadata:
name: $pod
labels:
app.kubernetes.io/part-of: aevolution
app.kubernetes.io/component: backup
spec:
serviceAccountName: aevolution-migrator
automountServiceAccountToken: false
restartPolicy: Never
activeDeadlineSeconds: 3600
nodeSelector:
aevolution.io/stores: "true"
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile: {type: RuntimeDefault}
containers:
- name: backup
image: $image
command: ["/bin/bash", "-c"]
args:
- |
set -eu
bash /opt/evo/tools/backup-world.sh --objects /objects --out /work/bundle --label "$env-$stamp"
touch /work/complete
exec sleep 1800
env:
- {name: PGSERVICE, value: evolution}
- {name: PGSERVICEFILE, value: /run/postgres/pg_service.conf}
- {name: EVO_PGPASS_SOURCE, value: /run/postgres/pgpass}
resources:
requests: {cpu: 100m, memory: 128Mi}
limits: {memory: 512Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: [ALL]}
volumeMounts:
- {name: objects, mountPath: /objects, readOnly: true}
- {name: connection, mountPath: /run/postgres, readOnly: true}
- {name: work, mountPath: /work}
- {name: scratch, mountPath: /tmp}
volumes:
- name: objects
persistentVolumeClaim: {claimName: aevolution-objects, readOnly: true}
- name: connection
secret:
secretName: evolution-postgres-migrator
defaultMode: 0440
items:
- {key: pg_service.conf, path: pg_service.conf}
- {key: pgpass, path: pgpass}
- {key: ca.crt, path: ca.crt}
- name: work
emptyDir: {sizeLimit: 12Gi}
- name: scratch
emptyDir: {medium: Memory, sizeLimit: 8Mi}
YAML
for _ in $(seq 1 1800); do
phase="$("${k[@]}" get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2> /dev/null || true)"
[ "$phase" = Failed ] && break
[ "$phase" = Running ] && "${k[@]}" exec -n "$ns" "$pod" -- test -f /work/complete 2> /dev/null && break
sleep 1
done
"${k[@]}" logs -n "$ns" "$pod" 2>&1 | sed 's/^/ /'
"${k[@]}" exec -n "$ns" "$pod" -- test -f /work/complete 2> /dev/null ||
{ say "FAIL backup did not complete (phase ${phase:-unknown})"; exit 3; }
mkdir -p "$dest"
"${k[@]}" exec -n "$ns" "$pod" -- tar -C /work/bundle -cf - . | tar -C "$dest" -xf -
(cd "$dest" && sha256sum --check --strict --quiet SHA256SUMS) || { say "FAIL fetched bundle checksum mismatch"; exit 3; }
say "bundle verified at $dest ($(du -sh "$dest" | cut -f1)); $(grep -c '^head ' "$dest/MANIFEST") region(s):"
grep '^head ' "$dest/MANIFEST" | sed 's/^/ /'
if [ -n "$verify" ]; then
bash "$root/tools/backup/verify-backup.sh" "$dest" --isolate "$verify"
else
say "next: tools/backup/verify-backup.sh $dest --isolate <world> (restore test on independent storage)"
fi
+121
View File
@@ -0,0 +1,121 @@
#!/usr/bin/env bash
# M8.7 verified restore of a backup bundle (ADR-0074) onto independent storage.
#
# tools/backup/restore-world.sh --bundle DIR --objects DIR
#
# Target connection: PG* environment naming an EMPTY database that the login owns
# (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE).
# The restore needs no source role, credential or key: the dump carries no owners
# or grants, so the target login owns everything restored. --objects must not
# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or
# PATH.
#
# Order, failing closed at each step:
# 1. every bundle file matches SHA256SUMS, no file is unlisted, and every object's
# name is its digest; nothing is touched before this passes;
# 2. the target has no `evolution` schema and runs the manifest's server version;
# 3. pg_restore in one transaction, then objects are copied;
# 4. the restored heads equal the MANIFEST heads exactly;
# 5. every region's full history is audited (evo-world-clone audit: each link's
# manifest, payload, trace replay and outcomes, from genesis to the head).
# On a failure after step 2 the restored schema and the copied objects are removed,
# so an unverified restore can never be mistaken for a world. The result is a
# verbatim, owner-only copy with the source's world identities: it has no runtime
# grants and is never served. To run it, clone it into an isolated world
# (tools/backup/verify-backup.sh --isolate); host-loss recovery in place is M8.8.
set -euo pipefail
usage() { echo "usage: $0 --bundle DIR --objects DIR" >&2; exit 2; }
bundle="" objects=""
while [ $# -gt 0 ]; do
case "$1" in
--bundle) bundle="${2:-}"; shift 2 ;;
--objects) objects="${2:-}"; shift 2 ;;
*) usage ;;
esac
done
[ -n "$bundle" ] && [ -n "$objects" ] || usage
bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")"
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
umask 077
begin_ms="$(date +%s%3N)"
refuse() { echo "RESTORE REFUSED: $*" >&2; exit 4; }
# 1. Bundle integrity, before anything else is touched.
[ -f "$bundle/MANIFEST" ] && [ -f "$bundle/SHA256SUMS" ] && [ -f "$bundle/database.dump" ] ||
refuse "incomplete bundle $bundle"
(cd "$bundle" && sha256sum --check --strict --quiet SHA256SUMS) || refuse "bundle checksum mismatch"
listed="$(sed -E 's/^[0-9a-f]{64} //' "$bundle/SHA256SUMS" | sort)"
present="$(cd "$bundle" && find . -type f ! -name SHA256SUMS -printf '%P\n' | sort)"
[ "$listed" = "$present" ] || refuse "bundle files differ from SHA256SUMS"
while IFS= read -r rel; do
case "$rel" in
objects/*)
[[ "$rel" =~ ^objects/w[0-9]+/r[0-9]+/([0-9a-f]{64})$ ]] || refuse "unexpected object path $rel"
grep -qE "^${BASH_REMATCH[1]} $rel\$" "$bundle/SHA256SUMS" || refuse "object $rel is not its digest"
;;
esac
done <<< "$present"
grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format"
expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)"
[ -n "$expected_heads" ] || refuse "manifest names no region"
# 2. Target: empty, same server version.
[ ! -e "$objects" ] || refuse "objects target already exists: $objects"
version="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version')")"
grep -qx "database $version" "$bundle/MANIFEST" || refuse "target server $version differs from the manifest"
existing="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")"
[ "$existing" = 0 ] || refuse "target database already has an evolution schema"
restored=0
cleanup() {
[ "$restored" = 1 ] && return
psql -X --no-password -q -v ON_ERROR_STOP=1 -c 'DROP SCHEMA IF EXISTS evolution CASCADE' > /dev/null 2>&1 || true
rm -rf -- "$objects"
echo "RESTORE FAILED: the restored schema and objects were removed" >&2
}
trap cleanup EXIT
# 3. Database in one transaction, then objects.
target="${PGDATABASE:-}"
[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE"
[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; }
pg_restore --no-password --exit-on-error --single-transaction --no-owner --no-privileges \
--dbname="$target" "$bundle/database.dump"
mkdir -m 700 -- "$objects"
(cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' |
xargs -0 -r -I{} mkdir -p -m 700 -- "$objects/{}")
(cd "$bundle/objects" && find . -type f -printf '%P\0' | xargs -0 -r -I{} cp -- {} "$objects/{}")
# 4. Heads exactly as recorded.
actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
SELECT h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick, h.commit_id,
encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
FROM evolution.region_heads h
JOIN evolution.region_leases l USING (world_id, region_id)
JOIN evolution.region_commits c ON c.world_id = h.world_id AND c.region_id = h.region_id
AND c.commit_sequence = h.commit_sequence" | sort)"
[ "$actual_heads" = "$expected_heads" ] || { echo "restored heads differ from the manifest" >&2; exit 3; }
# 5. Full history audit of every region.
pg=()
if [ -n "${PGSERVICEFILE:-}" ]; then
pg=(--pg-service "$PGSERVICEFILE")
[ -n "${PGPASSFILE:-}" ] && pg+=(--pg-passfile "$PGPASSFILE")
else
pg=(--pg-host "${PGHOST:?}" --pg-port "${PGPORT:-5432}" --pg-database "${PGDATABASE:?}"
--pg-user "${PGUSER:?}")
[ -n "${PGPASSFILE:-}" ] && pg+=(--pg-passfile "$PGPASSFILE")
fi
retained=()
while read -r _ build _; do retained+=(--retained "$build"); done < <(grep '^build ' "$bundle/MANIFEST")
commits=0
while read -r world region _ _ tick _; do
line="$("$clone_bin" audit --world "$world" --region "$region" --objects "$objects" \
"${pg[@]}" "${retained[@]}")" || { echo "audit failed for $world/$region" >&2; exit 3; }
[[ "$line" =~ ^audited\ commits=([0-9]+)\ tick=([0-9]+)\ ]] || { echo "audit output: $line" >&2; exit 3; }
[ "${BASH_REMATCH[2]}" = "$tick" ] || { echo "audit of $world/$region ended at ${BASH_REMATCH[2]}, not $tick" >&2; exit 3; }
commits=$((commits + BASH_REMATCH[1]))
echo "region $world/$region $line"
done <<< "$expected_heads"
restored=1
echo "RESTORE VERIFIED regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
+120
View File
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# M8.7 restore test on independent storage (ADR-0074).
#
# tools/backup/verify-backup.sh BUNDLE [--isolate WORLD [--source-world W]] [--keep]
#
# Starts a private, throwaway PostgreSQL cluster (initdb/pg_ctl/psql/pg_restore of
# the pinned server version on PATH) on local disk, with no connection to the
# source database or cluster, then:
# 1. restores BUNDLE with tools/backup/restore-world.sh as a fresh login that owns
# only the restore database (no source role, credential or key is used);
# 2. with --isolate WORLD, rebuilds a runnable isolated world: a separate database
# with its own owner, runtime group and login (migrated by migrate-postgres.sh),
# each region of the source world cloned into WORLD from the restored copy
# (evo-world-clone clone, source history audited first), and evo-server run on
# each cloned region under synthetic load until it acknowledges 20 ticks past
# the clone, then drained (exit 0).
# The cluster and its files are removed on exit unless --keep. Binaries come from
# EVO_WORLD_CLONE_BIN / EVO_SERVER_BIN or PATH. Port: EVO_VERIFY_PORT (55499).
set -euo pipefail
usage() { echo "usage: $0 BUNDLE [--isolate WORLD [--source-world W]] [--keep]" >&2; exit 2; }
[ $# -ge 1 ] || usage
bundle="$(realpath -- "$1")"; shift
isolate="" source_world="" keep=0
while [ $# -gt 0 ]; do
case "$1" in
--isolate) isolate="${2:-}"; shift 2 ;;
--source-world) source_world="${2:-}"; shift 2 ;;
--keep) keep=1; shift ;;
*) usage ;;
esac
done
[[ -z "$isolate" || "$isolate" =~ ^[1-9][0-9]*$ ]] || usage
root="$(cd "$(dirname "$0")/../.." && pwd)"
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
server_bin="${EVO_SERVER_BIN:-evo-server}"
port="${EVO_VERIFY_PORT:-55499}"
say() { echo "$(date -u +%H:%M:%S) $*"; }
cluster="$(mktemp -d /tmp/evo-verify.XXXXXXXX)"
chmod 700 "$cluster"
server_pid=""
finish() {
[ -n "$server_pid" ] && kill -9 "$server_pid" 2> /dev/null
pg_ctl -D "$cluster/data" -m immediate stop > /dev/null 2>&1 || true
if [ "$keep" = 1 ]; then say "kept $cluster"; else rm -rf -- "$cluster"; fi
}
trap finish EXIT
unset PGSERVICE PGSERVICEFILE PGPASSFILE PGHOSTADDR PGPASSWORD EVO_PGPASS_SOURCE
export PGHOST="$cluster" PGPORT="$port"
initdb -D "$cluster/data" -U evo_verify_admin --no-locale --encoding=UTF8 \
--auth-local=trust --auth-host=reject > "$cluster/initdb.log"
pg_ctl -D "$cluster/data" -l "$cluster/server.log" -w \
-o "-k $cluster -p $port -c listen_addresses=''" start > /dev/null
admin() { PGUSER=evo_verify_admin PGDATABASE=postgres psql -X -q -v ON_ERROR_STOP=1 "$@"; }
admin -c 'CREATE ROLE evo_restore_owner LOGIN' \
-c 'CREATE DATABASE evo_restored OWNER evo_restore_owner'
say "restoring $bundle on independent storage ($cluster)"
PGUSER=evo_restore_owner PGDATABASE=evo_restored \
bash "$root/tools/backup/restore-world.sh" --bundle "$bundle" --objects "$cluster/restored"
[ -n "$isolate" ] || { say "PASS restore verified"; exit 0; }
# The source world to rebuild: the only one in the bundle, or --source-world.
worlds="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2 | sort -u)"
if [ -z "$source_world" ]; then
[ "$(wc -l <<< "$worlds")" = 1 ] || { echo "bundle holds several worlds; pass --source-world" >&2; exit 2; }
source_world="$worlds"
fi
grep -qx "$source_world" <<< "$worlds" || { echo "world $source_world is not in the bundle" >&2; exit 2; }
[ "$source_world" != "$isolate" ] || { echo "the isolated world must differ from the source world" >&2; exit 2; }
admin -c 'CREATE ROLE evo_isolated_owner LOGIN' -c 'CREATE ROLE evo_isolated_runtime NOLOGIN' \
-c 'CREATE ROLE evo_isolated_login LOGIN IN ROLE evo_isolated_runtime' \
-c 'CREATE DATABASE evo_isolated OWNER evo_isolated_owner'
PGUSER=evo_isolated_owner PGDATABASE=evo_isolated EVO_PG_RUNTIME_ROLE=evo_isolated_runtime \
bash "$root/tools/migrate-postgres.sh" > /dev/null
retained=()
while read -r _ build _; do
retained+=(--retained "$build")
PGUSER=evo_isolated_owner PGDATABASE=evo_isolated psql -X -q -v ON_ERROR_STOP=1 -c \
"INSERT INTO evolution.retained_builds(build_identity) VALUES (decode('$build', 'hex'))"
done < <(grep '^build ' "$bundle/MANIFEST")
mkdir -m 700 "$cluster/isolated"
local_pg() { # local_pg <prefix> <database> <user>
echo "$1host $cluster $1port $port $1database $2 $1user $3"
}
while read -r _ world region frame _ tick _ _ build; do
[ "$world" = "$source_world" ] || continue
# shellcheck disable=SC2046 # word splitting of the fixed connection flags is intended.
cloned="$("$clone_bin" clone --source-world "$world" --source-region "$region" \
--target-world "$isolate" --target-region "$region" \
$(local_pg --source-pg- evo_restored evo_restore_owner) \
$(local_pg --target-pg- evo_isolated evo_isolated_owner) \
--source-objects "$cluster/restored" --target-objects "$cluster/isolated" \
--audit-max 1048576 "${retained[@]}")"
say "$cloned"
log="$cluster/server-$region.log"
"$server_bin" --profile lineage --world "$isolate" --region "$region" --frame "$frame" \
$(local_pg --pg- evo_isolated evo_isolated_login) --objects "$cluster/isolated" \
--build "$build" "${retained[@]}" --flush-ms 500 --drain-ms 20000 \
--synthetic-inputs development-v1 < /dev/null > "$log" 2>&1 &
server_pid=$!
target=$((tick + 20)) served=0
for _ in $(seq 1 600); do
if awk -v t="$target" '$1 == "ack" && $3 >= t {found = 1} END {exit !found}' "$log"; then served=1; break; fi
kill -0 "$server_pid" 2> /dev/null || break
sleep 0.1
done
running="$(grep -m1 '^phase running' "$log" || true)"
kill -TERM "$server_pid" 2> /dev/null || true
code=0; wait "$server_pid" || code=$?
server_pid=""
[ "$served" = 1 ] && [ "$code" = 0 ] ||
{ tail -n 20 "$log" >&2; echo "FAIL isolated $isolate/$region did not serve (exit $code)" >&2; exit 3; }
[[ "$running" == "phase running epoch=1 committed=$tick "* ]] ||
{ echo "FAIL isolated $isolate/$region resumed at '$running', not the backup head $tick" >&2; exit 3; }
say "isolated $isolate/$region served from tick $tick to $(awk '$1 == "ack" {t = $3} END {print t}' "$log") and drained"
done < <(grep '^head ' "$bundle/MANIFEST")
say "PASS restore verified and world $source_world rebuilt as isolated world $isolate"
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
# M8.7 backup and restore qualification (ADR-0074). Run by postgres-check.sh inside
# its throwaway cluster (PG* already set; EVO_SERVER_BIN, EVO_WORLD_CLONE_BIN and the
# runtime roles exported). A real evo-server commits under synthetic load while the
# backup is taken; the bundle is then restored on a separate cluster and rebuilt as a
# runnable isolated world, and damaged bundles are refused.
set -euo pipefail
root="$(cd "$(dirname "$0")/../.." && pwd)"
work="$(mktemp -d /tmp/evo-backup-check.XXXXXXXX)"
chmod 700 "$work"
server_pid=""
trap '[ -n "$server_pid" ] && kill -9 "$server_pid" 2> /dev/null; rm -rf -- "$work"' EXIT
fail() { echo "backup check FAILED: $*" >&2; exit 1; }
build=b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1
createdb evo_backup_src
export PGDATABASE=evo_backup_src
bash "$root/tools/migrate-postgres.sh" > /dev/null
mkdir -m 700 "$work/objects"
pg=(--pg-host "$PGHOST" --pg-port "$PGPORT" --pg-database evo_backup_src)
"$EVO_WORLD_CLONE_BIN" provision --world 300 --region 1 --opening development-v1 \
"${pg[@]}" --pg-user "$PGUSER" --objects "$work/objects" --build "$build" --retain-build true > /dev/null
"$EVO_SERVER_BIN" --profile lineage --world 300 --region 1 --frame 9 "${pg[@]}" \
--pg-user "$EVO_PG_RUNTIME_USER" --objects "$work/objects" --build "$build" \
--flush-ms 200 --lease-profile fast --synthetic-inputs development-v1 \
< /dev/null > "$work/server.log" 2>&1 &
server_pid=$!
acked_through() { awk -v t="$1" '$1 == "ack" && $3 >= t {f = 1} END {exit !f}' "$work/server.log"; }
max_ack() { awk '$1 == "ack" && $3 > m {m = $3} END {print m + 0}' "$work/server.log"; }
for _ in $(seq 1 300); do acked_through 60 && break; sleep 0.1; done
acked_through 60 || fail "the source server acknowledged nothing"
# Online backup while the server keeps committing.
acked_before="$(max_ack)"
bash "$root/tools/backup/backup-world.sh" --objects "$work/objects" --out "$work/bundle" --label check
read -r _ _ _ _ _ backup_tick _ < <(grep '^head 300 1 ' "$work/bundle/MANIFEST")
[ "$backup_tick" -ge "$acked_before" ] || fail "backup head $backup_tick misses acknowledged tick $acked_before"
for _ in $(seq 1 300); do acked_through $((backup_tick + 40)) && break; sleep 0.1; done
kill -TERM "$server_pid"; code=0; wait "$server_pid" || code=$?; server_pid=""
[ "$code" = 0 ] || fail "the source server exited $code"
source_tick="$(psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')"
[ "$source_tick" -gt "$backup_tick" ] || fail "the source did not move past the backup ($source_tick)"
grep -q '^build b1b1' "$work/bundle/MANIFEST" && grep -q '^schema 9 ' "$work/bundle/MANIFEST" ||
fail "manifest lacks builds or schema versions"
if grep -q -i -E 'password|pgpass' "$work/bundle/MANIFEST"; then fail "manifest names credentials"; fi
# Full restore on an independent cluster, rebuilt and served as isolated world 301.
EVO_VERIFY_PORT=$((PGPORT + 1)) bash "$root/tools/backup/verify-backup.sh" "$work/bundle" \
--isolate 301 > "$work/verify.log" 2>&1 || { cat "$work/verify.log" >&2; fail "verify-backup"; }
grep -q "^RESTORE VERIFIED regions=1 " "$work/verify.log" &&
grep -q "PASS restore verified and world 300 rebuilt as isolated world 301" "$work/verify.log" ||
{ cat "$work/verify.log" >&2; fail "verify-backup evidence"; }
# Damaged bundles, restored straight into scratch databases of this cluster.
case_n=0
expect_refusal() { # expect_refusal <exit code> <bundle> <what>
case_n=$((case_n + 1))
createdb "evo_restore_case_$case_n"
local code=0
PGDATABASE="evo_restore_case_$case_n" bash "$root/tools/backup/restore-world.sh" \
--bundle "$2" --objects "$work/restored-$case_n" > "$work/case-$case_n.log" 2>&1 || code=$?
[ "$code" = "$1" ] || { cat "$work/case-$case_n.log" >&2; fail "$3: exit $code, expected $1"; }
[ ! -e "$work/restored-$case_n" ] || fail "$3: restored objects were left behind"
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" = 0 ] ||
fail "$3: a restored schema was left behind"
}
damaged() { rm -rf "$work/damaged"; cp -a "$work/bundle" "$work/damaged"; }
resum() { (cd "$work/damaged" && find . -type f ! -name SHA256SUMS -printf '%P\0' | sort -z | xargs -0 sha256sum -- > SHA256SUMS); }
genesis="$(psql -XAtc "SELECT encode(payload_sha256, 'hex') FROM evolution.region_commits WHERE world_id = 300 AND commit_sequence = 0")"
genesis_object="$work/damaged/objects/w300/r1/$genesis"
damaged; printf 'corrupt' | dd of="$genesis_object" bs=1 conv=notrunc status=none
expect_refusal 4 "$work/damaged" "a corrupt object"
damaged; echo extra > "$work/damaged/objects/w300/r1/notes"
expect_refusal 4 "$work/damaged" "an unlisted file"
damaged; printf 'x' >> "$work/damaged/database.dump"
expect_refusal 4 "$work/damaged" "a damaged dump"
# Consistently re-summed damage passes the checksums and must fail verification.
damaged; rm "$genesis_object"; resum
expect_refusal 3 "$work/damaged" "a missing payload object"
grep -q 'audit failed for 300/1' "$work/case-$case_n.log" || fail "missing object: no audit failure"
damaged; sed -i -E "s/^(head 300 1 [0-9]+ [0-9]+) $backup_tick /\1 $((backup_tick - 1)) /" "$work/damaged/MANIFEST"; resum
expect_refusal 3 "$work/damaged" "a manifest head that disagrees with the database"
# A target that already holds a world is refused before anything is written.
createdb evo_restore_twice
PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$work/twice-a" > /dev/null
code=0
PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
--objects "$work/twice-b" > "$work/twice.log" 2>&1 || code=$?
[ "$code" = 4 ] && grep -q 'already has an evolution schema' "$work/twice.log" || fail "a second restore into one database"
[ "$(PGDATABASE=evo_restore_twice psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
fail "the refused second restore changed the first"
echo "backup and restore qualification passed (backup tick $backup_tick, acknowledged $acked_before, source $source_tick)"
+2 -2
View File
@@ -18,10 +18,10 @@ case "$suite" in
grep -Eq '^(CMakeLists.txt|CMakePresets.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|experiments/|tools/ci/linux-check.sh|tools/ci/tsan-profile.mjs|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
;;
postgres|postgres-tools)
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|migrations/|tools/prepare-postgres.sh|tools/ci/postgres-check.sh|tools/migrate-postgres.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|migrations/|tools/prepare-postgres.sh|tools/ci/postgres-check.sh|tools/ci/backup-check.sh|tools/backup/|tools/migrate-postgres.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
;;
deploy)
grep -Eq '^(deploy/|tools/ci/deploy-policy-check.py|\.gitea/workflows/)' <<<"$files"
grep -Eq '^(deploy/|tools/ci/deploy-policy-check.py|tools/ops/|tools/backup/|\.gitea/workflows/)' <<<"$files"
;;
server-image)
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|migrations/|tools/prepare-postgres.sh|tools/ci/server-image-smoke.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
+3 -2
View File
@@ -302,10 +302,11 @@ for path, doc in objects:
'RoleBinding', 'ClusterRole', 'ClusterRoleBinding'},
f'{path}: {doc["kind"]} must not live in the Argo workload path')
# The cleanup tool refuses World0 before kubectl runs (no kubectl on PATH here).
# The cleanup, drill and backup tools refuse World0 before kubectl runs (no kubectl on PATH here).
with tempfile.TemporaryDirectory() as empty:
env = dict(os.environ, PATH=empty + os.pathsep + os.path.dirname(sys.executable))
for tool in ('tools/ops/cleanup-environment.sh', 'tools/ops/region-drill.sh', 'tools/ops/fault-drill.sh'):
for tool in ('tools/ops/cleanup-environment.sh', 'tools/ops/region-drill.sh', 'tools/ops/fault-drill.sh',
'tools/backup/cluster-backup.sh'):
for bad in ('world0', 'evolution-world0', 'World0', 'prod', ''):
result = subprocess.run([shutil.which('bash') or '/bin/bash', os.path.join(ROOT, tool), bad],
capture_output=True, text=True, env=env)
+2
View File
@@ -30,6 +30,8 @@ export EVO_SERVER_BIN EVO_WORLD_CLONE_BIN
EVO_SERVER_BIN="$(dirname "${1:?path to evo-postgres-tests}")/evo-server"
EVO_WORLD_CLONE_BIN="$(dirname "$1")/evo-world-clone"
"$1"
# M8.7: online backup under load, restore on a separate cluster, isolated rebuild.
bash "$root/tools/ci/backup-check.sh"
v1_sha="$(sha256sum "$root/migrations/0001_region_bootstrap.sql")"
v1_sha="${v1_sha%% *}"
# Migration failure must not publish a partial schema.