feat(backup): M8.7 world backup bundles and verified restore (ADR-0074)
Online, snapshot-consistent bundles (pg_dump at an exported snapshot, immutable objects verified by digest, MANIFEST of versions/builds/heads, SHA256SUMS); a fail-closed restore that audits every region's full history on independent storage; an isolated rebuild through clones that serves under load; an operator-run in-cluster backup streamed off the cluster. backup-check.sh runs in every PostgreSQL build, with corruption, missing-object, tampered-manifest and non-empty-target refusals. Destination, encryption, schedule and WAL archiving remain on the backlog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -86,6 +86,8 @@ jobs:
|
||||
bash tools/ci/docker-build.sh postgres-tools "$image" aevolution-postgres-cache
|
||||
docker run --rm --network none --read-only --cap-drop ALL \
|
||||
--entrypoint /opt/postgres/bin/psql "$image" --version
|
||||
docker run --rm --network none --read-only --cap-drop ALL \
|
||||
--entrypoint /opt/postgres/bin/pg_dump "$image" --version
|
||||
- name: Publish configured migration tools
|
||||
if: ${{ gitea.ref == 'refs/heads/main' && gitea.event_name != 'pull_request' && vars.HARBOR_POSTGRES_REPOSITORY != '' }}
|
||||
env:
|
||||
|
||||
@@ -59,6 +59,7 @@ WORKDIR /source
|
||||
COPY deploy/ deploy/
|
||||
COPY tools/ci/deploy-policy-check.py tools/ci/
|
||||
COPY tools/ops/ tools/ops/
|
||||
COPY tools/backup/ tools/backup/
|
||||
RUN python3 tools/ci/deploy-policy-check.py
|
||||
# M8.5: monitoring configuration and alert tests with the pinned images' own tools.
|
||||
COPY --from=quay.io/prometheus/prometheus@sha256:6976aa8a60fec930796ce5772b8d12da7a318a5daa8d40d69c5c7819a05eeed7 /bin/promtool /usr/local/bin/promtool
|
||||
@@ -109,9 +110,13 @@ RUN apt-get update && apt-get install -y --no-install-recommends libssl3t64 zlib
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& groupadd --gid 10001 evo && useradd --uid 10001 --gid 10001 --no-create-home evo
|
||||
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/psql /opt/postgres/bin/psql
|
||||
# M8.7 backups (ADR-0074): the same pinned PostgreSQL build's dump tools.
|
||||
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/pg_dump /opt/postgres/bin/pg_dump
|
||||
COPY --from=postgres-toolchain /source/.cache/postgres/install/bin/pg_restore /opt/postgres/bin/pg_restore
|
||||
COPY --from=postgres-toolchain /source/.cache/postgres/install/lib/libpq.so* /opt/postgres/lib/
|
||||
COPY --from=postgres-toolchain /source/.cache/postgres/postgresql-18.6/COPYRIGHT /usr/share/doc/evo-postgres/COPYRIGHT
|
||||
COPY tools/migrate-postgres.sh /opt/evo/tools/migrate-postgres.sh
|
||||
COPY tools/backup/backup-world.sh /opt/evo/tools/backup-world.sh
|
||||
COPY migrations/ /opt/evo/migrations/
|
||||
ENV PATH="/opt/postgres/bin:${PATH}" LD_LIBRARY_PATH=/opt/postgres/lib
|
||||
USER 10001:10001
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# Egress from migrate, store-check and region pods to exactly the AEvolution
|
||||
# PostgreSQL VM (pg-forge-01) and nothing else (ADR-0070).
|
||||
# Egress from migrate, store-check, region and operator backup pods to exactly the
|
||||
# AEvolution PostgreSQL VM (pg-forge-01) and nothing else (ADR-0070, ADR-0074).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
@@ -13,7 +13,7 @@ spec:
|
||||
matchExpressions:
|
||||
- key: app.kubernetes.io/component
|
||||
operator: In
|
||||
values: [migrate, store-check, region]
|
||||
values: [migrate, store-check, region, backup]
|
||||
policyTypes: [Egress]
|
||||
egress:
|
||||
- to:
|
||||
|
||||
+3
-1
@@ -7,7 +7,9 @@ names it. Remove an item only when it is done, and record where it was done.
|
||||
| Item | Raised | Why deferred | Needed before |
|
||||
|---|---|---|---|
|
||||
| **World0 break-glass membership.** Decide who may act as `aevolution:world0-break-glass`, and how that group is granted: for example a short-lived client certificate with that organization, or an OIDC group. Decide also how its use is approved and recorded. The admission policy already enforces the group (ADR-0069). | M8.2, 2026-09-29 | Maintainer: figure out later | Any intentional deletion of World0 data or identity; M8.8 World0 launch |
|
||||
| **Backup destination for dev/stage stores.** There are no backups yet (maintainer, 2026-09-29). The plan is to build backup tooling for the AEvolution PostgreSQL VM and each `aevolution-objects` volume up to a pluggable destination, then add the destination. It must be off the host for anything beyond disposable environments. | M8.3, 2026-09-29 | The maintainer will supply a destination later | M8.7 backup and restore; any World0 data |
|
||||
| **Backup destination for dev/stage stores.** The tooling now exists up to the destination (ADR-0074): `tools/backup/cluster-backup.sh <env> --to DIR` writes a verified bundle to any directory, and `verify-backup.sh` restore-tests it. Still to choose: the permanent off-host destination, and with it (1) bundle encryption and key custody, (2) a scheduled CronJob synced by Argo, (3) retention (proposal: 30 daily, 12 weekly, 12 monthly), (4) incremental object copies. | M8.3, 2026-09-29; M8.7, 2026-09-30 | The maintainer will supply a destination later | Scheduled backups; any World0 data; M8.8 |
|
||||
| **WAL archiving on pg-forge-01.** Bundles are logical backups: the recovery point is the last bundle. Continuous archiving (`archive_command` or `pg_receivewal`) plus periodic base backups would allow point-in-time recovery. It needs the backup destination. | M8.7, 2026-09-30 | Needs the destination and VM configuration | M8.8 host-loss RPO decision |
|
||||
| **Harbor retention of build images.** A bundle's MANIFEST names every build (region image digest) that wrote history; recovery and replay need those images. Confirm Harbor's retention/tag-immutability policy keeps them (or pin them in a protected project). | M8.7, 2026-09-30 | Not yet checked | M8.8; any upgrade barrier that retires a build |
|
||||
| **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 |
|
||||
| **Store Job values.** Supply the published registry digests for `aevolution-postgres-tools` and `aevolution-server` (from each CI Publish step's `Digest:` line), and the NOLOGIN runtime group role for dev (`aevolution`) and for stage (`aevolution_stage`). Then include `stores/<env>` in each environment's path. | M8.3, 2026-09-29 | Values pending | M8.3 in-cluster evidence |
|
||||
| **Alert destination.** Alertmanager routes every alert to a null receiver (ADR-0072). Choose where alerts go (Discord, email, a webhook), and optionally a heartbeat receiver for the always-firing Watchdog. Receiver credentials go in a Secret. | M8.5, 2026-09-30 | The maintainer will decide later | Any claim that alerts reach a person; M8.6 alert evidence; World0 |
|
||||
|
||||
@@ -650,3 +650,13 @@ M8.6 implemented and locally qualified (2026-09-30), under ADR-0073:
|
||||
and an operator-driven database outage;
|
||||
- RTO and zero-lost-acknowledgement checks.
|
||||
The in-cluster drill results wait on the maintainer.
|
||||
|
||||
M8.7 implemented and locally qualified (2026-09-30), under ADR-0074:
|
||||
- online, snapshot-consistent world bundles (database dump, immutable objects,
|
||||
MANIFEST of versions, builds and heads, checksums);
|
||||
- a fail-closed restore that audits every region's full history on independent
|
||||
storage, and an isolated rebuild through clones that serves under load;
|
||||
- corruption, missing-object, tampered-manifest and non-empty-target refusals;
|
||||
- an operator-run in-cluster backup to a directory off the cluster.
|
||||
Pending the maintainer's destination: encryption, schedule, retention and WAL
|
||||
archiving (docs/BACKLOG.md). In-cluster bundle evidence waits on a run.
|
||||
|
||||
@@ -1105,3 +1105,23 @@ It restores the Git configuration on exit.
|
||||
|
||||
Earlier (M8.4 follow-up): `REGION_RETAINED` keeps earlier builds recoverable
|
||||
across image upgrades.
|
||||
|
||||
## M8.7 backup and restore (2026-09-30)
|
||||
|
||||
ADR-0074 adds world backup bundles and a verified restore:
|
||||
- `tools/backup/backup-world.sh`: an online backup cut by one exported database
|
||||
snapshot (MANIFEST heads and `pg_dump` read the same snapshot), then the
|
||||
immutable objects, each checked against its digest; `SHA256SUMS` over all of it.
|
||||
- `tools/backup/restore-world.sh`: checksums first, then an owner-only restore into
|
||||
an empty database, exact head comparison and a full history audit of every region;
|
||||
any failure removes what it restored.
|
||||
- `tools/backup/verify-backup.sh`: the restore on a private throwaway cluster with
|
||||
fresh roles, then (`--isolate WORLD`) a clone into an isolated world that serves
|
||||
under synthetic load and drains.
|
||||
- `tools/backup/cluster-backup.sh <dev|stage> --to DIR`: an operator Pod (read-only
|
||||
objects volume, migrator login) whose bundle is streamed off the cluster.
|
||||
- The postgres-tools image now carries `pg_dump` and `pg_restore` from the pinned
|
||||
18.6 build; the database egress policy admits the `backup` component.
|
||||
|
||||
The destination, encryption, schedule, retention and WAL archiving wait on the
|
||||
maintainer (docs/BACKLOG.md).
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
# ADR-0074: World backup bundles and verified restore
|
||||
|
||||
## Status
|
||||
Accepted for M8.7, 2026-09-30. It builds on:
|
||||
- ADR-0061 (immutable, content-addressed objects; upload verified before commit);
|
||||
- ADR-0067 (clones and history audit);
|
||||
- ADR-0070 (dev/stage stores);
|
||||
- ADR-0073 (synthetic load).
|
||||
|
||||
The permanent backup destination, encryption, scheduling and WAL archiving wait
|
||||
on the maintainer's choice of destination (docs/BACKLOG.md). Until then, this ADR
|
||||
covers everything up to that destination.
|
||||
|
||||
## Context
|
||||
M8.7 requires:
|
||||
- a coordinated manifest covering the database, the objects, the versions, the
|
||||
binaries and any in-flight state;
|
||||
- a restore onto independent storage;
|
||||
- retention and GC that respect clone and backup pins.
|
||||
|
||||
The evidence asked for is:
|
||||
- rebuilding an isolated world from a backup;
|
||||
- corruption and missing-object failures;
|
||||
- credential and key recovery.
|
||||
|
||||
The named risk is a backup that exists but cannot reconstruct causal state.
|
||||
|
||||
docs/infrastructure/OPERATIONS.md proposes a coordinated admission barrier. Two
|
||||
facts make a barrier unnecessary for a single database:
|
||||
- a region's durable state is one PostgreSQL database plus immutable objects;
|
||||
- every object a committed row references was verified durable before that row
|
||||
committed.
|
||||
|
||||
## Decision
|
||||
**Bundle.** `tools/backup/backup-world.sh --objects DIR --out DIR` writes:
|
||||
|
||||
| File | Content |
|
||||
|---|---|
|
||||
| `MANIFEST` | Format, time and label. The server version. Every schema migration and its checksum. Every retained build (image digest) and whether it is retired. Every region head (world, region, frame, sequence, tick, commit, digest, build). Row counts, and the object count and bytes |
|
||||
| `database.dump` | `pg_dump` custom format of schema `evolution`, with no owners and no grants |
|
||||
| `objects/w<world>/r<region>/<sha256>` | Every object in the store, each checked against its name while it is copied |
|
||||
| `SHA256SUMS` | Checksums of every other file |
|
||||
|
||||
**The cut.** One psql session opens a `REPEATABLE READ, READ ONLY` transaction and
|
||||
exports its snapshot. The session reads the heads for the MANIFEST, then runs
|
||||
`pg_dump --snapshot` on that same snapshot while the session holds it. Objects are
|
||||
copied afterwards.
|
||||
- Objects are never overwritten or deleted, and no GC exists. So every object the
|
||||
snapshot references is present and final when it is copied.
|
||||
- Objects written after the snapshot are copied as harmless orphans.
|
||||
- Nothing is written to the source, and admission never stops. The backup runs
|
||||
online while the host commits.
|
||||
|
||||
**In-flight state.**
|
||||
- Every acknowledgement made before the snapshot is in the bundle, because an
|
||||
acknowledgement follows its commit.
|
||||
- A commit whose transaction completes after the snapshot is not in the bundle.
|
||||
It was not acknowledged at the cut.
|
||||
- Unacknowledged inputs are never durable, by design (ADR-0064).
|
||||
- Lease rows are copied, but a restored copy is never served under them (see
|
||||
below).
|
||||
- There are no cross-region transfers yet. When they exist, they live in the same
|
||||
database and are covered by the same snapshot.
|
||||
|
||||
**Restore.** `tools/backup/restore-world.sh --bundle B --objects DIR` targets an
|
||||
empty database owned by the restoring login, and fails closed in this order:
|
||||
1. Every file matches `SHA256SUMS`, no file is unlisted, and every object's name
|
||||
is its digest. Nothing is touched before this passes.
|
||||
2. The target has no `evolution` schema and runs the manifest's server version.
|
||||
3. `pg_restore` runs in one transaction, then the objects are copied.
|
||||
4. The restored heads equal the MANIFEST heads exactly.
|
||||
5. Every region's full history is audited with `evo-world-clone audit`. The audit
|
||||
checks every link's manifest, payload, trace replay and outcomes, from genesis
|
||||
to the head.
|
||||
|
||||
A failure after step 2 removes the restored schema and objects, so an unverified
|
||||
restore cannot be mistaken for a world. The result is a verbatim, owner-only copy
|
||||
that keeps the source's world identities. It has no runtime grants and is never
|
||||
served.
|
||||
|
||||
**Isolated rebuild.** `tools/backup/verify-backup.sh BUNDLE --isolate WORLD`:
|
||||
1. Starts a private, throwaway PostgreSQL cluster on local disk.
|
||||
2. Restores the bundle as a fresh login that owns only the restore database.
|
||||
3. Creates a separate database with its own owner, runtime group and login, and
|
||||
migrates it with `migrate-postgres.sh`.
|
||||
4. Clones every region of the source world into `WORLD` with
|
||||
`evo-world-clone clone`, which audits the source history first. Provenance is
|
||||
kept in `region_origins`.
|
||||
5. Runs `evo-server` on each clone under synthetic load until it acknowledges 20
|
||||
ticks past the backup head, then drains it.
|
||||
|
||||
This is OPERATIONS.md steps 3–5 for a clone. Restoring in place under the source's
|
||||
own identity, with the old writers fenced, is host-loss recovery and belongs to
|
||||
M8.8.
|
||||
|
||||
**Credentials and keys.**
|
||||
- A bundle carries no roles, grants, passwords or connection details.
|
||||
- Restoring needs only a login that owns an empty database, and no source
|
||||
credential. `backup-check.sh` and `verify-backup.sh` prove this with fresh,
|
||||
unrelated roles.
|
||||
- Bundles are not encrypted yet, so there is no key to lose. Encryption and its key
|
||||
custody are chosen together with the destination. They are required before any
|
||||
World0 data leaves the host.
|
||||
|
||||
**Retention and GC.**
|
||||
- Each bundle is a full, self-contained copy, and no bundle depends on another.
|
||||
Deleting one never breaks another.
|
||||
- Clones copy their payloads into the target namespace (ADR-0067), so a clone does
|
||||
not depend on its source either.
|
||||
- The object store has no delete, and history never expires.
|
||||
- Any future GC must prove reachability from every live head, clone origin and
|
||||
retained build. Bundles hold their own copies, so GC cannot damage them.
|
||||
- Retention of bundles at the destination (the OPERATIONS proposal: 30 daily, 12
|
||||
weekly, 12 monthly) is applied when the destination exists.
|
||||
|
||||
**Binaries.** The MANIFEST names every build identity that wrote history. These
|
||||
are the region image digests that recovery and replay need. Harbor must keep those
|
||||
images; its retention policy is not yet verified (backlog).
|
||||
|
||||
**In the cluster.** `tools/backup/cluster-backup.sh <dev|stage> --to DIR [--verify WORLD]`
|
||||
is an operator action, like `fault-drill.sh`, and refuses World0 before kubectl
|
||||
runs. It works as follows:
|
||||
- It runs one short-lived Pod (component `backup`) with:
|
||||
- the environment's pinned postgres-tools image, which now includes `pg_dump`,
|
||||
`pg_restore` and `backup-world.sh`;
|
||||
- the migrator login;
|
||||
- the `aevolution-objects` volume mounted read-only on the store node;
|
||||
- a 12 GiB emptyDir for the bundle.
|
||||
- The database egress policy admits the `backup` component.
|
||||
- The bundle is streamed out with `kubectl exec tar` and its checksums are checked
|
||||
locally. Then the Pod is deleted, including on failure.
|
||||
|
||||
A scheduled CronJob, synced by Argo, follows once the destination exists.
|
||||
|
||||
**Qualification.** `tools/ci/backup-check.sh` runs inside `postgres-check.sh` for
|
||||
every PostgreSQL build:
|
||||
- It backs up a real `evo-server` while the server commits under synthetic load.
|
||||
The backup head covers every acknowledgement made before the backup, and the
|
||||
source then moves past it.
|
||||
- It runs `verify-backup.sh --isolate` on a second cluster.
|
||||
- It restores damaged bundles, each of which must be refused, leaving nothing
|
||||
behind: a corrupt object, an unlisted file, a damaged dump, a missing payload
|
||||
that has consistent checksums, and a MANIFEST head that disagrees with the
|
||||
database.
|
||||
- It restores a second time into a database that already holds a world, which
|
||||
must be refused.
|
||||
|
||||
## Consequences
|
||||
- **RPO.** The recovery point is the last verified bundle. Without WAL archiving,
|
||||
this is a logical backup and cannot recover to an arbitrary point in time.
|
||||
Continuous archiving on pg-forge-01 needs the destination (backlog). M8.8 decides
|
||||
the host-loss RPO for World0.
|
||||
- **Full copies.** Every bundle copies the whole object store. That is fine at
|
||||
dev/stage sizes (the objects volume is 10 GiB). Incremental object copies, which
|
||||
are safe because objects are content-addressed, come with the destination.
|
||||
- **Off host.** A bundle is off the host only if `--to` is on another machine. The
|
||||
maintainer's workstation qualifies only if it is not the hypervisor host.
|
||||
- **Restore test.** A scheduled job or a successful upload is not a restore test.
|
||||
Only `verify-backup.sh` output counts as restore evidence.
|
||||
|
||||
## Alternatives rejected
|
||||
- **Physical base backups (`pg_basebackup`) as the only copy.** They restore the
|
||||
whole server, with its roles, and cannot be audited per region or restored as an
|
||||
isolated world without a second step. They remain the likely basis for PITR once
|
||||
WAL archiving exists.
|
||||
- **An admission barrier before the cut.** It is unnecessary given
|
||||
upload-before-commit and immutable objects, and it would pause the world.
|
||||
- **Restoring directly as a runnable world with fresh grants.** It would serve the
|
||||
source's world identity from a second place, which could become competing
|
||||
authority. Isolation goes through the clone path, which re-identifies the world.
|
||||
- **A new object-storage dependency (MinIO, S3 SDK).** The destination is not
|
||||
chosen; a directory and `tar` need no new dependency.
|
||||
@@ -84,3 +84,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
|
||||
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
|
||||
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
|
||||
- [ADR-0074: World backup bundles and verified restore](0074-world-backup-and-restore.md) - accepted for M8.7 snapshot-consistent online bundles (dump + immutable objects + manifest), fail-closed audited restore on independent storage, isolated rebuild through clones; destination, encryption, schedule and WAL archiving pending.
|
||||
|
||||
@@ -2916,3 +2916,39 @@ Fixes (this segment):
|
||||
MSVC CTest 59/59 (the server is not part of the Windows build).
|
||||
`fault-drill.sh` passed `bash -n`.
|
||||
- Pending: a new image, then re-running the full, empty and db-outage scenarios.
|
||||
|
||||
## M8.7 backup and restore - 2026-09-30
|
||||
|
||||
- `tools/ci/backup-check.sh` (inside `postgres-check.sh`; real PostgreSQL 18.6, a
|
||||
real `evo-server` under synthetic load) passed in the Release, analysis and
|
||||
ASan/UBSan PostgreSQL builds:
|
||||
- An online backup taken while the server committed recorded head tick 60, at or
|
||||
past every acknowledgement before it (60). The source then moved on to tick 101.
|
||||
- `verify-backup.sh --isolate 301` on a second, throwaway cluster, as fresh
|
||||
roles: the restore audited 13 commits to tick 60 and matched the MANIFEST. The
|
||||
clone into world 301 resumed at `epoch=1 committed=60`, acknowledged to tick
|
||||
82, and drained with exit 0.
|
||||
- Refused bundles, each leaving no schema and no objects behind:
|
||||
- corrupt object, unlisted file, damaged dump: exit 4 at the checksum step;
|
||||
- missing payload object with consistent checksums: exit 3,
|
||||
`audit failed missing_payload sequence=0`;
|
||||
- MANIFEST head changed: exit 3, heads differ.
|
||||
- A second restore into a database that already holds a world was refused
|
||||
(exit 4), and the first restore was left unchanged.
|
||||
- The MANIFEST names no credentials.
|
||||
- Mutations, each caught by `backup-check.sh`:
|
||||
- restore without the history audit: the missing object was accepted (exit 0);
|
||||
- a failed restore that keeps its files: restored objects were left behind;
|
||||
- `pg_dump` outside the exported snapshot (2 s later): the restored heads
|
||||
differed from the MANIFEST, and verification failed.
|
||||
- The PostgreSQL migration, rollback and permission suites passed in all three
|
||||
builds. `deploy-policy-check.py`: 67 objects, 0 failures.
|
||||
`cluster-backup.sh` refuses World0 before kubectl (policy rule).
|
||||
- `bash -n` passed on every new script. shellcheck is not installed in WSL, so it
|
||||
did not run.
|
||||
- Not executed:
|
||||
- the postgres-tools image build with `pg_dump` (CI builds it on push);
|
||||
- `cluster-backup.sh` against dev or stage;
|
||||
- any restore of in-cluster data.
|
||||
- The permanent destination, encryption, schedule and WAL archiving are not
|
||||
implemented; they are pending the maintainer's choice of destination.
|
||||
|
||||
@@ -192,7 +192,7 @@ To delete World0 data or identity on purpose:
|
||||
1. Record the operation and its approval.
|
||||
2. Use a credential whose groups include `aevolution:world0-break-glass`, for
|
||||
example a short-lived client certificate with that organization.
|
||||
3. Take and verify a backup first (M8.7).
|
||||
3. Take and verify a backup first (M8.7; see "Backups" below).
|
||||
|
||||
Removing the admission policy itself is the break-glass of last resort. It needs
|
||||
cluster-admin and must be recorded the same way. Never test by deleting World0: the
|
||||
@@ -206,3 +206,29 @@ verify script uses server-side dry runs.
|
||||
- **World0 volumes.** `local-path` provisions volumes with Delete. Patch each World0
|
||||
PV to `Retain` when it is bound, and the policy then keeps it there. Volumes stay
|
||||
node-local until M8.3.
|
||||
|
||||
## Backups (M8.7, ADR-0074)
|
||||
|
||||
Dev and stage only; the tools refuse World0. From a machine with kubectl access
|
||||
that is not the hypervisor host:
|
||||
|
||||
```bash
|
||||
# 1. Sync the environment's application once so the egress policy admits the
|
||||
# backup component and the store overlay pins a postgres-tools image that
|
||||
# contains pg_dump (any image built after ADR-0074).
|
||||
# 2. Back up to a local directory (the destination until one is chosen):
|
||||
tools/backup/cluster-backup.sh dev --to /path/to/backups
|
||||
# 3. Restore-test it on independent storage and rebuild it as isolated world 900
|
||||
# (WSL: the pinned PostgreSQL build and the Linux PostgreSQL build of the tools).
|
||||
PATH="$PWD/.cache/postgres/install/bin:$PATH" \
|
||||
LD_LIBRARY_PATH="$PWD/.cache/postgres/install/lib" \
|
||||
EVO_WORLD_CLONE_BIN="$PWD/build/postgres-linux/evo-world-clone" \
|
||||
EVO_SERVER_BIN="$PWD/build/postgres-linux/evo-server" \
|
||||
tools/backup/verify-backup.sh /path/to/backups/dev-<time> --isolate 900
|
||||
```
|
||||
|
||||
Only `verify-backup.sh` output that ends in `PASS restore verified and world ...
|
||||
rebuilt as isolated world ...` counts as restore evidence. The restore uses fresh
|
||||
roles on a throwaway cluster; nothing reaches the source database. A restored copy
|
||||
keeps the source world identity and is never served; restoring in place after host
|
||||
loss is M8.8.
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
# Operations and disaster recovery
|
||||
|
||||
Status: proposal requiring measured acceptance before World0. No deployment,
|
||||
backup or distributed restore has been exercised in this foundation.
|
||||
Status: proposal requiring measured acceptance before World0. M8.7 (ADR-0074)
|
||||
implements snapshot-consistent world backup bundles and a verified restore on
|
||||
independent storage with an isolated rebuild; the destination, encryption,
|
||||
schedule and WAL archiving are pending, and no host-loss restore in place has been
|
||||
exercised (M8.8).
|
||||
|
||||
| Failure | Proposed objective / gate |
|
||||
|---|---|
|
||||
@@ -36,8 +39,10 @@ pre-upgrade checkpoints, subject to capacity evidence. Genome/ancestry/lifecycle
|
||||
history does not expire with backup retention. GC respects all clone/backup pins
|
||||
and complete replay chains.
|
||||
|
||||
1. First implementation uses a coordinated admission/transfer barrier, avoiding
|
||||
an unfinished distributed-snapshot algorithm.
|
||||
1. First implementation (ADR-0074) needs no admission barrier: one exported
|
||||
database snapshot is the cut, and objects are immutable and verified before
|
||||
commit, so copying them after the snapshot is consistent. A barrier returns
|
||||
only if state ever spans more than one database.
|
||||
2. Commit region boundaries; seal DB recovery point, referenced objects, ongoing
|
||||
transfer state, region watermarks and versions into the manifest.
|
||||
3. Verify/copy objects, restore isolated PostgreSQL at the compatible point.
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bash
|
||||
# M8.7 world backup (ADR-0074): one consistent, self-verifying bundle of an
|
||||
# AEvolution database and its object store.
|
||||
#
|
||||
# tools/backup/backup-world.sh --objects DIR --out DIR [--label TEXT]
|
||||
#
|
||||
# Connection: PG* environment (PGSERVICE/PGSERVICEFILE, or PGHOST/PGDATABASE/...),
|
||||
# optionally EVO_PGPASS_SOURCE (copied to a private file, as migrate does). The
|
||||
# login must read the whole `evolution` schema (the migrator/owner). Never print
|
||||
# the connection or enable shell tracing here.
|
||||
#
|
||||
# Cut: one exported REPEATABLE READ snapshot names the region heads recorded in the
|
||||
# MANIFEST and is the snapshot pg_dump reads. Objects are copied afterwards: every
|
||||
# object a committed row references was verified durable before its commit, and
|
||||
# objects are immutable and never deleted (no GC), so each one the snapshot
|
||||
# references is present and final when copied. Objects written after the snapshot
|
||||
# are copied too and are harmless orphans in the bundle. Nothing is written to the
|
||||
# source database or the object volume, and no admission barrier is needed.
|
||||
#
|
||||
# Bundle (DIR): MANIFEST, database.dump (pg_dump custom format, schema `evolution`,
|
||||
# no owners or grants: a restore never needs the source's roles or credentials),
|
||||
# objects/w<world>/r<region>/<sha256>, SHA256SUMS over every other file. A source
|
||||
# object whose bytes differ from its name fails the backup.
|
||||
set -euo pipefail
|
||||
usage() { echo "usage: $0 --objects DIR --out DIR [--label TEXT]" >&2; exit 2; }
|
||||
objects="" out="" label=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--objects) objects="${2:-}"; shift 2 ;;
|
||||
--out) out="${2:-}"; shift 2 ;;
|
||||
--label) label="${2:-}"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$objects" ] && [ -n "$out" ] || usage
|
||||
objects="${objects%/}"
|
||||
[[ "$label" =~ ^[A-Za-z0-9._:-]*$ ]] || { echo "label: letters, digits and ._:- only" >&2; exit 2; }
|
||||
[ -d "$objects" ] && [ ! -L "$objects" ] || { echo "objects: not a real directory: $objects" >&2; exit 2; }
|
||||
if [ -e "$out" ] && [ -n "$(ls -A "$out")" ]; then echo "out: not empty: $out" >&2; exit 2; fi
|
||||
umask 077
|
||||
if [ -n "${EVO_PGPASS_SOURCE:-}" ]; then
|
||||
export PGPASSFILE
|
||||
PGPASSFILE="$(mktemp /tmp/evo-pgpass.XXXXXXXX)"
|
||||
trap 'rm -f -- "$PGPASSFILE"' EXIT
|
||||
cat "$EVO_PGPASS_SOURCE" > "$PGPASSFILE"
|
||||
fi
|
||||
mkdir -p "$out"
|
||||
started="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
begin_ms="$(date +%s%3N)"
|
||||
|
||||
# The snapshot is held by this psql session while pg_dump reads it (\!).
|
||||
export EVO_BACKUP_DUMP="$out/database.dump"
|
||||
psql -X --no-password -q -A -t -v ON_ERROR_STOP=1 > "$out/.cut" <<'SQL'
|
||||
\pset fieldsep ' '
|
||||
BEGIN ISOLATION LEVEL REPEATABLE READ, READ ONLY;
|
||||
SELECT pg_export_snapshot() AS snapshot \gset
|
||||
SELECT 'database', current_setting('server_version');
|
||||
SELECT 'schema', version, sha256 FROM evolution.schema_migrations ORDER BY version;
|
||||
SELECT 'build', encode(build_identity, 'hex'),
|
||||
CASE WHEN retired_at IS NULL THEN 'retained' ELSE 'retired' END
|
||||
FROM evolution.retained_builds ORDER BY 2;
|
||||
SELECT 'head', h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick,
|
||||
h.commit_id, encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
|
||||
FROM evolution.region_heads h
|
||||
JOIN evolution.region_leases l USING (world_id, region_id)
|
||||
JOIN evolution.region_commits c ON c.world_id = h.world_id AND c.region_id = h.region_id
|
||||
AND c.commit_sequence = h.commit_sequence
|
||||
ORDER BY h.world_id, h.region_id;
|
||||
SELECT 'rows', (SELECT count(*) FROM evolution.region_commits),
|
||||
(SELECT count(*) FROM evolution.region_decisions),
|
||||
(SELECT count(*) FROM evolution.region_outbox),
|
||||
(SELECT count(*) FROM evolution.region_origins);
|
||||
\setenv EVO_BACKUP_SNAPSHOT :snapshot
|
||||
\! pg_dump --no-password --format=custom --schema=evolution --no-owner --no-privileges --snapshot="$EVO_BACKUP_SNAPSHOT" --file="$EVO_BACKUP_DUMP" && echo 'dumped' || echo 'dump-failed'
|
||||
COMMIT;
|
||||
SQL
|
||||
grep -qx 'dumped' "$out/.cut" || { echo "backup failed: pg_dump did not complete" >&2; exit 3; }
|
||||
pg_restore --list "$out/database.dump" > /dev/null || { echo "backup failed: unreadable dump" >&2; exit 3; }
|
||||
|
||||
# Objects, verified by name while copying; private temporaries are skipped.
|
||||
count=0 bytes=0
|
||||
while IFS= read -r -d '' file; do
|
||||
rel="${file#"$objects"/}"
|
||||
name="${rel##*/}"
|
||||
[[ "$rel" =~ ^w[0-9]+/r[0-9]+/[0-9a-f]{64}$ ]] || continue
|
||||
mkdir -p "$out/objects/${rel%/*}"
|
||||
cp -- "$file" "$out/objects/$rel"
|
||||
sum="$(sha256sum -- "$out/objects/$rel")"
|
||||
[ "${sum%% *}" = "$name" ] || { echo "backup failed: source object $rel is corrupt" >&2; exit 3; }
|
||||
count=$((count + 1))
|
||||
bytes=$((bytes + $(stat -c %s -- "$out/objects/$rel")))
|
||||
done < <(find "$objects" -mindepth 3 -maxdepth 3 -type f ! -name '.tmp-*' -print0)
|
||||
|
||||
{
|
||||
echo "format aevolution-backup-v1"
|
||||
echo "created $started"
|
||||
echo "label ${label:--}"
|
||||
grep -E '^(database|schema|build|head|rows) ' "$out/.cut"
|
||||
echo "objects $count $bytes"
|
||||
} > "$out/MANIFEST"
|
||||
rm -f "$out/.cut"
|
||||
heads="$(grep -c '^head ' "$out/MANIFEST" || true)"
|
||||
[ "$heads" -gt 0 ] || { echo "backup failed: no provisioned regions" >&2; exit 3; }
|
||||
(cd "$out" && find . -type f ! -name SHA256SUMS -printf '%P\0' | sort -z | xargs -0 sha256sum -- > SHA256SUMS)
|
||||
echo "backup complete regions=$heads objects=$count object_bytes=$bytes dump_bytes=$(stat -c %s "$out/database.dump") duration_ms=$(( $(date +%s%3N) - begin_ms )) out=$out"
|
||||
@@ -0,0 +1,135 @@
|
||||
#!/usr/bin/env bash
|
||||
# M8.7 operator-run backup of a disposable environment to storage off the cluster
|
||||
# (ADR-0074).
|
||||
#
|
||||
# tools/backup/cluster-backup.sh <dev|stage> --to DIR [--verify WORLD] [--context NAME]
|
||||
#
|
||||
# Runs one short-lived Pod in evolution-<env> (image: the postgres-tools digest
|
||||
# pinned in deploy/kubernetes/stores/<env>; migrator login; the aevolution-objects
|
||||
# volume mounted read-only on the store node) that runs backup-world.sh into an
|
||||
# emptyDir. The bundle is then streamed out with `kubectl exec tar` to
|
||||
# DIR/<env>-<UTC time>/, its checksums are verified here, and the Pod is deleted
|
||||
# (also on failure). DIR is the destination: until the maintainer chooses the
|
||||
# permanent one (docs/BACKLOG.md), it is any directory on a machine outside the
|
||||
# hypervisor host. --verify WORLD then runs verify-backup.sh on this machine
|
||||
# (restore on a private cluster, rebuilt and served as isolated world WORLD); it
|
||||
# needs the PostgreSQL 18.6 binaries and evo-server/evo-world-clone.
|
||||
# World0 is refused before anything touches the cluster. The Pod is an operator
|
||||
# action, like fault-drill.sh; it is not part of any Argo application. A scheduled
|
||||
# backup (CronJob in Git, synced by Argo) follows once the destination exists.
|
||||
set -euo pipefail
|
||||
usage() { echo "usage: $0 <dev|stage> --to DIR [--verify WORLD] [--context NAME]" >&2; exit 2; }
|
||||
[ $# -ge 1 ] || usage
|
||||
env="$1"; shift
|
||||
case "$env" in
|
||||
dev|stage) ;;
|
||||
*) echo "refused: '$env' is not a disposable environment (only dev or stage; never world0)" >&2
|
||||
exit 2 ;;
|
||||
esac
|
||||
to="" verify="" k=(kubectl --request-timeout=30s)
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--to) to="${2:-}"; shift 2 ;;
|
||||
--verify) verify="${2:-}"; shift 2 ;;
|
||||
--context) k+=(--context "${2:?}"); shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$to" ] && [ -d "$to" ] || { echo "--to must be an existing directory" >&2; exit 2; }
|
||||
root="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
ns="evolution-$env"
|
||||
image="$(awk '/name: aevolution-postgres-tools/ {f = 1} f && /newName:/ {n = $2} f && /digest:/ {print n "@" $2; exit}' \
|
||||
"$root/deploy/kubernetes/stores/$env/kustomization.yaml")"
|
||||
[[ "$image" =~ ^[a-z0-9./-]+@sha256:[0-9a-f]{64}$ ]] || { echo "no pinned postgres-tools image for $env" >&2; exit 2; }
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
pod="aevolution-backup-$(date -u +%Y%m%d%H%M%S)"
|
||||
dest="$to/$env-$stamp"
|
||||
say() { echo "$(date -u +%H:%M:%S) $*"; }
|
||||
cleanup() { "${k[@]}" delete pod -n "$ns" "$pod" --wait=false > /dev/null 2>&1 || true; }
|
||||
trap cleanup EXIT
|
||||
|
||||
say "starting $pod in $ns ($image)"
|
||||
"${k[@]}" apply -n "$ns" -f - > /dev/null <<YAML
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: $pod
|
||||
labels:
|
||||
app.kubernetes.io/part-of: aevolution
|
||||
app.kubernetes.io/component: backup
|
||||
spec:
|
||||
serviceAccountName: aevolution-migrator
|
||||
automountServiceAccountToken: false
|
||||
restartPolicy: Never
|
||||
activeDeadlineSeconds: 3600
|
||||
nodeSelector:
|
||||
aevolution.io/stores: "true"
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 10001
|
||||
runAsGroup: 10001
|
||||
fsGroup: 10001
|
||||
seccompProfile: {type: RuntimeDefault}
|
||||
containers:
|
||||
- name: backup
|
||||
image: $image
|
||||
command: ["/bin/bash", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -eu
|
||||
bash /opt/evo/tools/backup-world.sh --objects /objects --out /work/bundle --label "$env-$stamp"
|
||||
touch /work/complete
|
||||
exec sleep 1800
|
||||
env:
|
||||
- {name: PGSERVICE, value: evolution}
|
||||
- {name: PGSERVICEFILE, value: /run/postgres/pg_service.conf}
|
||||
- {name: EVO_PGPASS_SOURCE, value: /run/postgres/pgpass}
|
||||
resources:
|
||||
requests: {cpu: 100m, memory: 128Mi}
|
||||
limits: {memory: 512Mi}
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities: {drop: [ALL]}
|
||||
volumeMounts:
|
||||
- {name: objects, mountPath: /objects, readOnly: true}
|
||||
- {name: connection, mountPath: /run/postgres, readOnly: true}
|
||||
- {name: work, mountPath: /work}
|
||||
- {name: scratch, mountPath: /tmp}
|
||||
volumes:
|
||||
- name: objects
|
||||
persistentVolumeClaim: {claimName: aevolution-objects, readOnly: true}
|
||||
- name: connection
|
||||
secret:
|
||||
secretName: evolution-postgres-migrator
|
||||
defaultMode: 0440
|
||||
items:
|
||||
- {key: pg_service.conf, path: pg_service.conf}
|
||||
- {key: pgpass, path: pgpass}
|
||||
- {key: ca.crt, path: ca.crt}
|
||||
- name: work
|
||||
emptyDir: {sizeLimit: 12Gi}
|
||||
- name: scratch
|
||||
emptyDir: {medium: Memory, sizeLimit: 8Mi}
|
||||
YAML
|
||||
|
||||
for _ in $(seq 1 1800); do
|
||||
phase="$("${k[@]}" get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2> /dev/null || true)"
|
||||
[ "$phase" = Failed ] && break
|
||||
[ "$phase" = Running ] && "${k[@]}" exec -n "$ns" "$pod" -- test -f /work/complete 2> /dev/null && break
|
||||
sleep 1
|
||||
done
|
||||
"${k[@]}" logs -n "$ns" "$pod" 2>&1 | sed 's/^/ /'
|
||||
"${k[@]}" exec -n "$ns" "$pod" -- test -f /work/complete 2> /dev/null ||
|
||||
{ say "FAIL backup did not complete (phase ${phase:-unknown})"; exit 3; }
|
||||
|
||||
mkdir -p "$dest"
|
||||
"${k[@]}" exec -n "$ns" "$pod" -- tar -C /work/bundle -cf - . | tar -C "$dest" -xf -
|
||||
(cd "$dest" && sha256sum --check --strict --quiet SHA256SUMS) || { say "FAIL fetched bundle checksum mismatch"; exit 3; }
|
||||
say "bundle verified at $dest ($(du -sh "$dest" | cut -f1)); $(grep -c '^head ' "$dest/MANIFEST") region(s):"
|
||||
grep '^head ' "$dest/MANIFEST" | sed 's/^/ /'
|
||||
if [ -n "$verify" ]; then
|
||||
bash "$root/tools/backup/verify-backup.sh" "$dest" --isolate "$verify"
|
||||
else
|
||||
say "next: tools/backup/verify-backup.sh $dest --isolate <world> (restore test on independent storage)"
|
||||
fi
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env bash
|
||||
# M8.7 verified restore of a backup bundle (ADR-0074) onto independent storage.
|
||||
#
|
||||
# tools/backup/restore-world.sh --bundle DIR --objects DIR
|
||||
#
|
||||
# Target connection: PG* environment naming an EMPTY database that the login owns
|
||||
# (PGHOST/PGPORT/PGDATABASE/PGUSER, or PGSERVICE with PGSERVICEFILE/PGPASSFILE).
|
||||
# The restore needs no source role, credential or key: the dump carries no owners
|
||||
# or grants, so the target login owns everything restored. --objects must not
|
||||
# exist yet (it is created 0700). evo-world-clone comes from EVO_WORLD_CLONE_BIN or
|
||||
# PATH.
|
||||
#
|
||||
# Order, failing closed at each step:
|
||||
# 1. every bundle file matches SHA256SUMS, no file is unlisted, and every object's
|
||||
# name is its digest; nothing is touched before this passes;
|
||||
# 2. the target has no `evolution` schema and runs the manifest's server version;
|
||||
# 3. pg_restore in one transaction, then objects are copied;
|
||||
# 4. the restored heads equal the MANIFEST heads exactly;
|
||||
# 5. every region's full history is audited (evo-world-clone audit: each link's
|
||||
# manifest, payload, trace replay and outcomes, from genesis to the head).
|
||||
# On a failure after step 2 the restored schema and the copied objects are removed,
|
||||
# so an unverified restore can never be mistaken for a world. The result is a
|
||||
# verbatim, owner-only copy with the source's world identities: it has no runtime
|
||||
# grants and is never served. To run it, clone it into an isolated world
|
||||
# (tools/backup/verify-backup.sh --isolate); host-loss recovery in place is M8.8.
|
||||
set -euo pipefail
|
||||
usage() { echo "usage: $0 --bundle DIR --objects DIR" >&2; exit 2; }
|
||||
bundle="" objects=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--bundle) bundle="${2:-}"; shift 2 ;;
|
||||
--objects) objects="${2:-}"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$bundle" ] && [ -n "$objects" ] || usage
|
||||
bundle="$(realpath -- "$bundle")" objects="$(realpath -m -- "$objects")"
|
||||
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
|
||||
umask 077
|
||||
begin_ms="$(date +%s%3N)"
|
||||
refuse() { echo "RESTORE REFUSED: $*" >&2; exit 4; }
|
||||
|
||||
# 1. Bundle integrity, before anything else is touched.
|
||||
[ -f "$bundle/MANIFEST" ] && [ -f "$bundle/SHA256SUMS" ] && [ -f "$bundle/database.dump" ] ||
|
||||
refuse "incomplete bundle $bundle"
|
||||
(cd "$bundle" && sha256sum --check --strict --quiet SHA256SUMS) || refuse "bundle checksum mismatch"
|
||||
listed="$(sed -E 's/^[0-9a-f]{64} //' "$bundle/SHA256SUMS" | sort)"
|
||||
present="$(cd "$bundle" && find . -type f ! -name SHA256SUMS -printf '%P\n' | sort)"
|
||||
[ "$listed" = "$present" ] || refuse "bundle files differ from SHA256SUMS"
|
||||
while IFS= read -r rel; do
|
||||
case "$rel" in
|
||||
objects/*)
|
||||
[[ "$rel" =~ ^objects/w[0-9]+/r[0-9]+/([0-9a-f]{64})$ ]] || refuse "unexpected object path $rel"
|
||||
grep -qE "^${BASH_REMATCH[1]} $rel\$" "$bundle/SHA256SUMS" || refuse "object $rel is not its digest"
|
||||
;;
|
||||
esac
|
||||
done <<< "$present"
|
||||
grep -qx 'format aevolution-backup-v1' "$bundle/MANIFEST" || refuse "unknown bundle format"
|
||||
expected_heads="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2- | sort)"
|
||||
[ -n "$expected_heads" ] || refuse "manifest names no region"
|
||||
|
||||
# 2. Target: empty, same server version.
|
||||
[ ! -e "$objects" ] || refuse "objects target already exists: $objects"
|
||||
version="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT current_setting('server_version')")"
|
||||
grep -qx "database $version" "$bundle/MANIFEST" || refuse "target server $version differs from the manifest"
|
||||
existing="$(psql -X --no-password -A -t -v ON_ERROR_STOP=1 -c "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")"
|
||||
[ "$existing" = 0 ] || refuse "target database already has an evolution schema"
|
||||
|
||||
restored=0
|
||||
cleanup() {
|
||||
[ "$restored" = 1 ] && return
|
||||
psql -X --no-password -q -v ON_ERROR_STOP=1 -c 'DROP SCHEMA IF EXISTS evolution CASCADE' > /dev/null 2>&1 || true
|
||||
rm -rf -- "$objects"
|
||||
echo "RESTORE FAILED: the restored schema and objects were removed" >&2
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
# 3. Database in one transaction, then objects.
|
||||
target="${PGDATABASE:-}"
|
||||
[ -n "${PGSERVICE:-}" ] && target="service=$PGSERVICE"
|
||||
[ -n "$target" ] || { echo "set PGDATABASE or PGSERVICE for the target" >&2; exit 2; }
|
||||
pg_restore --no-password --exit-on-error --single-transaction --no-owner --no-privileges \
|
||||
--dbname="$target" "$bundle/database.dump"
|
||||
mkdir -m 700 -- "$objects"
|
||||
(cd "$bundle/objects" && find . -mindepth 2 -maxdepth 2 -type d -printf '%P\0' |
|
||||
xargs -0 -r -I{} mkdir -p -m 700 -- "$objects/{}")
|
||||
(cd "$bundle/objects" && find . -type f -printf '%P\0' | xargs -0 -r -I{} cp -- {} "$objects/{}")
|
||||
|
||||
# 4. Heads exactly as recorded.
|
||||
actual_heads="$(psql -X --no-password -A -t -F ' ' -v ON_ERROR_STOP=1 -c "
|
||||
SELECT h.world_id, h.region_id, l.frame_id, h.commit_sequence, h.end_tick, h.commit_id,
|
||||
encode(h.digest, 'hex'), encode(c.build_identity, 'hex')
|
||||
FROM evolution.region_heads h
|
||||
JOIN evolution.region_leases l USING (world_id, region_id)
|
||||
JOIN evolution.region_commits c ON c.world_id = h.world_id AND c.region_id = h.region_id
|
||||
AND c.commit_sequence = h.commit_sequence" | sort)"
|
||||
[ "$actual_heads" = "$expected_heads" ] || { echo "restored heads differ from the manifest" >&2; exit 3; }
|
||||
|
||||
# 5. Full history audit of every region.
|
||||
pg=()
|
||||
if [ -n "${PGSERVICEFILE:-}" ]; then
|
||||
pg=(--pg-service "$PGSERVICEFILE")
|
||||
[ -n "${PGPASSFILE:-}" ] && pg+=(--pg-passfile "$PGPASSFILE")
|
||||
else
|
||||
pg=(--pg-host "${PGHOST:?}" --pg-port "${PGPORT:-5432}" --pg-database "${PGDATABASE:?}"
|
||||
--pg-user "${PGUSER:?}")
|
||||
[ -n "${PGPASSFILE:-}" ] && pg+=(--pg-passfile "$PGPASSFILE")
|
||||
fi
|
||||
retained=()
|
||||
while read -r _ build _; do retained+=(--retained "$build"); done < <(grep '^build ' "$bundle/MANIFEST")
|
||||
commits=0
|
||||
while read -r world region _ _ tick _; do
|
||||
line="$("$clone_bin" audit --world "$world" --region "$region" --objects "$objects" \
|
||||
"${pg[@]}" "${retained[@]}")" || { echo "audit failed for $world/$region" >&2; exit 3; }
|
||||
[[ "$line" =~ ^audited\ commits=([0-9]+)\ tick=([0-9]+)\ ]] || { echo "audit output: $line" >&2; exit 3; }
|
||||
[ "${BASH_REMATCH[2]}" = "$tick" ] || { echo "audit of $world/$region ended at ${BASH_REMATCH[2]}, not $tick" >&2; exit 3; }
|
||||
commits=$((commits + BASH_REMATCH[1]))
|
||||
echo "region $world/$region $line"
|
||||
done <<< "$expected_heads"
|
||||
restored=1
|
||||
echo "RESTORE VERIFIED regions=$(wc -l <<< "$expected_heads") commits=$commits duration_ms=$(( $(date +%s%3N) - begin_ms ))"
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env bash
|
||||
# M8.7 restore test on independent storage (ADR-0074).
|
||||
#
|
||||
# tools/backup/verify-backup.sh BUNDLE [--isolate WORLD [--source-world W]] [--keep]
|
||||
#
|
||||
# Starts a private, throwaway PostgreSQL cluster (initdb/pg_ctl/psql/pg_restore of
|
||||
# the pinned server version on PATH) on local disk, with no connection to the
|
||||
# source database or cluster, then:
|
||||
# 1. restores BUNDLE with tools/backup/restore-world.sh as a fresh login that owns
|
||||
# only the restore database (no source role, credential or key is used);
|
||||
# 2. with --isolate WORLD, rebuilds a runnable isolated world: a separate database
|
||||
# with its own owner, runtime group and login (migrated by migrate-postgres.sh),
|
||||
# each region of the source world cloned into WORLD from the restored copy
|
||||
# (evo-world-clone clone, source history audited first), and evo-server run on
|
||||
# each cloned region under synthetic load until it acknowledges 20 ticks past
|
||||
# the clone, then drained (exit 0).
|
||||
# The cluster and its files are removed on exit unless --keep. Binaries come from
|
||||
# EVO_WORLD_CLONE_BIN / EVO_SERVER_BIN or PATH. Port: EVO_VERIFY_PORT (55499).
|
||||
set -euo pipefail
|
||||
usage() { echo "usage: $0 BUNDLE [--isolate WORLD [--source-world W]] [--keep]" >&2; exit 2; }
|
||||
[ $# -ge 1 ] || usage
|
||||
bundle="$(realpath -- "$1")"; shift
|
||||
isolate="" source_world="" keep=0
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--isolate) isolate="${2:-}"; shift 2 ;;
|
||||
--source-world) source_world="${2:-}"; shift 2 ;;
|
||||
--keep) keep=1; shift ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[[ -z "$isolate" || "$isolate" =~ ^[1-9][0-9]*$ ]] || usage
|
||||
root="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
clone_bin="${EVO_WORLD_CLONE_BIN:-evo-world-clone}"
|
||||
server_bin="${EVO_SERVER_BIN:-evo-server}"
|
||||
port="${EVO_VERIFY_PORT:-55499}"
|
||||
say() { echo "$(date -u +%H:%M:%S) $*"; }
|
||||
|
||||
cluster="$(mktemp -d /tmp/evo-verify.XXXXXXXX)"
|
||||
chmod 700 "$cluster"
|
||||
server_pid=""
|
||||
finish() {
|
||||
[ -n "$server_pid" ] && kill -9 "$server_pid" 2> /dev/null
|
||||
pg_ctl -D "$cluster/data" -m immediate stop > /dev/null 2>&1 || true
|
||||
if [ "$keep" = 1 ]; then say "kept $cluster"; else rm -rf -- "$cluster"; fi
|
||||
}
|
||||
trap finish EXIT
|
||||
unset PGSERVICE PGSERVICEFILE PGPASSFILE PGHOSTADDR PGPASSWORD EVO_PGPASS_SOURCE
|
||||
export PGHOST="$cluster" PGPORT="$port"
|
||||
initdb -D "$cluster/data" -U evo_verify_admin --no-locale --encoding=UTF8 \
|
||||
--auth-local=trust --auth-host=reject > "$cluster/initdb.log"
|
||||
pg_ctl -D "$cluster/data" -l "$cluster/server.log" -w \
|
||||
-o "-k $cluster -p $port -c listen_addresses=''" start > /dev/null
|
||||
admin() { PGUSER=evo_verify_admin PGDATABASE=postgres psql -X -q -v ON_ERROR_STOP=1 "$@"; }
|
||||
admin -c 'CREATE ROLE evo_restore_owner LOGIN' \
|
||||
-c 'CREATE DATABASE evo_restored OWNER evo_restore_owner'
|
||||
|
||||
say "restoring $bundle on independent storage ($cluster)"
|
||||
PGUSER=evo_restore_owner PGDATABASE=evo_restored \
|
||||
bash "$root/tools/backup/restore-world.sh" --bundle "$bundle" --objects "$cluster/restored"
|
||||
[ -n "$isolate" ] || { say "PASS restore verified"; exit 0; }
|
||||
|
||||
# The source world to rebuild: the only one in the bundle, or --source-world.
|
||||
worlds="$(grep '^head ' "$bundle/MANIFEST" | cut -d' ' -f2 | sort -u)"
|
||||
if [ -z "$source_world" ]; then
|
||||
[ "$(wc -l <<< "$worlds")" = 1 ] || { echo "bundle holds several worlds; pass --source-world" >&2; exit 2; }
|
||||
source_world="$worlds"
|
||||
fi
|
||||
grep -qx "$source_world" <<< "$worlds" || { echo "world $source_world is not in the bundle" >&2; exit 2; }
|
||||
[ "$source_world" != "$isolate" ] || { echo "the isolated world must differ from the source world" >&2; exit 2; }
|
||||
|
||||
admin -c 'CREATE ROLE evo_isolated_owner LOGIN' -c 'CREATE ROLE evo_isolated_runtime NOLOGIN' \
|
||||
-c 'CREATE ROLE evo_isolated_login LOGIN IN ROLE evo_isolated_runtime' \
|
||||
-c 'CREATE DATABASE evo_isolated OWNER evo_isolated_owner'
|
||||
PGUSER=evo_isolated_owner PGDATABASE=evo_isolated EVO_PG_RUNTIME_ROLE=evo_isolated_runtime \
|
||||
bash "$root/tools/migrate-postgres.sh" > /dev/null
|
||||
retained=()
|
||||
while read -r _ build _; do
|
||||
retained+=(--retained "$build")
|
||||
PGUSER=evo_isolated_owner PGDATABASE=evo_isolated psql -X -q -v ON_ERROR_STOP=1 -c \
|
||||
"INSERT INTO evolution.retained_builds(build_identity) VALUES (decode('$build', 'hex'))"
|
||||
done < <(grep '^build ' "$bundle/MANIFEST")
|
||||
mkdir -m 700 "$cluster/isolated"
|
||||
local_pg() { # local_pg <prefix> <database> <user>
|
||||
echo "$1host $cluster $1port $port $1database $2 $1user $3"
|
||||
}
|
||||
|
||||
while read -r _ world region frame _ tick _ _ build; do
|
||||
[ "$world" = "$source_world" ] || continue
|
||||
# shellcheck disable=SC2046 # word splitting of the fixed connection flags is intended.
|
||||
cloned="$("$clone_bin" clone --source-world "$world" --source-region "$region" \
|
||||
--target-world "$isolate" --target-region "$region" \
|
||||
$(local_pg --source-pg- evo_restored evo_restore_owner) \
|
||||
$(local_pg --target-pg- evo_isolated evo_isolated_owner) \
|
||||
--source-objects "$cluster/restored" --target-objects "$cluster/isolated" \
|
||||
--audit-max 1048576 "${retained[@]}")"
|
||||
say "$cloned"
|
||||
log="$cluster/server-$region.log"
|
||||
"$server_bin" --profile lineage --world "$isolate" --region "$region" --frame "$frame" \
|
||||
$(local_pg --pg- evo_isolated evo_isolated_login) --objects "$cluster/isolated" \
|
||||
--build "$build" "${retained[@]}" --flush-ms 500 --drain-ms 20000 \
|
||||
--synthetic-inputs development-v1 < /dev/null > "$log" 2>&1 &
|
||||
server_pid=$!
|
||||
target=$((tick + 20)) served=0
|
||||
for _ in $(seq 1 600); do
|
||||
if awk -v t="$target" '$1 == "ack" && $3 >= t {found = 1} END {exit !found}' "$log"; then served=1; break; fi
|
||||
kill -0 "$server_pid" 2> /dev/null || break
|
||||
sleep 0.1
|
||||
done
|
||||
running="$(grep -m1 '^phase running' "$log" || true)"
|
||||
kill -TERM "$server_pid" 2> /dev/null || true
|
||||
code=0; wait "$server_pid" || code=$?
|
||||
server_pid=""
|
||||
[ "$served" = 1 ] && [ "$code" = 0 ] ||
|
||||
{ tail -n 20 "$log" >&2; echo "FAIL isolated $isolate/$region did not serve (exit $code)" >&2; exit 3; }
|
||||
[[ "$running" == "phase running epoch=1 committed=$tick "* ]] ||
|
||||
{ echo "FAIL isolated $isolate/$region resumed at '$running', not the backup head $tick" >&2; exit 3; }
|
||||
say "isolated $isolate/$region served from tick $tick to $(awk '$1 == "ack" {t = $3} END {print t}' "$log") and drained"
|
||||
done < <(grep '^head ' "$bundle/MANIFEST")
|
||||
say "PASS restore verified and world $source_world rebuilt as isolated world $isolate"
|
||||
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
# M8.7 backup and restore qualification (ADR-0074). Run by postgres-check.sh inside
|
||||
# its throwaway cluster (PG* already set; EVO_SERVER_BIN, EVO_WORLD_CLONE_BIN and the
|
||||
# runtime roles exported). A real evo-server commits under synthetic load while the
|
||||
# backup is taken; the bundle is then restored on a separate cluster and rebuilt as a
|
||||
# runnable isolated world, and damaged bundles are refused.
|
||||
set -euo pipefail
|
||||
root="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
work="$(mktemp -d /tmp/evo-backup-check.XXXXXXXX)"
|
||||
chmod 700 "$work"
|
||||
server_pid=""
|
||||
trap '[ -n "$server_pid" ] && kill -9 "$server_pid" 2> /dev/null; rm -rf -- "$work"' EXIT
|
||||
fail() { echo "backup check FAILED: $*" >&2; exit 1; }
|
||||
build=b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1
|
||||
createdb evo_backup_src
|
||||
export PGDATABASE=evo_backup_src
|
||||
bash "$root/tools/migrate-postgres.sh" > /dev/null
|
||||
mkdir -m 700 "$work/objects"
|
||||
pg=(--pg-host "$PGHOST" --pg-port "$PGPORT" --pg-database evo_backup_src)
|
||||
"$EVO_WORLD_CLONE_BIN" provision --world 300 --region 1 --opening development-v1 \
|
||||
"${pg[@]}" --pg-user "$PGUSER" --objects "$work/objects" --build "$build" --retain-build true > /dev/null
|
||||
"$EVO_SERVER_BIN" --profile lineage --world 300 --region 1 --frame 9 "${pg[@]}" \
|
||||
--pg-user "$EVO_PG_RUNTIME_USER" --objects "$work/objects" --build "$build" \
|
||||
--flush-ms 200 --lease-profile fast --synthetic-inputs development-v1 \
|
||||
< /dev/null > "$work/server.log" 2>&1 &
|
||||
server_pid=$!
|
||||
acked_through() { awk -v t="$1" '$1 == "ack" && $3 >= t {f = 1} END {exit !f}' "$work/server.log"; }
|
||||
max_ack() { awk '$1 == "ack" && $3 > m {m = $3} END {print m + 0}' "$work/server.log"; }
|
||||
for _ in $(seq 1 300); do acked_through 60 && break; sleep 0.1; done
|
||||
acked_through 60 || fail "the source server acknowledged nothing"
|
||||
|
||||
# Online backup while the server keeps committing.
|
||||
acked_before="$(max_ack)"
|
||||
bash "$root/tools/backup/backup-world.sh" --objects "$work/objects" --out "$work/bundle" --label check
|
||||
read -r _ _ _ _ _ backup_tick _ < <(grep '^head 300 1 ' "$work/bundle/MANIFEST")
|
||||
[ "$backup_tick" -ge "$acked_before" ] || fail "backup head $backup_tick misses acknowledged tick $acked_before"
|
||||
for _ in $(seq 1 300); do acked_through $((backup_tick + 40)) && break; sleep 0.1; done
|
||||
kill -TERM "$server_pid"; code=0; wait "$server_pid" || code=$?; server_pid=""
|
||||
[ "$code" = 0 ] || fail "the source server exited $code"
|
||||
source_tick="$(psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')"
|
||||
[ "$source_tick" -gt "$backup_tick" ] || fail "the source did not move past the backup ($source_tick)"
|
||||
grep -q '^build b1b1' "$work/bundle/MANIFEST" && grep -q '^schema 9 ' "$work/bundle/MANIFEST" ||
|
||||
fail "manifest lacks builds or schema versions"
|
||||
if grep -q -i -E 'password|pgpass' "$work/bundle/MANIFEST"; then fail "manifest names credentials"; fi
|
||||
|
||||
# Full restore on an independent cluster, rebuilt and served as isolated world 301.
|
||||
EVO_VERIFY_PORT=$((PGPORT + 1)) bash "$root/tools/backup/verify-backup.sh" "$work/bundle" \
|
||||
--isolate 301 > "$work/verify.log" 2>&1 || { cat "$work/verify.log" >&2; fail "verify-backup"; }
|
||||
grep -q "^RESTORE VERIFIED regions=1 " "$work/verify.log" &&
|
||||
grep -q "PASS restore verified and world 300 rebuilt as isolated world 301" "$work/verify.log" ||
|
||||
{ cat "$work/verify.log" >&2; fail "verify-backup evidence"; }
|
||||
|
||||
# Damaged bundles, restored straight into scratch databases of this cluster.
|
||||
case_n=0
|
||||
expect_refusal() { # expect_refusal <exit code> <bundle> <what>
|
||||
case_n=$((case_n + 1))
|
||||
createdb "evo_restore_case_$case_n"
|
||||
local code=0
|
||||
PGDATABASE="evo_restore_case_$case_n" bash "$root/tools/backup/restore-world.sh" \
|
||||
--bundle "$2" --objects "$work/restored-$case_n" > "$work/case-$case_n.log" 2>&1 || code=$?
|
||||
[ "$code" = "$1" ] || { cat "$work/case-$case_n.log" >&2; fail "$3: exit $code, expected $1"; }
|
||||
[ ! -e "$work/restored-$case_n" ] || fail "$3: restored objects were left behind"
|
||||
[ "$(PGDATABASE="evo_restore_case_$case_n" psql -XAtc "SELECT count(*) FROM pg_namespace WHERE nspname = 'evolution'")" = 0 ] ||
|
||||
fail "$3: a restored schema was left behind"
|
||||
}
|
||||
damaged() { rm -rf "$work/damaged"; cp -a "$work/bundle" "$work/damaged"; }
|
||||
resum() { (cd "$work/damaged" && find . -type f ! -name SHA256SUMS -printf '%P\0' | sort -z | xargs -0 sha256sum -- > SHA256SUMS); }
|
||||
genesis="$(psql -XAtc "SELECT encode(payload_sha256, 'hex') FROM evolution.region_commits WHERE world_id = 300 AND commit_sequence = 0")"
|
||||
genesis_object="$work/damaged/objects/w300/r1/$genesis"
|
||||
|
||||
damaged; printf 'corrupt' | dd of="$genesis_object" bs=1 conv=notrunc status=none
|
||||
expect_refusal 4 "$work/damaged" "a corrupt object"
|
||||
damaged; echo extra > "$work/damaged/objects/w300/r1/notes"
|
||||
expect_refusal 4 "$work/damaged" "an unlisted file"
|
||||
damaged; printf 'x' >> "$work/damaged/database.dump"
|
||||
expect_refusal 4 "$work/damaged" "a damaged dump"
|
||||
# Consistently re-summed damage passes the checksums and must fail verification.
|
||||
damaged; rm "$genesis_object"; resum
|
||||
expect_refusal 3 "$work/damaged" "a missing payload object"
|
||||
grep -q 'audit failed for 300/1' "$work/case-$case_n.log" || fail "missing object: no audit failure"
|
||||
damaged; sed -i -E "s/^(head 300 1 [0-9]+ [0-9]+) $backup_tick /\1 $((backup_tick - 1)) /" "$work/damaged/MANIFEST"; resum
|
||||
expect_refusal 3 "$work/damaged" "a manifest head that disagrees with the database"
|
||||
# A target that already holds a world is refused before anything is written.
|
||||
createdb evo_restore_twice
|
||||
PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
|
||||
--objects "$work/twice-a" > /dev/null
|
||||
code=0
|
||||
PGDATABASE=evo_restore_twice bash "$root/tools/backup/restore-world.sh" --bundle "$work/bundle" \
|
||||
--objects "$work/twice-b" > "$work/twice.log" 2>&1 || code=$?
|
||||
[ "$code" = 4 ] && grep -q 'already has an evolution schema' "$work/twice.log" || fail "a second restore into one database"
|
||||
[ "$(PGDATABASE=evo_restore_twice psql -XAtc 'SELECT end_tick FROM evolution.region_heads WHERE world_id = 300')" = "$backup_tick" ] ||
|
||||
fail "the refused second restore changed the first"
|
||||
echo "backup and restore qualification passed (backup tick $backup_tick, acknowledged $acked_before, source $source_tick)"
|
||||
@@ -18,10 +18,10 @@ case "$suite" in
|
||||
grep -Eq '^(CMakeLists.txt|CMakePresets.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|experiments/|tools/ci/linux-check.sh|tools/ci/tsan-profile.mjs|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
|
||||
;;
|
||||
postgres|postgres-tools)
|
||||
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|migrations/|tools/prepare-postgres.sh|tools/ci/postgres-check.sh|tools/migrate-postgres.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
|
||||
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|tests/|migrations/|tools/prepare-postgres.sh|tools/ci/postgres-check.sh|tools/ci/backup-check.sh|tools/backup/|tools/migrate-postgres.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
|
||||
;;
|
||||
deploy)
|
||||
grep -Eq '^(deploy/|tools/ci/deploy-policy-check.py|\.gitea/workflows/)' <<<"$files"
|
||||
grep -Eq '^(deploy/|tools/ci/deploy-policy-check.py|tools/ops/|tools/backup/|\.gitea/workflows/)' <<<"$files"
|
||||
;;
|
||||
server-image)
|
||||
grep -Eq '^(CMakeLists.txt|CMakePresets.json|cmake/|include/|src/|apps/|migrations/|tools/prepare-postgres.sh|tools/ci/server-image-smoke.sh|deploy/docker/Dockerfile|\.gitea/workflows/)' <<<"$files"
|
||||
|
||||
@@ -302,10 +302,11 @@ for path, doc in objects:
|
||||
'RoleBinding', 'ClusterRole', 'ClusterRoleBinding'},
|
||||
f'{path}: {doc["kind"]} must not live in the Argo workload path')
|
||||
|
||||
# The cleanup tool refuses World0 before kubectl runs (no kubectl on PATH here).
|
||||
# The cleanup, drill and backup tools refuse World0 before kubectl runs (no kubectl on PATH here).
|
||||
with tempfile.TemporaryDirectory() as empty:
|
||||
env = dict(os.environ, PATH=empty + os.pathsep + os.path.dirname(sys.executable))
|
||||
for tool in ('tools/ops/cleanup-environment.sh', 'tools/ops/region-drill.sh', 'tools/ops/fault-drill.sh'):
|
||||
for tool in ('tools/ops/cleanup-environment.sh', 'tools/ops/region-drill.sh', 'tools/ops/fault-drill.sh',
|
||||
'tools/backup/cluster-backup.sh'):
|
||||
for bad in ('world0', 'evolution-world0', 'World0', 'prod', ''):
|
||||
result = subprocess.run([shutil.which('bash') or '/bin/bash', os.path.join(ROOT, tool), bad],
|
||||
capture_output=True, text=True, env=env)
|
||||
|
||||
@@ -30,6 +30,8 @@ export EVO_SERVER_BIN EVO_WORLD_CLONE_BIN
|
||||
EVO_SERVER_BIN="$(dirname "${1:?path to evo-postgres-tests}")/evo-server"
|
||||
EVO_WORLD_CLONE_BIN="$(dirname "$1")/evo-world-clone"
|
||||
"$1"
|
||||
# M8.7: online backup under load, restore on a separate cluster, isolated rebuild.
|
||||
bash "$root/tools/ci/backup-check.sh"
|
||||
v1_sha="$(sha256sum "$root/migrations/0001_region_bootstrap.sql")"
|
||||
v1_sha="${v1_sha%% *}"
|
||||
# Migration failure must not publish a partial schema.
|
||||
|
||||
Reference in New Issue
Block a user