Review P1 (object durability): a name installed by a put whose directory fsync failed was later reused as success without syncing again, so a publication could acknowledge an object that might vanish on power loss. Reuse (existing and lost-link-race paths) now fsyncs the verified file and the directory before succeeding, and namespace opens always sync their parent. A test-only ObjectStoreFaults hook injects directory sync failures; tests cover failed sync then retries, namespace creation syncs, and publication refusing to proceed until durability holds. Review P1 (pre-v5 history): v5 recovery requires one durable decision per terminal actor, but v3/v4 published checkpoints carry none, so an accepted upgrade could make regions unrecoverable. pre_v5_guard.sql now runs before 0005 in every upgrade path and refuses (EVL10) when any published pre-v5 checkpoint exists; genesis-only regions pass. 0005 is unchanged. CI publishes a real v4 checkpoint and proves the upgrade is refused with nothing changed; a committed death without a decision is shown to recover as quarantined history. Validated: PostgreSQL suites Release x3, clang-tidy, ASan/UBSan x2; reverting each fix is caught by its test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
168 lines
9.9 KiB
Markdown
168 lines
9.9 KiB
Markdown
# PostgreSQL bootstrap (M7.2) and region fencing (M7.3)
|
|
|
|
ADR-0059 pins server/libpq 18.6. `evo::postgres` is optional, Linux-only and separate
|
|
from all simulation targets. Preparation explicitly downloads a verified source
|
|
archive; ordinary CMake configure does not fetch anything. PostgreSQL license is
|
|
permissive; source COPYRIGHT and system TLS license information must accompany
|
|
distributed tooling. No custom hashing/cryptography is introduced.
|
|
|
|
## Local isolated verification
|
|
|
|
Linux prerequisites: GCC14, CMake>=3.28, Ninja, make, bison, flex, curl, bzip2,
|
|
OpenSSL development headers and zlib development headers. Optional clang19 tools
|
|
support analysis/ASan. Run as a normal user, not root:
|
|
|
|
```sh
|
|
bash tools/prepare-postgres.sh
|
|
cmake --preset release -B build/postgres -DCMAKE_CXX_COMPILER=g++-14 \
|
|
-DEVO_BUILD_POSTGRES=ON -DPostgreSQL_ROOT="$PWD/.cache/postgres/install"
|
|
cmake --build build/postgres --target evo-postgres-tests --parallel 2
|
|
bash tools/ci/postgres-check.sh "$PWD/build/postgres/evo-postgres-tests"
|
|
```
|
|
|
|
The test script overrides all connection coordinates and creates a new private
|
|
0700 /tmp directory, private Unix socket and disposable local cluster. It creates
|
|
the `evo_runtime_test` group and the `evo_runtime_login` member, then runs fencing
|
|
tests as that real runtime login. TCP is off;
|
|
local trust applies only inside that private directory. Cluster stops on exit;
|
|
logs/data are retained there for diagnosis. It never uses production PG credentials
|
|
or deletes a pre-existing database. Do not point this script at a shared server.
|
|
|
|
## Adapter contract
|
|
|
|
One connection belongs to one I/O owner; one outstanding parameterized statement.
|
|
No hidden threads, pool, transaction replay or automatic reconnect. Connections
|
|
and queries take 1..30000ms deadlines on a steady clock. Remote hostaddr must be
|
|
numeric, host supplies TLS identity, CA file is explicit, TLS verify-full required.
|
|
Local sockets require no PGHOSTADDR/PGSERVICE override. GSS negotiation is disabled.
|
|
Query SQL <=16KiB, <=16 parameters of <=4KiB, response <=128 rows/16 columns/64KiB
|
|
decoded values; single-row mode prevents accumulating a whole result set. A single
|
|
server field is allocated by libpq before its length can be checked: this is not
|
|
an arbitrary-query memory sandbox. Only trusted bounded persistence SQL is admitted.
|
|
Server statement/lock timeouts default to 10s/5s; callers choose shorter I/O deadlines.
|
|
|
|
Admission config/bounds errors send nothing. SQL errors include only SQLSTATE and
|
|
require caller rollback when inside a transaction. Receive bounds, network errors
|
|
and deadlines close the connection and report unknown: a mutation may have
|
|
committed. Resolve durable request IDs at M7.5; never treat disconnect as rollback.
|
|
Allocation exceptions propagate to the host boundary; do not catch and reuse an
|
|
incomplete session. No PostgreSQL diagnostic strings or connection values are logged.
|
|
|
|
## Migrations and privilege boundary
|
|
|
|
`tools/migrate-postgres.sh` uses psql environment/service files, no URI arguments.
|
|
It requires exactly server 18.6 and serializes migrations with a transaction
|
|
advisory lock. It hashes each checked-in SQL file and applies everything in one
|
|
transaction. The recorded history must be an exact prefix: clean->v1->v2, v1->v2,
|
|
or a v2 no-op. Unknown versions, checksum changes and partial foreign schemas fail
|
|
closed. There is no down migration or destructive repair. The u64 domain rejects
|
|
fractional, negative and above-u64 values without signed truncation.
|
|
|
|
Use a dedicated owner/migrator login with CREATE on the database. Before
|
|
migrating, operators create a NOLOGIN runtime group role and pass its name as
|
|
`EVO_PG_RUNTIME_ROLE`. The runtime login is a separate member of that group only.
|
|
Every migrate run, including a no-op, audits effective privileges of the group and
|
|
every role transitively in it, and rolls back on any violation:
|
|
- privileged attributes or owner identity;
|
|
- membership in any other role, whether inherited or SET-able (for example
|
|
`pg_write_all_data`);
|
|
- any table privilege in `evolution`, including via PUBLIC;
|
|
- CREATE on the schema, the database or `public`;
|
|
- EXECUTE outside the ten fenced functions (schema v5).
|
|
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
|
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
|
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
|
default function execution. Operators own database/role provisioning; the migration
|
|
never creates logins or passwords, and never grants superuser.
|
|
|
|
## Fenced region ownership (schema v2, ADR-0060)
|
|
|
|
In schema v3 the runtime role has schema USAGE and EXECUTE on six SECURITY DEFINER
|
|
functions: `acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
|
|
`load_region_head`, `publish_region_checkpoint` and `load_region_checkpoint`. It
|
|
cannot read or write tables, create objects, call internal check/parse functions or
|
|
provision regions. (Schema v2's `advance_region_head` is dropped in v3.) Each mutation locks the lease row
|
|
and checks holder, epoch and expiry with `clock_timestamp()` after the lock wait.
|
|
Clients never send times. Rejections: EVL01 busy, EVL02 fenced, EVL03 head mismatch,
|
|
EVL04 not provisioned, EVL05 epoch/sequence exhausted, EVL06 unsupported
|
|
profile/version/build or a changed rules/quantum, EVL07 reused commit ID, and 22023
|
|
malformed arguments or manifest. `evo::postgres` maps them to `LeaseStoreError`; a
|
|
timeout or broken connection is `unknown`.
|
|
|
|
## Checkpoint publication (schema v3, ADR-0061)
|
|
|
|
Each commit row stores the exact 280-byte manifest, and a constraint ties the
|
|
commit digest to `sha256(manifest)`. The head has a composite foreign key to its
|
|
commit, so even the owner cannot silently repoint or edit it.
|
|
|
|
Payloads are immutable files under a trusted object root:
|
|
`<root>/w<world>/r<region>/<sha256 hex>`, mode 0700 directories and 0600 files.
|
|
Mount a dedicated volume that only the region host's user can write. Nothing is
|
|
ever deleted; orphans from failed transactions are expected.
|
|
|
|
Operator steps:
|
|
1. Record each promoted build (image digest, 32 bytes) before a host uses it:
|
|
`INSERT INTO evolution.retained_builds(build_identity) VALUES ('\x<64 hex>');`
|
|
2. Provision genesis only through `provision_environmental_region` on an owner
|
|
connection. It uploads and verifies the tick-0 payload, then calls the
|
|
owner-only `provision_region_checkpoint(manifest)`.
|
|
|
|
Schema v4 (ADR-0062) additionally admits profile 2 (`environmental-lineage-v1`,
|
|
payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
|
|
profile, history, RNG and durability declarations. Provision history-capable
|
|
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
|
|
|
|
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
|
|
- `region_decisions` and `region_outbox`, with one terminal outcome per actor;
|
|
- `resolve_region_commit`, `load_region_decisions`, `pending_region_outbox` and
|
|
`acknowledge_region_outbox`, bringing the runtime function set to ten.
|
|
Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
|
|
a first delivery time. Owners can disable triggers, so treat that as an audited
|
|
break-glass action. EVL08 means an outcome was already final. After an
|
|
ambiguous publication, resolve the exact prepared bytes before doing anything
|
|
else. Upgrading a v3/v4 database that has published checkpoints (not just
|
|
genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions,
|
|
or wait for an explicit legacy-history conversion.
|
|
|
|
There is no operator CLI for provisioning yet, and no real world has been
|
|
provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
|
have no payload.
|
|
|
|
## Host lease discipline
|
|
|
|
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
|
|
interval, 2 s safety margin, 2 s renewal deadline and 5 s commit deadline. The
|
|
local deadline runs from request send time on CLOCK_BOOTTIME. If a renewal outcome
|
|
is unknown, keep the earlier deadline and reconnect. Once fenced, stop admission,
|
|
simulation and output, then acquire a new epoch and recover. Never resume the old
|
|
epoch.
|
|
|
|
## Kubernetes Secrets
|
|
|
|
Create Secret **evolution-postgres-migrator** in evolution-dev outside Git. Keys:
|
|
`pg_service.conf`, `pgpass`, `ca.crt`. The service file has section
|
|
`[evolution-migrator]` with host, numeric hostaddr, port, dbname, user,
|
|
sslmode=verify-full, sslrootcert=/run/postgres/ca.crt and connect_timeout=5.
|
|
pgpass uses libpq's escaped host:port:database:user:password format. Do not paste
|
|
values into chat, manifests or image arguments. Configure cluster Secret RBAC and
|
|
encryption at rest through the existing platform policy.
|
|
|
|
The example Job mounts the Secret read-only; the entrypoint copies pgpass into a
|
|
private 0600 temporary file because libpq rejects group-readable password files.
|
|
The copy is removed on exit. Use a distinct **evolution-postgres-runtime** Secret
|
|
with a restricted login for the later region pod; never mount migrator credentials
|
|
into it. Rotation requires a new connection/Job; no hot reload is claimed.
|
|
|
|
`deploy/kubernetes/postgres-migration/job.yaml.example` is intentionally outside the
|
|
active Argo overlay: select a built immutable postgres-tools digest and verify DB
|
|
target/role before adding it. Existing default-deny egress needs a specific DB and
|
|
DNS allowance chosen for the actual endpoint. No live database/Secret was created
|
|
and no World0 resources are touched by this bootstrap.
|
|
|
|
CI builds the `postgres-tools` Docker target and publishes it when Gitea variable
|
|
`HARBOR_POSTGRES_REPOSITORY` is set (for example `aevolution-postgres-tools`). It uses
|
|
the existing HARBOR_PROJECT and Harbor robot secrets. Pin the reported registry
|
|
digest in the example Job; do not use the simulation image. See deploy/README.md.
|
|
Local Unix-socket integration is qualified; remote TLS/authentication, actual Secret
|
|
mounts and container execution still require environment-specific verification.
|