Foundation / checks (deploy) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 1m50s
Foundation / checks (network) (push) Successful in 9m8s
Foundation / checks (client) (push) Successful in 12m10s
Foundation / postgres-tools (push) Canceled after 0s
Foundation / server-image (push) Canceled after 0s
Foundation / image (push) Canceled after 0s
Foundation / checks (postgres) (push) Canceled after 7m33s
Foundation / checks (checks) (push) Canceled after 18m49s
Foundation / checks (tsan) (push) Canceled after 6m38s
Migration v10 keeps, per world and account, the highest accepted grant serial and the region session in control with a fencing generation and database-time expiry, reachable only through claim/renew/release functions under the caller's region lease. Serials survive restarts and apply across regions; a cross-region takeover returns a wait so control never overlaps. RegionSessions takes durable control via a ControlLedger before admitting a session, delays Welcome by the wait, renews, and closes sessions whose control is fenced or cannot be confirmed in time. PostgresControlLedger implements the ledger. ADR-0091. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
268 lines
15 KiB
Markdown
268 lines
15 KiB
Markdown
# PostgreSQL bootstrap (M7.2) and region fencing (M7.3)
|
|
|
|
ADR-0059 pins server/libpq 18.6. `evo::postgres` is optional, Linux-only and separate
|
|
from all simulation targets. Preparation explicitly downloads a verified source
|
|
archive; ordinary CMake configure does not fetch anything. PostgreSQL license is
|
|
permissive; source COPYRIGHT and system TLS license information must accompany
|
|
distributed tooling. No custom hashing/cryptography is introduced.
|
|
|
|
## Local isolated verification
|
|
|
|
Linux prerequisites: GCC14, CMake>=3.28, Ninja, make, bison, flex, curl, bzip2,
|
|
OpenSSL development headers and zlib development headers. Optional clang19 tools
|
|
support analysis/ASan. Run as a normal user, not root:
|
|
|
|
```sh
|
|
bash tools/prepare-postgres.sh
|
|
cmake --preset release -B build/postgres -DCMAKE_CXX_COMPILER=g++-14 \
|
|
-DEVO_BUILD_POSTGRES=ON -DPostgreSQL_ROOT="$PWD/.cache/postgres/install"
|
|
cmake --build build/postgres --target evo-postgres-tests --parallel 2
|
|
bash tools/ci/postgres-check.sh "$PWD/build/postgres/evo-postgres-tests"
|
|
```
|
|
|
|
The test script overrides all connection coordinates and creates a new private
|
|
0700 /tmp directory, private Unix socket and disposable local cluster. It creates
|
|
the `evo_runtime_test` group and the `evo_runtime_login` member, then runs fencing
|
|
tests as that real runtime login. TCP is off;
|
|
local trust applies only inside that private directory. Cluster stops on exit;
|
|
logs/data are retained there for diagnosis. It never uses production PG credentials
|
|
or deletes a pre-existing database. Do not point this script at a shared server.
|
|
|
|
## Adapter contract
|
|
|
|
One connection belongs to one I/O owner; one outstanding parameterized statement.
|
|
No hidden threads, pool, transaction replay or automatic reconnect. Connections
|
|
and queries take 1..30000ms deadlines on a steady clock. Remote hostaddr must be
|
|
numeric, host supplies TLS identity, CA file is explicit, TLS verify-full required.
|
|
Local sockets require no PGHOSTADDR/PGSERVICE override. GSS negotiation is disabled.
|
|
Query SQL <=16KiB, <=16 parameters of <=4KiB, response <=128 rows/16 columns/64KiB
|
|
decoded values; single-row mode prevents accumulating a whole result set. A single
|
|
server field is allocated by libpq before its length can be checked: this is not
|
|
an arbitrary-query memory sandbox. Only trusted bounded persistence SQL is admitted.
|
|
Server statement/lock timeouts default to 10s/5s; callers choose shorter I/O deadlines.
|
|
|
|
Admission config/bounds errors send nothing. SQL errors include only SQLSTATE and
|
|
require caller rollback when inside a transaction. Receive bounds, network errors
|
|
and deadlines close the connection and report unknown: a mutation may have
|
|
committed. Resolve durable request IDs at M7.5; never treat disconnect as rollback.
|
|
Allocation exceptions propagate to the host boundary; do not catch and reuse an
|
|
incomplete session. No PostgreSQL diagnostic strings or connection values are logged.
|
|
|
|
## Migrations and privilege boundary
|
|
|
|
`tools/migrate-postgres.sh` uses psql environment/service files, no URI arguments.
|
|
It requires exactly server 18.6 and serializes migrations with a transaction
|
|
advisory lock. It hashes each checked-in SQL file and applies everything in one
|
|
transaction. The recorded history must be an exact prefix: clean->v1->v2, v1->v2,
|
|
or a v2 no-op. Unknown versions, checksum changes and partial foreign schemas fail
|
|
closed. There is no down migration or destructive repair. The u64 domain rejects
|
|
fractional, negative and above-u64 values without signed truncation.
|
|
|
|
Use a dedicated owner/migrator login with CREATE on the database. Before
|
|
migrating, operators create a NOLOGIN runtime group role and pass its name as
|
|
`EVO_PG_RUNTIME_ROLE`. The runtime login is a separate member of that group only.
|
|
Every migrate run, including a no-op, audits effective privileges of the group and
|
|
every role transitively in it, and rolls back on any violation:
|
|
- privileged attributes or owner identity;
|
|
- membership in any other role, whether inherited or SET-able (for example
|
|
`pg_write_all_data`);
|
|
- any table privilege in `evolution`, including via PUBLIC;
|
|
- CREATE on the schema, the database or `public`;
|
|
- EXECUTE outside the twelve fenced functions (schema v8 adds the build-aware
|
|
`acquire_region_lease`).
|
|
Attribute checks alone are not enough: a login with `pg_write_all_data` can update
|
|
lease rows directly. Rerun the migrate Job after adding or changing runtime logins
|
|
or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and
|
|
default function execution. Operators own database/role provisioning; the migration
|
|
never creates logins or passwords, and never grants superuser.
|
|
|
|
## Fenced region ownership (schema v2, ADR-0060)
|
|
|
|
In schema v3 the runtime role has schema USAGE and EXECUTE on six SECURITY DEFINER
|
|
functions: `acquire_region_lease`, `renew_region_lease`, `release_region_lease`,
|
|
`load_region_head`, `publish_region_checkpoint` and `load_region_checkpoint`. It
|
|
cannot read or write tables, create objects, call internal check/parse functions or
|
|
provision regions. (Schema v2's `advance_region_head` is dropped in v3.) Each mutation locks the lease row
|
|
and checks holder, epoch and expiry with `clock_timestamp()` after the lock wait.
|
|
Clients never send times. Rejections: EVL01 busy, EVL02 fenced, EVL03 head mismatch,
|
|
EVL04 not provisioned, EVL05 epoch/sequence exhausted, EVL06 unsupported
|
|
profile/version/build or a changed rules/quantum, EVL07 reused commit ID, and 22023
|
|
malformed arguments or manifest. `evo::postgres` maps them to `LeaseStoreError`; a
|
|
timeout or broken connection is `unknown`.
|
|
|
|
## Checkpoint publication (schema v3, ADR-0061)
|
|
|
|
Each commit row stores the exact 280-byte manifest, and a constraint ties the
|
|
commit digest to `sha256(manifest)`. The head has a composite foreign key to its
|
|
commit, so even the owner cannot silently repoint or edit it.
|
|
|
|
Payloads are immutable files under a trusted object root:
|
|
`<root>/w<world>/r<region>/<sha256 hex>`, mode 0700 directories and 0600 files.
|
|
Mount a dedicated volume that only the region host's user can write. Nothing is
|
|
ever deleted; orphans from failed transactions are expected.
|
|
|
|
Operator steps:
|
|
1. Record each promoted build (image digest, 32 bytes) before a host uses it:
|
|
`INSERT INTO evolution.retained_builds(build_identity) VALUES ('\x<64 hex>');`
|
|
2. Provision genesis only through `provision_environmental_region` on an owner
|
|
connection. It uploads and verifies the tick-0 payload, then calls the
|
|
owner-only `provision_region_checkpoint(manifest)`.
|
|
|
|
Schema v4 (ADR-0062) additionally admits profile 2 (`environmental-lineage-v1`,
|
|
payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
|
|
profile, history, RNG and durability declarations. Provision history-capable
|
|
regions with `provision_lineage_region`; hosts use `publish/recover_lineage_checkpoint`.
|
|
|
|
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
|
|
- `region_decisions` and `region_outbox`, with one terminal outcome per actor;
|
|
- `resolve_region_commit`, `load_region_decisions`, `pending_region_outbox` and
|
|
`acknowledge_region_outbox`, bringing the runtime function set to ten.
|
|
Triggers make commits and decisions immutable (EVL09); outbox rows may only gain
|
|
a first delivery time. Owners can disable triggers, so treat that as an audited
|
|
break-glass action. EVL08 means an outcome was already final. After an
|
|
ambiguous publication, resolve the exact prepared bytes before doing anything
|
|
else. Upgrading a v3/v4 database that has published checkpoints (not just
|
|
genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions,
|
|
or wait for an explicit legacy-history conversion.
|
|
|
|
Schema v7 (ADR-0067) adds clones:
|
|
- `region_origins`, immutable (EVL09);
|
|
- the owner-only `provision_region_clone`;
|
|
- clone-genesis parsing, and a head constraint that allows sequence 0 at the
|
|
cloned tick.
|
|
The runtime function set is unchanged.
|
|
|
|
Clone with `evo-world-clone clone`:
|
|
- Use a **separate target database** with its own runtime group and login, and
|
|
migrate it with that group as `EVO_PG_RUNTIME_ROLE`.
|
|
- Retain the source's build identity in the target before cloning.
|
|
- The source connection is the source runtime login (read functions only). It is
|
|
switched to read-only transactions.
|
|
- The target connection is the target owner.
|
|
- The tool refuses the same database or the same user on both sides.
|
|
|
|
A clone world never receives original regions, and an original world never
|
|
receives clones. `evo-world-clone audit` verifies a region's whole retained
|
|
history; it is read-only and needs only runtime read access. Never point a clone
|
|
target at a production database or World0.
|
|
|
|
Schema v8 (ADR-0068) binds authority to builds:
|
|
- Hosts from M7.9b onwards acquire with their build identity. Only a retained
|
|
build that has not been retired may hold a lease.
|
|
- No new commit may name a retired build (EVL06). History written by a retired
|
|
build stays recoverable, so keep it retained.
|
|
- Hosts built before M7.9b use build-less acquisition, which keeps working until
|
|
the first barrier.
|
|
|
|
To upgrade the build:
|
|
1. Migrate the schema. Running hosts keep their leases.
|
|
2. Insert the new build into `retained_builds`.
|
|
3. Deploy hosts with `--build <new> --retained <old>`.
|
|
4. On an owner connection, run
|
|
`SELECT evolution.upgrade_barrier('\x<old build>');`
|
|
It retires the old build and fences every current holder in the database. It
|
|
returns the number of leases fenced, and refuses to retire the last current
|
|
build.
|
|
|
|
Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The
|
|
barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions,
|
|
lease checks, provisioning and the retired-build trigger take it shared, and
|
|
provisioning also takes a per-world lock (71420003). A barrier therefore briefly
|
|
waits for in-flight commits and blocks new ones. Keep other advisory locks away
|
|
from these two keys.
|
|
|
|
Old images then quarantine (exit 4) instead of serving. Rules and format changes
|
|
are refused explicitly; there is no converter.
|
|
|
|
There is no operator CLI for provisioning original regions yet, and no real world
|
|
has been provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads
|
|
have no payload.
|
|
|
|
## Account control (schema v10, ADR-0091)
|
|
|
|
`evolution.account_controls` holds, per world and account, the highest accepted grant
|
|
serial and the region session in control (generation, region epoch, database-time
|
|
expiry). Runtime access is only through `claim_account_control`,
|
|
`renew_account_control` and `release_account_control`, each under the caller's exact
|
|
unexpired region lease. A claim refuses serials not above the stored one (`EVL11`),
|
|
returns a wait when another region still holds control, and bumps the generation;
|
|
renewal requires the same generation, region and epoch and an unexpired control
|
|
(`EVL12` otherwise). The privilege audit includes all three functions. Lock order: upgrade
|
|
lock (shared), region lease row, account control row.
|
|
|
|
## Host lease discipline
|
|
|
|
Hosts use `LeaseAuthority` with `LeasePolicy` defaults: 20 s lease, 5 s renewal
|
|
interval, 2 s safety margin, 2 s renewal deadline and 5 s commit deadline. The
|
|
local deadline runs from request send time on CLOCK_BOOTTIME. If a renewal outcome
|
|
is unknown, keep the earlier deadline and reconnect. Once fenced, stop admission,
|
|
simulation and output, then acquire a new epoch and recover. Never resume the old
|
|
epoch.
|
|
|
|
## Kubernetes Secrets
|
|
|
|
Create Secret **evolution-postgres-migrator** in evolution-dev outside Git. Keys:
|
|
`pg_service.conf`, `pgpass`, `ca.crt`. The service file has section
|
|
`[evolution]` (the same section name in every login Secret, M8.3) with host, numeric hostaddr, port, dbname, user,
|
|
sslmode=verify-full, sslrootcert=/run/postgres/ca.crt and connect_timeout=5.
|
|
pgpass uses libpq's escaped host:port:database:user:password format. Do not paste
|
|
values into chat, manifests or image arguments. Configure cluster Secret RBAC and
|
|
encryption at rest through the existing platform policy.
|
|
|
|
The example Job mounts the Secret read-only; the entrypoint copies pgpass into a
|
|
private 0600 temporary file because libpq rejects group-readable password files.
|
|
The copy is removed on exit. Use a distinct **evolution-postgres-runtime** Secret
|
|
with a restricted login for the later region pod; never mount migrator credentials
|
|
into it. Rotation requires a new connection/Job; no hot reload is claimed.
|
|
|
|
`deploy/kubernetes/postgres-migration/job.yaml.example` is intentionally outside the
|
|
active Argo overlay: select a built immutable postgres-tools digest and verify DB
|
|
target/role before adding it. Existing default-deny egress needs a specific DB and
|
|
DNS allowance chosen for the actual endpoint. No live database/Secret was created
|
|
and no World0 resources are touched by this bootstrap.
|
|
|
|
CI builds the `postgres-tools` Docker target and publishes it when Gitea variable
|
|
`HARBOR_POSTGRES_REPOSITORY` is set (for example `aevolution-postgres-tools`). It uses
|
|
the existing HARBOR_PROJECT and Harbor robot secrets. Pin the reported registry
|
|
digest in the example Job; do not use the simulation image. See deploy/README.md.
|
|
Local Unix-socket integration is qualified; remote TLS/authentication, actual Secret
|
|
mounts and container execution still require environment-specific verification.
|
|
|
|
## Region server process (M7.6, ADR-0064)
|
|
|
|
`evo-server` is built with the PostgreSQL build and hosts one provisioned region
|
|
as the runtime login. It is a development host with no network gateway.
|
|
Tick inputs are stdin lines:
|
|
|
|
input <request:32 hex> <temperature_mK> <source_enabled> <bite0..4> <terminal0..1>
|
|
|
|
Events are stdout lines: `phase`, `ready`, `ack`, `outcome`, `dropped` and a
|
|
final `stats`. Only `ack` and `outcome` lines are durable facts.
|
|
`--ready-file` mirrors readiness for probes. SIGTERM drains (commits pending ticks
|
|
and releases the lease) within `--drain-ms`.
|
|
|
|
Exit codes:
|
|
- 0: drained;
|
|
- 3: the drain deadline passed, and the next owner reconciles;
|
|
- 4: quarantined;
|
|
- 2: usage error.
|
|
|
|
`--crash-at <point>:<n>` (before-upload, before-transaction, after-commit or
|
|
after-ack) is a test-only fault injector that SIGKILLs the process at that
|
|
boundary; never set it in shared environments. Use the default 20 s lease in any
|
|
shared environment. `--lease-profile fast` exists
|
|
only for local tests. The runtime credentials come from `--pg-*` flags and a
|
|
passfile; never place passwords on the command line.
|
|
|
|
### Operations endpoints (M7.8, ADR-0066)
|
|
|
|
`--ops-listen <numeric address:port>` serves three endpoints:
|
|
- `GET /livez`: the process loop is responding. It says nothing about authority.
|
|
- `GET /readyz`: 200 only while this process holds the lease and has a verified
|
|
recovered head; 503 with the phase otherwise.
|
|
- `GET /metrics`: Prometheus text.
|
|
|
|
A probe may wait up to one bounded I/O step (the 5 s commit deadline), so probe
|
|
timeouts must exceed that. Expose port 9100 only to the monitoring namespace;
|
|
the endpoints are unauthenticated plain HTTP.
|