The adapter bounded each parameter to 4 KiB of text (ADR-0059), and a cellular publication sends its replay trace as one hex bytea parameter, so a 64-tick batch or a one-tick batch of about 85 lives was refused client-side as "invalid". CI run 372 hit this in the M11.4c-1 publication test. ADR-0126 raises the bound to 4 MiB, enough for the largest trace the codec admits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
17 KiB
PostgreSQL bootstrap (M7.2) and region fencing (M7.3)
ADR-0059 pins server/libpq 18.6. evo::postgres is optional, Linux-only and separate
from all simulation targets. Preparation explicitly downloads a verified source
archive; ordinary CMake configure does not fetch anything. PostgreSQL license is
permissive; source COPYRIGHT and system TLS license information must accompany
distributed tooling. No custom hashing/cryptography is introduced.
Local isolated verification
Linux prerequisites: GCC14, CMake>=3.28, Ninja, make, bison, flex, curl, bzip2, OpenSSL development headers and zlib development headers. Optional clang19 tools support analysis/ASan. Run as a normal user, not root:
bash tools/prepare-postgres.sh
cmake --preset release -B build/postgres -DCMAKE_CXX_COMPILER=g++-14 \
-DEVO_BUILD_POSTGRES=ON -DPostgreSQL_ROOT="$PWD/.cache/postgres/install"
cmake --build build/postgres --target evo-postgres-tests --parallel 2
bash tools/ci/postgres-check.sh "$PWD/build/postgres/evo-postgres-tests"
The test script overrides all connection coordinates and creates a new private
0700 /tmp directory, private Unix socket and disposable local cluster. It creates
the evo_runtime_test group and the evo_runtime_login member, then runs fencing
tests as that real runtime login. TCP is off;
local trust applies only inside that private directory. Cluster stops on exit;
logs/data are retained there for diagnosis. It never uses production PG credentials
or deletes a pre-existing database. Do not point this script at a shared server.
Adapter contract
One connection belongs to one I/O owner; one outstanding parameterized statement. No hidden threads, pool, transaction replay or automatic reconnect. Connections and queries take 1..30000ms deadlines on a steady clock. Remote hostaddr must be numeric, host supplies TLS identity, CA file is explicit, TLS verify-full required. Local sockets require no PGHOSTADDR/PGSERVICE override. GSS negotiation is disabled. Query SQL <=16KiB, <=16 parameters of <=4MiB (ADR-0126; 4KiB before cellular traces), response <=128 rows/16 columns/64KiB decoded values; single-row mode prevents accumulating a whole result set. A single server field is allocated by libpq before its length can be checked: this is not an arbitrary-query memory sandbox. Only trusted bounded persistence SQL is admitted. Server statement/lock timeouts default to 10s/5s; callers choose shorter I/O deadlines.
Admission config/bounds errors send nothing. SQL errors include only SQLSTATE and require caller rollback when inside a transaction. Receive bounds, network errors and deadlines close the connection and report unknown: a mutation may have committed. Resolve durable request IDs at M7.5; never treat disconnect as rollback. Allocation exceptions propagate to the host boundary; do not catch and reuse an incomplete session. No PostgreSQL diagnostic strings or connection values are logged.
Migrations and privilege boundary
tools/migrate-postgres.sh uses psql environment/service files, no URI arguments.
It requires exactly server 18.6 and serializes migrations with a transaction
advisory lock. It hashes each checked-in SQL file and applies everything in one
transaction. The recorded history must be an exact prefix: clean->v1->v2, v1->v2,
or a v2 no-op. Unknown versions, checksum changes and partial foreign schemas fail
closed. There is no down migration or destructive repair. The u64 domain rejects
fractional, negative and above-u64 values without signed truncation.
Use a dedicated owner/migrator login with CREATE on the database. Before
migrating, operators create a NOLOGIN runtime group role and pass its name as
EVO_PG_RUNTIME_ROLE. The runtime login is a separate member of that group only.
Every migrate run, including a no-op, audits effective privileges of the group and
every role transitively in it, and rolls back on any violation:
- privileged attributes or owner identity;
- membership in any other role, whether inherited or SET-able (for example
pg_write_all_data); - any table privilege in
evolution, including via PUBLIC; - CREATE on the schema, the database or
public; - EXECUTE outside the twelve fenced functions (schema v8 adds the build-aware
acquire_region_lease). Attribute checks alone are not enough: a login withpg_write_all_datacan update lease rows directly. Rerun the migrate Job after adding or changing runtime logins or grants; it is the audit. Grants made later without a rerun are not detected. Bootstrap revokes PUBLIC access to the schema, tables and default function execution. Operators own database/role provisioning; the migration never creates logins or passwords, and never grants superuser.
Fenced region ownership (schema v2, ADR-0060)
In schema v3 the runtime role has schema USAGE and EXECUTE on six SECURITY DEFINER
functions: acquire_region_lease, renew_region_lease, release_region_lease,
load_region_head, publish_region_checkpoint and load_region_checkpoint. It
cannot read or write tables, create objects, call internal check/parse functions or
provision regions. (Schema v2's advance_region_head is dropped in v3.) Each mutation locks the lease row
and checks holder, epoch and expiry with clock_timestamp() after the lock wait.
Clients never send times. Rejections: EVL01 busy, EVL02 fenced, EVL03 head mismatch,
EVL04 not provisioned, EVL05 epoch/sequence exhausted, EVL06 unsupported
profile/version/build or a changed rules/quantum, EVL07 reused commit ID, and 22023
malformed arguments or manifest. evo::postgres maps them to LeaseStoreError; a
timeout or broken connection is unknown.
Checkpoint publication (schema v3, ADR-0061)
Each commit row stores the exact 280-byte manifest, and a constraint ties the
commit digest to sha256(manifest). The head has a composite foreign key to its
commit, so even the owner cannot silently repoint or edit it.
Payloads are immutable files under a trusted object root:
<root>/w<world>/r<region>/<sha256 hex>, mode 0700 directories and 0600 files.
Mount a dedicated volume that only the region host's user can write. Nothing is
ever deleted; orphans from failed transactions are expected.
Operator steps:
- Record each promoted build (image digest, 32 bytes) before a host uses it:
INSERT INTO evolution.retained_builds(build_identity) VALUES ('\x<64 hex>'); - Provision genesis only through
provision_environmental_regionon an owner connection. It uploads and verifies the tick-0 payload, then calls the owner-onlyprovision_region_checkpoint(manifest).
Schema v4 (ADR-0062) additionally admits profile 2 (environmental-lineage-v1,
payload ≤ 4,268,144 bytes) and requires every commit to keep its predecessor's
profile, history, RNG and durability declarations. Provision history-capable
regions with provision_lineage_region; hosts use publish/recover_lineage_checkpoint.
Schema v5 (ADR-0063) requires manifest v2 with a decision block and adds:
region_decisionsandregion_outbox, with one terminal outcome per actor;resolve_region_commit,load_region_decisions,pending_region_outboxandacknowledge_region_outbox, bringing the runtime function set to ten. Triggers make commits and decisions immutable (EVL09); outbox rows may only gain a first delivery time. Owners can disable triggers, so treat that as an audited break-glass action. EVL08 means an outcome was already final. After an ambiguous publication, resolve the exact prepared bytes before doing anything else. Upgrading a v3/v4 database that has published checkpoints (not just genesis) is refused with EVL10 and leaves it unchanged: re-provision those regions, or wait for an explicit legacy-history conversion.
Schema v7 (ADR-0067) adds clones:
region_origins, immutable (EVL09);- the owner-only
provision_region_clone; - clone-genesis parsing, and a head constraint that allows sequence 0 at the cloned tick. The runtime function set is unchanged.
Clone with evo-world-clone clone:
- Use a separate target database with its own runtime group and login, and
migrate it with that group as
EVO_PG_RUNTIME_ROLE. - Retain the source's build identity in the target before cloning.
- The source connection is the source runtime login (read functions only). It is switched to read-only transactions.
- The target connection is the target owner.
- The tool refuses the same database or the same user on both sides.
A clone world never receives original regions, and an original world never
receives clones. evo-world-clone audit verifies a region's whole retained
history; it is read-only and needs only runtime read access. Never point a clone
target at a production database or World0.
Schema v8 (ADR-0068) binds authority to builds:
- Hosts from M7.9b onwards acquire with their build identity. Only a retained build that has not been retired may hold a lease.
- No new commit may name a retired build (EVL06). History written by a retired build stays recoverable, so keep it retained.
- Hosts built before M7.9b use build-less acquisition, which keeps working until the first barrier.
To upgrade the build:
- Migrate the schema. Running hosts keep their leases.
- Insert the new build into
retained_builds. - Deploy hosts with
--build <new> --retained <old>. - On an owner connection, run
SELECT evolution.upgrade_barrier('\x<old build>');It retires the old build and fences every current holder in the database. It returns the number of leases fenced, and refuses to retire the last current build.
Schema v9 serializes all of this under one lock order (ADR-0068 amendment). The barrier takes a database-wide advisory lock (71420002) exclusively. Acquisitions, lease checks, provisioning and the retired-build trigger take it shared, and provisioning also takes a per-world lock (71420003). A barrier therefore briefly waits for in-flight commits and blocks new ones. Keep other advisory locks away from these two keys.
Old images then quarantine (exit 4) instead of serving. Rules and format changes are refused explicitly; there is no converter.
There is no operator CLI for provisioning original regions yet, and no real world has been provisioned. Migrating v2→v3 fails closed if any v2 head exists, because v2 heads have no payload.
Account control (schema v10, ADR-0091)
evolution.account_controls holds, per world and account, the highest accepted grant
serial and the region session in control (generation, region epoch, database-time
expiry). Runtime access is only through claim_account_control,
renew_account_control and release_account_control, each under the caller's exact
unexpired region lease. A claim refuses serials not above the stored one (EVL11),
returns a wait when another region still holds control, and bumps the generation;
renewal requires the same generation, region and epoch and an unexpired control
(EVL12 otherwise). The privilege audit includes all three functions. Lock order: upgrade
lock (shared), region lease row, account control row.
Action records (schema v11, ADR-0092)
publish_region_checkpoint(manifest, decisions, actions) records a batch's action block
(region-actions-v1) in the same statement as its
checkpoint: one evolution.region_actions row per new (world, account, action id), and the
block in the append-only region_commit_actions (committed history itself is never
updated). Exact retries and resolve_region_commit(..., actions) compare the block.
find_region_action reads one record under the caller's exact region lease. The privilege
audit covers all three functions; the tables and check_action_block are owner-only.
Probe records and handoff (schema v12, ADR-0096)
EXPERIMENT (non-biological probes, ADR-0094). evolution.probe_records holds one row per
(world, account): owning region, generation, position, test inventory and the open
transfer while frozen. evolution.probe_transfers holds the transfer state machine:
prepared, staged, committed, aborted. Runtime access is only through open_probe,
save_probe, prepare_probe_transfer, stage_probe_transfer, commit_probe_transfer,
abort_probe_transfer, find_probe_transfer and handoff_account_control, each under the
caller's exact unexpired region lease. Every transfer call can be retried with its id and
returns the recorded outcome. SQLSTATEs:
EVL13: probe fenced;EVL14: unknown transfer;EVL15: not staged by this epoch.
Lock order: upgrade lock (shared), own region lease row, probe record, transfer;
handoff_account_control reads the transfer, then locks the account control row. The
privilege audit covers all eight functions, and the tables are owner-only. Schema v13 adds
incoming_probe_transfers, a read-only function under the lease that lists the open
transfers addressed to the caller's region (for ghosts). Schema v14 adds
overdue_probe_transfers: read-only, under the lease, it lists the caller's open outgoing
transfers prepared more than the given age (1 s to 1 day) ago by the database clock. A
restarted source uses it to abort transfers whose client never arrived (the dead process's
deadlines were in memory). Deploying v12-v14 needs a rebuilt migrations (tools) image.
Host lease discipline
Hosts use LeaseAuthority with LeasePolicy defaults: 20 s lease, 5 s renewal
interval, 2 s safety margin, 2 s renewal deadline and 5 s commit deadline. The
local deadline runs from request send time on CLOCK_BOOTTIME. If a renewal outcome
is unknown, keep the earlier deadline and reconnect. Once fenced, stop admission,
simulation and output, then acquire a new epoch and recover. Never resume the old
epoch.
Kubernetes Secrets
Create Secret evolution-postgres-migrator in evolution-dev outside Git. Keys:
pg_service.conf, pgpass, ca.crt. The service file has section
[evolution] (the same section name in every login Secret, M8.3) with host, numeric hostaddr, port, dbname, user,
sslmode=verify-full, sslrootcert=/run/postgres/ca.crt and connect_timeout=5.
pgpass uses libpq's escaped host:port:database:user:password format. Do not paste
values into chat, manifests or image arguments. Configure cluster Secret RBAC and
encryption at rest through the existing platform policy.
The example Job mounts the Secret read-only; the entrypoint copies pgpass into a private 0600 temporary file because libpq rejects group-readable password files. The copy is removed on exit. Use a distinct evolution-postgres-runtime Secret with a restricted login for the later region pod; never mount migrator credentials into it. Rotation requires a new connection/Job; no hot reload is claimed.
deploy/kubernetes/postgres-migration/job.yaml.example is intentionally outside the
active Argo overlay: select a built immutable postgres-tools digest and verify DB
target/role before adding it. Existing default-deny egress needs a specific DB and
DNS allowance chosen for the actual endpoint. No live database/Secret was created
and no World0 resources are touched by this bootstrap.
CI builds the postgres-tools Docker target and publishes it when Gitea variable
HARBOR_POSTGRES_REPOSITORY is set (for example aevolution-postgres-tools). It uses
the existing HARBOR_PROJECT and Harbor robot secrets. Pin the reported registry
digest in the example Job; do not use the simulation image. See deploy/README.md.
Local Unix-socket integration is qualified; remote TLS/authentication, actual Secret
mounts and container execution still require environment-specific verification.
Region server process (M7.6, ADR-0064)
evo-server is built with the PostgreSQL build and hosts one provisioned region
as the runtime login. It is a development host with no network gateway.
Tick inputs are stdin lines:
input <request:32 hex> <temperature_mK> <source_enabled> <bite0..4> <terminal0..1>
Events are stdout lines: phase, ready, ack, outcome, dropped and a
final stats. Only ack and outcome lines are durable facts.
--ready-file mirrors readiness for probes. SIGTERM drains (commits pending ticks
and releases the lease) within --drain-ms.
With EVO_BUILD_QUIC, --m10-listen ADDRESS:PORT --m10-cert F --m10-key F --m10-issuer F --m10-environment dev|stage|world0 opens the M10 session listener (ADR-0091/0092;
development and test only, port 0 picks an ephemeral port printed as m10 listening).
Session control and action records use a separate runtime connection under the host's
lease. Events: m10 opened|closed|refused, m10 committed, m10 dropped.
evo-session-probe is the matching headless client (dev issuer keys and grants only).
Exit codes:
- 0: drained;
- 3: the drain deadline passed, and the next owner reconciles;
- 4: quarantined;
- 2: usage error.
--crash-at <point>:<n> (before-upload, before-transaction, after-commit or
after-ack) is a test-only fault injector that SIGKILLs the process at that
boundary; never set it in shared environments. Use the default 20 s lease in any
shared environment. --lease-profile fast exists
only for local tests. The runtime credentials come from --pg-* flags and a
passfile; never place passwords on the command line.
Operations endpoints (M7.8, ADR-0066)
--ops-listen <numeric address:port> serves three endpoints:
GET /livez: the process loop is responding. It says nothing about authority.GET /readyz: 200 only while this process holds the lease and has a verified recovered head; 503 with the phase otherwise.GET /metrics: Prometheus text.
A probe may wait up to one bounded I/O step (the 5 s commit deadline), so probe timeouts must exceed that. Expose port 9100 only to the monitoring namespace; the endpoints are unauthenticated plain HTTP.