[Reviewed] # ADR-0066: Operations endpoints and development server image ## Status Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015 (observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is still no gateway, alerting rules or production promotion. ## Context M7.8 needs: - separate liveness and readiness; - metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds; - a non-root image with no embedded secrets. The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids unbounded or entity labels, and allows a tiny direct exposition with format tests. The constitution forbids hidden threads and unreviewed dependencies. ## Decision **Endpoints** (`evo-server --ops-listen `): - `/livez` returns 200 whenever the host loop answers. It says nothing about authority, and is used only for restarts. - `/readyz` returns 200 only when `HostStatus::ready` holds: running, recovered and tail-verified, not draining, and `LeaseAuthority::may_originate`. Otherwise it returns 503 with the phase name. It is used for traffic, and later gateway routing must additionally honor the lease epoch. - `/metrics` returns Prometheus text 0.0.4. **Responder** (`OpsServer`, Linux, no dependency): - Numeric bind only. GET only (405 otherwise), exact paths (404), malformed requests 400, requests over 4 KiB 431. - One request per connection, `Connection: close`, `Cache-Control: no-store`. - At most 8 connections; any excess is closed on accept, never queued. - Each connection has a 1 s deadline, and responses are capped at 16 KiB. - `serve()` never blocks and runs on the host's own loop, so answers describe current state and no thread is added. A complete request that waited behind one bounded host step is still answered. **Metrics** (`evo::ops`, portable and unit-tested): | Group | Series | |---|---| | Authority | ready; one-hot phase (6 values); lease epoch; lease remaining (ms) | | Progress | simulated and committed tick and sequence; commit lag; checkpoint age (ms) | | Queue | tick debt, current and max; queued inputs and bytes; their capacities | | Counters | commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons) | That is 30 series with fixed names. The only labels are `phase` and `reason`, each from a fixed enum. There are no world, region, entity or request labels: the scrape target carries region identity. The exposition is about 3.5 KB and at most 8 KiB. Commit lag saturates at zero rather than wrapping. **Image** (`server-runtime` Docker target) contains `evo-server`, `evo-world-clone`, the pinned libpq, and the matching pinned MsQuic shared library when built with the QUIC listener enabled. It installs only the runtime libraries it needs (libstdc++, libssl3, zlib, CA certificates): - It runs as UID/GID 10001 with a read-only root filesystem and all capabilities dropped. It exposes port 9100 and has no default configuration. - With missing settings it exits 2 with usage; there is no default region, database or build. - CI builds it and smoke-tests it: the usage exit code under restrictions, non-root identity, resolved libraries, and no credential files. Publishing requires `HARBOR_SERVER_REPOSITORY`. **Deployment example** (`deploy/kubernetes/region-server`, not in Argo): - One replica with the Recreate strategy. - The runtime Secret is copied to a private 0600 file by an init container. - An object PVC; startup, readiness and liveness probes with 8 s timeouts, longer than one bounded I/O step. - A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress. ## Alternatives Considered - **prometheus-cpp or OpenTelemetry**: add dependencies and threads for about 30 fixed series. - **An HTTP library**: a dependency for three GET paths. - **A separate probe thread**: would answer while the host is stuck in I/O, which hides a blocked loop and adds shared state. - **Readiness from process health or phase alone**: exactly the named risk. ## Benefits - Probes and metrics cannot claim authority the lease does not grant. - Label cardinality is fixed by construction. - The image carries nothing environment-specific. ## Costs - Probes can wait for one bounded host step (up to the 5 s commit deadline), so probe timeouts must exceed it. - The responder is intentionally minimal (no keep-alive, TLS or auth). - The metrics endpoint must be network-restricted. ## Risks - Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the control. - The build-tree RUNPATH stays in the binary; it is harmless in the image, where the path does not exist, and `LD_LIBRARY_PATH` names libpq. - Local Docker was not run in this gate; the CI job is configured but its first execution is pending. ## Consequences - M7.9 measures recovery and batching latency through these metrics. - A later gateway must honor readiness and the lease epoch before routing. - Promoting to World0 needs a scanned, signed image and a reviewed alerting policy (M8). ## Amendment: bounded accept work (2026-09-29 review) `serve()` used to accept, and close excess connections, until the backlog was empty. A replenished stream of connections could therefore hold the host loop without bound. Each call now makes at most `max_accepts_per_serve` (4) accept attempts, refused ones included. Pending connections wait in the 16-entry listen backlog, and deadlines are judged after the accept work. Tests check the per-call budget, including at the connection cap, and that an accepted probe is still answered under replenished pressure. A real server keeps its lease and readiness while a client opens and closes connections continuously through 60 ticks. ## Amendment: object volume capacity (2026-09-30, M8.5) Two gauges were added, `evo_region_objects_available_bytes` and `evo_region_objects_reserve_bytes`, so that ADR-0070's free-space reserve can be alerted on before new objects are refused (ADR-0072). The fixed exposition is now 32 series. It still has only the `phase` and `reason` labels, and stays under the 8 KiB bound.