6.0 KiB
ADR-0066: Operations endpoints and development server image
Status
Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015 (observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is still no gateway, alerting rules or production promotion.
Context
M7.8 needs:
- separate liveness and readiness;
- metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
- a non-root image with no embedded secrets.
The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids unbounded or entity labels, and allows a tiny direct exposition with format tests. The constitution forbids hidden threads and unreviewed dependencies.
Decision
Endpoints (evo-server --ops-listen <numeric address:port>):
/livezreturns 200 whenever the host loop answers. It says nothing about authority, and is used only for restarts./readyzreturns 200 only whenHostStatus::readyholds: running, recovered and tail-verified, not draining, andLeaseAuthority::may_originate. Otherwise it returns 503 with the phase name. It is used for traffic, and later gateway routing must additionally honor the lease epoch./metricsreturns Prometheus text 0.0.4.
Responder (OpsServer, Linux, no dependency):
- Numeric bind only. GET only (405 otherwise), exact paths (404), malformed requests 400, requests over 4 KiB 431.
- One request per connection,
Connection: close,Cache-Control: no-store. - At most 8 connections; any excess is closed on accept, never queued.
- Each connection has a 1 s deadline, and responses are capped at 16 KiB.
serve()never blocks and runs on the host's own loop, so answers describe current state and no thread is added. A complete request that waited behind one bounded host step is still answered.
Metrics (evo::ops, portable and unit-tested):
| Group | Series |
|---|---|
| Authority | ready; one-hot phase (6 values); lease epoch; lease remaining (ms) |
| Progress | simulated and committed tick and sequence; commit lag; checkpoint age (ms) |
| Queue | tick debt, current and max; queued inputs and bytes; their capacities |
| Counters | commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons) |
That is 30 series with fixed names. The only labels are phase and reason,
each from a fixed enum. There are no world, region, entity or request labels:
the scrape target carries region identity. The exposition is about 3.5 KB and at
most 8 KiB. Commit lag saturates at zero rather than wrapping.
Image (server-runtime Docker target) contains evo-server,
evo-world-clone, the pinned libpq, and the matching pinned MsQuic shared library
when built with the QUIC listener enabled. It installs only the runtime libraries
it needs (libstdc++, libssl3, zlib, CA certificates):
- It runs as UID/GID 10001 with a read-only root filesystem and all capabilities dropped. It exposes port 9100 and has no default configuration.
- With missing settings it exits 2 with usage; there is no default region, database or build.
- CI builds it and smoke-tests it: the usage exit code under restrictions,
non-root identity, resolved libraries, and no credential files. Publishing
requires
HARBOR_SERVER_REPOSITORY.
Deployment example (deploy/kubernetes/region-server, not in Argo):
- One replica with the Recreate strategy.
- The runtime Secret is copied to a private 0600 file by an init container.
- An object PVC; startup, readiness and liveness probes with 8 s timeouts, longer than one bounded I/O step.
- A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.
Alternatives Considered
- prometheus-cpp or OpenTelemetry: add dependencies and threads for about 30 fixed series.
- An HTTP library: a dependency for three GET paths.
- A separate probe thread: would answer while the host is stuck in I/O, which hides a blocked loop and adds shared state.
- Readiness from process health or phase alone: exactly the named risk.
Benefits
- Probes and metrics cannot claim authority the lease does not grant.
- Label cardinality is fixed by construction.
- The image carries nothing environment-specific.
Costs
- Probes can wait for one bounded host step (up to the 5 s commit deadline), so probe timeouts must exceed it.
- The responder is intentionally minimal (no keep-alive, TLS or auth).
- The metrics endpoint must be network-restricted.
Risks
- Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the control.
- The build-tree RUNPATH stays in the binary; it is harmless in the image, where
the path does not exist, and
LD_LIBRARY_PATHnames libpq. - Local Docker was not run in this gate; the CI job is configured but its first execution is pending.
Consequences
- M7.9 measures recovery and batching latency through these metrics.
- A later gateway must honor readiness and the lease epoch before routing.
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy (M8).
Amendment: bounded accept work (2026-09-29 review)
serve() used to accept, and close excess connections, until the backlog was
empty. A replenished stream of connections could therefore hold the host loop
without bound. Each call now makes at most max_accepts_per_serve (4) accept
attempts, refused ones included. Pending connections wait in the 16-entry listen
backlog, and deadlines are judged after the accept work.
Tests check the per-call budget, including at the connection cap, and that an accepted probe is still answered under replenished pressure. A real server keeps its lease and readiness while a client opens and closes connections continuously through 60 ticks.
Amendment: object volume capacity (2026-09-30, M8.5)
Two gauges were added, evo_region_objects_available_bytes and
evo_region_objects_reserve_bytes, so that ADR-0070's free-space reserve can be
alerted on before new objects are refused (ADR-0072). The fixed exposition is now
32 series. It still has only the phase and reason labels, and stays under the
8 KiB bound.