Files
aevolution/docs/adr/0066-operations-endpoints.md
T
dingo 432bae6e2a
Foundation / checks (deploy) (push) Successful in 2m51s
Foundation / checks (ecs) (push) Successful in 1m52s
Foundation / checks (client) (push) Successful in 12m46s
Foundation / checks (network) (push) Successful in 9m44s
Foundation / checks (tsan) (push) Successful in 7m51s
Foundation / checks (checks) (push) Successful in 31m29s
Foundation / checks (postgres) (push) Successful in 39m28s
Foundation / postgres-tools (push) Successful in 10s
Foundation / image (push) Successful in 2m17s
Foundation / server-image (push) Successful in 3m20s
Bundle MsQuic in server runtime image
2026-10-04 19:13:10 -04:00

6.0 KiB

ADR-0066: Operations endpoints and development server image

Status

Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015 (observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is still no gateway, alerting rules or production promotion.

Context

M7.8 needs:

  • separate liveness and readiness;
  • metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
  • a non-root image with no embedded secrets.

The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids unbounded or entity labels, and allows a tiny direct exposition with format tests. The constitution forbids hidden threads and unreviewed dependencies.

Decision

Endpoints (evo-server --ops-listen <numeric address:port>):

  • /livez returns 200 whenever the host loop answers. It says nothing about authority, and is used only for restarts.
  • /readyz returns 200 only when HostStatus::ready holds: running, recovered and tail-verified, not draining, and LeaseAuthority::may_originate. Otherwise it returns 503 with the phase name. It is used for traffic, and later gateway routing must additionally honor the lease epoch.
  • /metrics returns Prometheus text 0.0.4.

Responder (OpsServer, Linux, no dependency):

  • Numeric bind only. GET only (405 otherwise), exact paths (404), malformed requests 400, requests over 4 KiB 431.
  • One request per connection, Connection: close, Cache-Control: no-store.
  • At most 8 connections; any excess is closed on accept, never queued.
  • Each connection has a 1 s deadline, and responses are capped at 16 KiB.
  • serve() never blocks and runs on the host's own loop, so answers describe current state and no thread is added. A complete request that waited behind one bounded host step is still answered.

Metrics (evo::ops, portable and unit-tested):

Group Series
Authority ready; one-hot phase (6 values); lease epoch; lease remaining (ms)
Progress simulated and committed tick and sequence; commit lag; checkpoint age (ms)
Queue tick debt, current and max; queued inputs and bytes; their capacities
Counters commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons)

That is 30 series with fixed names. The only labels are phase and reason, each from a fixed enum. There are no world, region, entity or request labels: the scrape target carries region identity. The exposition is about 3.5 KB and at most 8 KiB. Commit lag saturates at zero rather than wrapping.

Image (server-runtime Docker target) contains evo-server, evo-world-clone, the pinned libpq, and the matching pinned MsQuic shared library when built with the QUIC listener enabled. It installs only the runtime libraries it needs (libstdc++, libssl3, zlib, CA certificates):

  • It runs as UID/GID 10001 with a read-only root filesystem and all capabilities dropped. It exposes port 9100 and has no default configuration.
  • With missing settings it exits 2 with usage; there is no default region, database or build.
  • CI builds it and smoke-tests it: the usage exit code under restrictions, non-root identity, resolved libraries, and no credential files. Publishing requires HARBOR_SERVER_REPOSITORY.

Deployment example (deploy/kubernetes/region-server, not in Argo):

  • One replica with the Recreate strategy.
  • The runtime Secret is copied to a private 0600 file by an init container.
  • An object PVC; startup, readiness and liveness probes with 8 s timeouts, longer than one bounded I/O step.
  • A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.

Alternatives Considered

  • prometheus-cpp or OpenTelemetry: add dependencies and threads for about 30 fixed series.
  • An HTTP library: a dependency for three GET paths.
  • A separate probe thread: would answer while the host is stuck in I/O, which hides a blocked loop and adds shared state.
  • Readiness from process health or phase alone: exactly the named risk.

Benefits

  • Probes and metrics cannot claim authority the lease does not grant.
  • Label cardinality is fixed by construction.
  • The image carries nothing environment-specific.

Costs

  • Probes can wait for one bounded host step (up to the 5 s commit deadline), so probe timeouts must exceed it.
  • The responder is intentionally minimal (no keep-alive, TLS or auth).
  • The metrics endpoint must be network-restricted.

Risks

  • Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the control.
  • The build-tree RUNPATH stays in the binary; it is harmless in the image, where the path does not exist, and LD_LIBRARY_PATH names libpq.
  • Local Docker was not run in this gate; the CI job is configured but its first execution is pending.

Consequences

  • M7.9 measures recovery and batching latency through these metrics.
  • A later gateway must honor readiness and the lease epoch before routing.
  • Promoting to World0 needs a scanned, signed image and a reviewed alerting policy (M8).

Amendment: bounded accept work (2026-09-29 review)

serve() used to accept, and close excess connections, until the backlog was empty. A replenished stream of connections could therefore hold the host loop without bound. Each call now makes at most max_accepts_per_serve (4) accept attempts, refused ones included. Pending connections wait in the 16-entry listen backlog, and deadlines are judged after the accept work.

Tests check the per-call budget, including at the connection cap, and that an accepted probe is still answered under replenished pressure. A real server keeps its lease and readiness while a client opens and closes connections continuously through 60 ticks.

Amendment: object volume capacity (2026-09-30, M8.5)

Two gauges were added, evo_region_objects_available_bytes and evo_region_objects_reserve_bytes, so that ADR-0070's free-space reserve can be alerted on before new objects are refused (ADR-0072). The fixed exposition is now 32 series. It still has only the phase and reason labels, and stays under the 8 KiB bound.