Files
aevolution/docs/adr/0066-operations-endpoints.md
T
dingoandClaude Opus 5.5 b4493b3d0f Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 06:16:36 -04:00

5.9 KiB

ADR-0066: Operations endpoints and development server image

Status

Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015 (observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is still no gateway, alerting rules or production promotion.

Context

M7.8 needs:

  • separate liveness and readiness;
  • metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
  • a non-root image with no embedded secrets.

The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids unbounded or entity labels, and allows a tiny direct exposition with format tests. The constitution forbids hidden threads and unreviewed dependencies.

Decision

Endpoints (evo-server --ops-listen <numeric address:port>):

  • /livez returns 200 whenever the host loop answers. It says nothing about authority, and is used only for restarts.
  • /readyz returns 200 only when HostStatus::ready holds: running, recovered and tail-verified, not draining, and LeaseAuthority::may_originate. Otherwise it returns 503 with the phase name. It is used for traffic, and later gateway routing must additionally honor the lease epoch.
  • /metrics returns Prometheus text 0.0.4.

Responder (OpsServer, Linux, no dependency):

  • Numeric bind only. GET only (405 otherwise), exact paths (404), malformed requests 400, requests over 4 KiB 431.
  • One request per connection, Connection: close, Cache-Control: no-store.
  • At most 8 connections; any excess is closed on accept, never queued.
  • Each connection has a 1 s deadline, and responses are capped at 16 KiB.
  • serve() never blocks and runs on the host's own loop, so answers describe current state and no thread is added. A complete request that waited behind one bounded host step is still answered.

Metrics (evo::ops, portable and unit-tested):

Group Series
Authority ready; one-hot phase (6 values); lease epoch; lease remaining (ms)
Progress simulated and committed tick and sequence; commit lag; checkpoint age (ms)
Queue tick debt, current and max; queued inputs and bytes; their capacities
Counters commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons)

That is 30 series with fixed names. The only labels are phase and reason, each from a fixed enum. There are no world, region, entity or request labels: the scrape target carries region identity. The exposition is about 3.5 KB and at most 8 KiB. Commit lag saturates at zero rather than wrapping.

Image (server-runtime Docker target) contains evo-server and the pinned libpq, and installs only the runtime libraries it needs (libstdc++, libssl3, zlib, CA certificates):

  • It runs as UID/GID 10001 with a read-only root filesystem and all capabilities dropped. It exposes port 9100 and has no default configuration.
  • With missing settings it exits 2 with usage; there is no default region, database or build.
  • CI builds it and smoke-tests it: the usage exit code under restrictions, non-root identity, resolved libraries, and no credential files. Publishing requires HARBOR_SERVER_REPOSITORY.

Deployment example (deploy/kubernetes/region-server, not in Argo):

  • One replica with the Recreate strategy.
  • The runtime Secret is copied to a private 0600 file by an init container.
  • An object PVC; startup, readiness and liveness probes with 8 s timeouts, longer than one bounded I/O step.
  • A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.

Alternatives Considered

  • prometheus-cpp or OpenTelemetry: add dependencies and threads for about 30 fixed series.
  • An HTTP library: a dependency for three GET paths.
  • A separate probe thread: would answer while the host is stuck in I/O, which hides a blocked loop and adds shared state.
  • Readiness from process health or phase alone: exactly the named risk.

Benefits

  • Probes and metrics cannot claim authority the lease does not grant.
  • Label cardinality is fixed by construction.
  • The image carries nothing environment-specific.

Costs

  • Probes can wait for one bounded host step (up to the 5 s commit deadline), so probe timeouts must exceed it.
  • The responder is intentionally minimal (no keep-alive, TLS or auth).
  • The metrics endpoint must be network-restricted.

Risks

  • Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the control.
  • The build-tree RUNPATH stays in the binary; it is harmless in the image, where the path does not exist, and LD_LIBRARY_PATH names libpq.
  • Local Docker was not run in this gate; the CI job is configured but its first execution is pending.

Consequences

  • M7.9 measures recovery and batching latency through these metrics.
  • A later gateway must honor readiness and the lease epoch before routing.
  • Promoting to World0 needs a scanned, signed image and a reviewed alerting policy (M8).

Amendment: bounded accept work (2026-09-29 review)

serve() used to accept, and close excess connections, until the backlog was empty. A replenished stream of connections could therefore hold the host loop without bound. Each call now makes at most max_accepts_per_serve (4) accept attempts, refused ones included. Pending connections wait in the 16-entry listen backlog, and deadlines are judged after the accept work.

Tests check the per-call budget, including at the connection cap, and that an accepted probe is still answered under replenished pressure. A real server keeps its lease and readiness while a client opens and closes connections continuously through 60 ticks.

Amendment: object volume capacity (2026-09-30, M8.5)

Two gauges were added, evo_region_objects_available_bytes and evo_region_objects_reserve_bytes, so that ADR-0070's free-space reserve can be alerted on before new objects are refused (ADR-0072). The fixed exposition is now 32 series. It still has only the phase and reason labels, and stays under the 8 KiB bound.