Files
aevolution/docs/adr/0066-operations-endpoints.md
T
dingoandClaude Opus 5.5 b4493b3d0f Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-30 06:16:36 -04:00

121 lines
5.9 KiB
Markdown

# ADR-0066: Operations endpoints and development server image
## Status
Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015
(observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is
still no gateway, alerting rules or production promotion.
## Context
M7.8 needs:
- separate liveness and readiness;
- metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
- a non-root image with no embedded secrets.
The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids
unbounded or entity labels, and allows a tiny direct exposition with format tests.
The constitution forbids hidden threads and unreviewed dependencies.
## Decision
**Endpoints** (`evo-server --ops-listen <numeric address:port>`):
- `/livez` returns 200 whenever the host loop answers. It says nothing about
authority, and is used only for restarts.
- `/readyz` returns 200 only when `HostStatus::ready` holds: running, recovered and
tail-verified, not draining, and `LeaseAuthority::may_originate`. Otherwise it
returns 503 with the phase name. It is used for traffic, and later gateway
routing must additionally honor the lease epoch.
- `/metrics` returns Prometheus text 0.0.4.
**Responder** (`OpsServer`, Linux, no dependency):
- Numeric bind only. GET only (405 otherwise), exact paths (404), malformed
requests 400, requests over 4 KiB 431.
- One request per connection, `Connection: close`, `Cache-Control: no-store`.
- At most 8 connections; any excess is closed on accept, never queued.
- Each connection has a 1 s deadline, and responses are capped at 16 KiB.
- `serve()` never blocks and runs on the host's own loop, so answers describe
current state and no thread is added. A complete request that waited behind one
bounded host step is still answered.
**Metrics** (`evo::ops`, portable and unit-tested):
| Group | Series |
|---|---|
| Authority | ready; one-hot phase (6 values); lease epoch; lease remaining (ms) |
| Progress | simulated and committed tick and sequence; commit lag; checkpoint age (ms) |
| Queue | tick debt, current and max; queued inputs and bytes; their capacities |
| Counters | commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons) |
That is 30 series with fixed names. The only labels are `phase` and `reason`,
each from a fixed enum. There are no world, region, entity or request labels:
the scrape target carries region identity. The exposition is about 3.5 KB and at
most 8 KiB. Commit lag saturates at zero rather than wrapping.
**Image** (`server-runtime` Docker target) contains `evo-server` and the pinned
libpq, and installs only the runtime libraries it needs (libstdc++, libssl3,
zlib, CA certificates):
- It runs as UID/GID 10001 with a read-only root filesystem and all capabilities
dropped. It exposes port 9100 and has no default configuration.
- With missing settings it exits 2 with usage; there is no default region,
database or build.
- CI builds it and smoke-tests it: the usage exit code under restrictions,
non-root identity, resolved libraries, and no credential files. Publishing
requires `HARBOR_SERVER_REPOSITORY`.
**Deployment example** (`deploy/kubernetes/region-server`, not in Argo):
- One replica with the Recreate strategy.
- The runtime Secret is copied to a private 0600 file by an init container.
- An object PVC; startup, readiness and liveness probes with 8 s timeouts,
longer than one bounded I/O step.
- A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.
## Alternatives Considered
- **prometheus-cpp or OpenTelemetry**: add dependencies and threads for about 30
fixed series.
- **An HTTP library**: a dependency for three GET paths.
- **A separate probe thread**: would answer while the host is stuck in I/O, which
hides a blocked loop and adds shared state.
- **Readiness from process health or phase alone**: exactly the named risk.
## Benefits
- Probes and metrics cannot claim authority the lease does not grant.
- Label cardinality is fixed by construction.
- The image carries nothing environment-specific.
## Costs
- Probes can wait for one bounded host step (up to the 5 s commit deadline), so
probe timeouts must exceed it.
- The responder is intentionally minimal (no keep-alive, TLS or auth).
- The metrics endpoint must be network-restricted.
## Risks
- Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the
control.
- The build-tree RUNPATH stays in the binary; it is harmless in the image, where
the path does not exist, and `LD_LIBRARY_PATH` names libpq.
- Local Docker was not run in this gate; the CI job is configured but its first
execution is pending.
## Consequences
- M7.9 measures recovery and batching latency through these metrics.
- A later gateway must honor readiness and the lease epoch before routing.
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
(M8).
## Amendment: bounded accept work (2026-09-29 review)
`serve()` used to accept, and close excess connections, until the backlog was
empty. A replenished stream of connections could therefore hold the host loop
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
attempts, refused ones included. Pending connections wait in the 16-entry listen
backlog, and deadlines are judged after the accept work.
Tests check the per-call budget, including at the connection cap, and that an
accepted probe is still answered under replenished pressure. A real server keeps
its lease and readiness while a client opens and closes connections continuously
through 60 ticks.
## Amendment: object volume capacity (2026-09-30, M8.5)
Two gauges were added, `evo_region_objects_available_bytes` and
`evo_region_objects_reserve_bytes`, so that ADR-0070's free-space reserve can be
alerted on before new objects are refused (ADR-0072). The fixed exposition is now
32 series. It still has only the `phase` and `reason` labels, and stays under the
8 KiB bound.