Files
aevolution/docs/adr/0066-operations-endpoints.md
T
2026-10-08 12:08:53 -04:00

123 lines
6.0 KiB
Markdown

[Reviewed]
# ADR-0066: Operations endpoints and development server image
## Status
Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015
(observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is
still no gateway, alerting rules or production promotion.
## Context
M7.8 needs:
- separate liveness and readiness;
- metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
- a non-root image with no embedded secrets.
The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids
unbounded or entity labels, and allows a tiny direct exposition with format tests.
The constitution forbids hidden threads and unreviewed dependencies.
## Decision
**Endpoints** (`evo-server --ops-listen <numeric address:port>`):
- `/livez` returns 200 whenever the host loop answers. It says nothing about
authority, and is used only for restarts.
- `/readyz` returns 200 only when `HostStatus::ready` holds: running, recovered and
tail-verified, not draining, and `LeaseAuthority::may_originate`. Otherwise it
returns 503 with the phase name. It is used for traffic, and later gateway
routing must additionally honor the lease epoch.
- `/metrics` returns Prometheus text 0.0.4.
**Responder** (`OpsServer`, Linux, no dependency):
- Numeric bind only. GET only (405 otherwise), exact paths (404), malformed
requests 400, requests over 4 KiB 431.
- One request per connection, `Connection: close`, `Cache-Control: no-store`.
- At most 8 connections; any excess is closed on accept, never queued.
- Each connection has a 1 s deadline, and responses are capped at 16 KiB.
- `serve()` never blocks and runs on the host's own loop, so answers describe
current state and no thread is added. A complete request that waited behind one
bounded host step is still answered.
**Metrics** (`evo::ops`, portable and unit-tested):
| Group | Series |
|---|---|
| Authority | ready; one-hot phase (6 values); lease epoch; lease remaining (ms) |
| Progress | simulated and committed tick and sequence; commit lag; checkpoint age (ms) |
| Queue | tick debt, current and max; queued inputs and bytes; their capacities |
| Counters | commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons) |
That is 30 series with fixed names. The only labels are `phase` and `reason`,
each from a fixed enum. There are no world, region, entity or request labels:
the scrape target carries region identity. The exposition is about 3.5 KB and at
most 8 KiB. Commit lag saturates at zero rather than wrapping.
**Image** (`server-runtime` Docker target) contains `evo-server`,
`evo-world-clone`, the pinned libpq, and the matching pinned MsQuic shared library
when built with the QUIC listener enabled. It installs only the runtime libraries
it needs (libstdc++, libssl3, zlib, CA certificates):
- It runs as UID/GID 10001 with a read-only root filesystem and all capabilities
dropped. It exposes port 9100 and has no default configuration.
- With missing settings it exits 2 with usage; there is no default region,
database or build.
- CI builds it and smoke-tests it: the usage exit code under restrictions,
non-root identity, resolved libraries, and no credential files. Publishing
requires `HARBOR_SERVER_REPOSITORY`.
**Deployment example** (`deploy/kubernetes/region-server`, not in Argo):
- One replica with the Recreate strategy.
- The runtime Secret is copied to a private 0600 file by an init container.
- An object PVC; startup, readiness and liveness probes with 8 s timeouts,
longer than one bounded I/O step.
- A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.
## Alternatives Considered
- **prometheus-cpp or OpenTelemetry**: add dependencies and threads for about 30
fixed series.
- **An HTTP library**: a dependency for three GET paths.
- **A separate probe thread**: would answer while the host is stuck in I/O, which
hides a blocked loop and adds shared state.
- **Readiness from process health or phase alone**: exactly the named risk.
## Benefits
- Probes and metrics cannot claim authority the lease does not grant.
- Label cardinality is fixed by construction.
- The image carries nothing environment-specific.
## Costs
- Probes can wait for one bounded host step (up to the 5 s commit deadline), so
probe timeouts must exceed it.
- The responder is intentionally minimal (no keep-alive, TLS or auth).
- The metrics endpoint must be network-restricted.
## Risks
- Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the
control.
- The build-tree RUNPATH stays in the binary; it is harmless in the image, where
the path does not exist, and `LD_LIBRARY_PATH` names libpq.
- Local Docker was not run in this gate; the CI job is configured but its first
execution is pending.
## Consequences
- M7.9 measures recovery and batching latency through these metrics.
- A later gateway must honor readiness and the lease epoch before routing.
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
(M8).
## Amendment: bounded accept work (2026-09-29 review)
`serve()` used to accept, and close excess connections, until the backlog was
empty. A replenished stream of connections could therefore hold the host loop
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
attempts, refused ones included. Pending connections wait in the 16-entry listen
backlog, and deadlines are judged after the accept work.
Tests check the per-call budget, including at the connection cap, and that an
accepted probe is still answered under replenished pressure. A real server keeps
its lease and readiness while a client opens and closes connections continuously
through 60 ticks.
## Amendment: object volume capacity (2026-09-30, M8.5)
Two gauges were added, `evo_region_objects_available_bytes` and
`evo_region_objects_reserve_bytes`, so that ADR-0070's free-space reserve can be
alerted on before new objects are refused (ADR-0072). The fixed exposition is now
32 series. It still has only the `phase` and `reason` labels, and stays under the
8 KiB bound.