123 lines
6.0 KiB
Markdown
123 lines
6.0 KiB
Markdown
[Reviewed]
|
|
# ADR-0066: Operations endpoints and development server image
|
|
|
|
## Status
|
|
Accepted for M7.8, 2026-09-28. It implements the M7 part of ADR-0015
|
|
(observability). It builds on ADR-0064 (host) and ADR-0065 (recovery). There is
|
|
still no gateway, alerting rules or production promotion.
|
|
|
|
## Context
|
|
M7.8 needs:
|
|
- separate liveness and readiness;
|
|
- metrics for tick debt, commit lag, lease safety, checkpoint age and queue bounds;
|
|
- a non-root image with no embedded secrets.
|
|
|
|
The named risk is a healthy process mistaken for valid authority. ADR-0015 forbids
|
|
unbounded or entity labels, and allows a tiny direct exposition with format tests.
|
|
The constitution forbids hidden threads and unreviewed dependencies.
|
|
|
|
## Decision
|
|
**Endpoints** (`evo-server --ops-listen <numeric address:port>`):
|
|
- `/livez` returns 200 whenever the host loop answers. It says nothing about
|
|
authority, and is used only for restarts.
|
|
- `/readyz` returns 200 only when `HostStatus::ready` holds: running, recovered and
|
|
tail-verified, not draining, and `LeaseAuthority::may_originate`. Otherwise it
|
|
returns 503 with the phase name. It is used for traffic, and later gateway
|
|
routing must additionally honor the lease epoch.
|
|
- `/metrics` returns Prometheus text 0.0.4.
|
|
|
|
**Responder** (`OpsServer`, Linux, no dependency):
|
|
- Numeric bind only. GET only (405 otherwise), exact paths (404), malformed
|
|
requests 400, requests over 4 KiB 431.
|
|
- One request per connection, `Connection: close`, `Cache-Control: no-store`.
|
|
- At most 8 connections; any excess is closed on accept, never queued.
|
|
- Each connection has a 1 s deadline, and responses are capped at 16 KiB.
|
|
- `serve()` never blocks and runs on the host's own loop, so answers describe
|
|
current state and no thread is added. A complete request that waited behind one
|
|
bounded host step is still answered.
|
|
|
|
**Metrics** (`evo::ops`, portable and unit-tested):
|
|
|
|
| Group | Series |
|
|
|---|---|
|
|
| Authority | ready; one-hot phase (6 values); lease epoch; lease remaining (ms) |
|
|
| Progress | simulated and committed tick and sequence; commit lag; checkpoint age (ms) |
|
|
| Queue | tick debt, current and max; queued inputs and bytes; their capacities |
|
|
| Counters | commits; store failures; unknown outcomes; fences; acknowledgements; outcomes; dropped (4 reasons) |
|
|
|
|
That is 30 series with fixed names. The only labels are `phase` and `reason`,
|
|
each from a fixed enum. There are no world, region, entity or request labels:
|
|
the scrape target carries region identity. The exposition is about 3.5 KB and at
|
|
most 8 KiB. Commit lag saturates at zero rather than wrapping.
|
|
|
|
**Image** (`server-runtime` Docker target) contains `evo-server`,
|
|
`evo-world-clone`, the pinned libpq, and the matching pinned MsQuic shared library
|
|
when built with the QUIC listener enabled. It installs only the runtime libraries
|
|
it needs (libstdc++, libssl3, zlib, CA certificates):
|
|
- It runs as UID/GID 10001 with a read-only root filesystem and all capabilities
|
|
dropped. It exposes port 9100 and has no default configuration.
|
|
- With missing settings it exits 2 with usage; there is no default region,
|
|
database or build.
|
|
- CI builds it and smoke-tests it: the usage exit code under restrictions,
|
|
non-root identity, resolved libraries, and no credential files. Publishing
|
|
requires `HARBOR_SERVER_REPOSITORY`.
|
|
|
|
**Deployment example** (`deploy/kubernetes/region-server`, not in Argo):
|
|
- One replica with the Recreate strategy.
|
|
- The runtime Secret is copied to a private 0600 file by an init container.
|
|
- An object PVC; startup, readiness and liveness probes with 8 s timeouts,
|
|
longer than one bounded I/O step.
|
|
- A NetworkPolicy allowing only monitoring ingress on 9100 and the database egress.
|
|
|
|
## Alternatives Considered
|
|
- **prometheus-cpp or OpenTelemetry**: add dependencies and threads for about 30
|
|
fixed series.
|
|
- **An HTTP library**: a dependency for three GET paths.
|
|
- **A separate probe thread**: would answer while the host is stuck in I/O, which
|
|
hides a blocked loop and adds shared state.
|
|
- **Readiness from process health or phase alone**: exactly the named risk.
|
|
|
|
## Benefits
|
|
- Probes and metrics cannot claim authority the lease does not grant.
|
|
- Label cardinality is fixed by construction.
|
|
- The image carries nothing environment-specific.
|
|
|
|
## Costs
|
|
- Probes can wait for one bounded host step (up to the 5 s commit deadline), so
|
|
probe timeouts must exceed it.
|
|
- The responder is intentionally minimal (no keep-alive, TLS or auth).
|
|
- The metrics endpoint must be network-restricted.
|
|
|
|
## Risks
|
|
- Metrics and probes are unauthenticated plain HTTP; the NetworkPolicy is the
|
|
control.
|
|
- The build-tree RUNPATH stays in the binary; it is harmless in the image, where
|
|
the path does not exist, and `LD_LIBRARY_PATH` names libpq.
|
|
- Local Docker was not run in this gate; the CI job is configured but its first
|
|
execution is pending.
|
|
|
|
## Consequences
|
|
- M7.9 measures recovery and batching latency through these metrics.
|
|
- A later gateway must honor readiness and the lease epoch before routing.
|
|
- Promoting to World0 needs a scanned, signed image and a reviewed alerting policy
|
|
(M8).
|
|
|
|
## Amendment: bounded accept work (2026-09-29 review)
|
|
`serve()` used to accept, and close excess connections, until the backlog was
|
|
empty. A replenished stream of connections could therefore hold the host loop
|
|
without bound. Each call now makes at most `max_accepts_per_serve` (4) accept
|
|
attempts, refused ones included. Pending connections wait in the 16-entry listen
|
|
backlog, and deadlines are judged after the accept work.
|
|
|
|
Tests check the per-call budget, including at the connection cap, and that an
|
|
accepted probe is still answered under replenished pressure. A real server keeps
|
|
its lease and readiness while a client opens and closes connections continuously
|
|
through 60 ticks.
|
|
|
|
## Amendment: object volume capacity (2026-09-30, M8.5)
|
|
Two gauges were added, `evo_region_objects_available_bytes` and
|
|
`evo_region_objects_reserve_bytes`, so that ADR-0070's free-space reserve can be
|
|
alerted on before new objects are refused (ADR-0072). The fixed exposition is now
|
|
32 series. It still has only the `phase` and `reason` labels, and stays under the
|
|
8 KiB bound.
|