Files
aevolution/docs/adr/0072-observability.md
2026-10-08 12:08:53 -04:00

127 lines
6.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
[Reviewed]
# ADR-0072: Observability, alerting and background delivery for dev and stage
## Status
Accepted for M8.5, 2026-09-30. It implements the M8 part of ADR-0015
(observability), and builds on:
- ADR-0066 (the operations endpoints);
- ADR-0070 (the object reserve);
- ADR-0071 (the region host).
The maintainer chose an AEvolution-only Prometheus, and left the alert destination
for later.
## Context
M8.5 asks for:
- tick, lease, commit, storage and backup alerts, and bounded logs;
- an evaluation of the database outbox before any optional JetStream, with no
mandatory Redis or Kafka;
- evidence: alert injection, and a telemetry cardinality and cost report.
The named risks are broker redelivery changing authority, and logs filling a disk.
The cluster has no monitoring stack beyond metrics-server (INVENTORY.md). Each
region exposes 32 fixed series. The only labels are a 6-value phase and a 4-value
drop reason.
## Decision
**Stack.** Namespace `aevolution-monitoring`, with platform-owned guardrails, and
a manually synced Argo application `aevolution-monitoring`.
- **Components.** One Prometheus and one Alertmanager:
- `quay.io/prometheus/prometheus:v3.13.3`, pinned by digest `6976aa8a…`;
- `quay.io/prometheus/alertmanager:v0.34.1`, pinned by digest `e9733baf…`.
Both are Apache-2.0, and Prometheus is a CNCF graduated project. 3.13 is a
maintained line: 3.13.3 of 2026-09-07 carries current security fixes.
- **Prometheus configuration:**
- Static scrape targets: the `evo-region-ops` Service in dev and stage, labelled
with `environment`, `world` and `region`. No service discovery, so no token
and no RBAC.
- 15 s scrape and evaluation, a 64-sample and 64 KB cap per scrape, and
retention of 15 days or 3 GB.
- A 5 Gi `local-path` volume. Metrics are not world data, so Delete reclaim is
acceptable.
- **Network.** The region namespaces admit port 9100 only from Prometheus.
Prometheus reaches only the region operations ports and Alertmanager;
Alertmanager accepts only Prometheus. Pods run as nobody, with a read-only root
filesystem and no capabilities.
- **Routing.** Alertmanager routes everything to a **null** receiver until a
destination is chosen (docs/BACKLOG.md). Alerts remain visible in its API and
UI.
**Metrics added.** `evo_region_objects_available_bytes` and
`evo_region_objects_reserve_bytes` make volume capacity alertable before new
objects are refused. The fixed exposition is now 32 series.
**Alerts** (`config/evo-region.rules.yml`, 16 rules). Thresholds come from
measured behaviour:
| Area | Alerts |
|---|---|
| Authority | Down (2 min), NotReady (2 min; a forced-deletion recovery takes about 20 s and does not fire), Quarantined, LeaseMargin (below 10 s while ready; healthy is 15–20 s), Fenced |
| Progress | StoreFailures, UnknownOutcomes, CommitLag (over 256 ticks), TickDebt and CheckpointStale (only while inputs or lag wait, because an idle region always accrues debt), QueueSaturated, InputsDropped (not counting `refused`) |
| Storage | ObjectsLow (below 2× the reserve), ObjectsFull (below the reserve) |
| Pipeline | Watchdog (always firing, for a later heartbeat receiver), AlertmanagerDown |
**Backup alerts** wait for backups (M8.7). There is nothing to alert on until a
backup destination exists; see docs/BACKLOG.md.
**Alert injection.** `promtool test rules` checks each alert firing on injected
series and staying silent in the healthy, idle and 20 s recovery cases. The CI
`deploy` stage runs it with the `promtool` and `amtool` copied from the same
pinned images, together with syntax checks of the Prometheus and Alertmanager
configurations.
**Logs.** Container logs are rotated by the kubelet. The defaults are 10 Mi per
file and 5 files per container, 50 Mi at most per container. The actual RKE2
setting has not been verified on the nodes yet. At most:
- `evo-server` writes one ack line of about 60 bytes per tick, about 100 MB a day
at 20 Hz, so the rotation keeps the most recent 12 hours;
- Prometheus and Alertmanager log little.
Nothing writes logs to a volume. Durable history lives in PostgreSQL and the
object store, never in logs.
**Background delivery.** The database outbox stays the delivery mechanism, and no
broker is adopted:
- `region_outbox` is written in the same transaction as each permanent outcome,
delivery is idempotent (`acknowledge_region_outbox`), and delivery never
mutates simulation state (ADR-0063).
- There is no consumer of outcomes yet: no gateway, client or projection. A relay
worker would deliver to nothing, so it is deferred until a consumer exists.
- JetStream is not adopted. There is no demonstrated delivery workload, and a
broker would add a second durability layer to operate and back up. It could
also never hold authority, since its redelivery could only re-announce an
already-final outcome. Redis and Kafka are not needed.
**Cardinality and cost** (per 15 s scrape):
| Source | Series |
|---|---|
| Each region | 32, plus about 5 per-target scrape series |
| Two regions (dev, stage) | about 75 |
| Prometheus self-scrape | about 1,000 (typical) |
| Alertmanager | about 200 |
| **Total** | **about 1,300** |
That is about 87 samples per second. At about 1.5 bytes per compressed sample,
that is about 11 MB a day, or about 170 MB over 15 days, well inside the 3 GB
retention cap and the 5 Gi volume. Each added region adds about 37 series. The
request is 100m CPU and 256 Mi, with a 1 Gi memory limit. Alertmanager requests
50m and 64 Mi, with a 128 Mi limit.
## Consequences
- Alerts are evaluated and visible, but not delivered until a receiver is chosen.
The Watchdog is ready for a dead-man's-switch receiver.
- Changing a threshold is a Git change that must keep the rule tests passing.
- The kubelet log rotation values should be confirmed on the nodes.
- World0 monitoring (M8.8) reuses the rules with its own target.
## Alternatives rejected
- **kube-prometheus-stack or the Prometheus Operator.** Cluster-wide CRDs and
RBAC for a single project; not the maintainer's choice.
- **Kubernetes service discovery.** It needs a token and cluster read RBAC for no
gain with two static targets.
- **JetStream, Kafka or Redis for outcomes.** There is no workload yet, and each
would add another stateful system.