# ADR-0072: Observability, alerting and background delivery for dev and stage ## Status Accepted for M8.5, 2026-09-30. It implements the M8 part of ADR-0015 (observability), and builds on: - ADR-0066 (the operations endpoints); - ADR-0070 (the object reserve); - ADR-0071 (the region host). The maintainer chose an AEvolution-only Prometheus, and left the alert destination for later. ## Context M8.5 asks for: - tick, lease, commit, storage and backup alerts, and bounded logs; - an evaluation of the database outbox before any optional JetStream, with no mandatory Redis or Kafka; - evidence: alert injection, and a telemetry cardinality and cost report. The named risks are broker redelivery changing authority, and logs filling a disk. The cluster has no monitoring stack beyond metrics-server (INVENTORY.md). Each region exposes 32 fixed series. The only labels are a 6-value phase and a 4-value drop reason. ## Decision **Stack.** Namespace `aevolution-monitoring`, with platform-owned guardrails, and a manually synced Argo application `aevolution-monitoring`. - **Components.** One Prometheus and one Alertmanager: - `quay.io/prometheus/prometheus:v3.13.3`, pinned by digest `6976aa8a…`; - `quay.io/prometheus/alertmanager:v0.34.1`, pinned by digest `e9733baf…`. Both are Apache-2.0, and Prometheus is a CNCF graduated project. 3.13 is a maintained line: 3.13.3 of 2026-09-07 carries current security fixes. - **Prometheus configuration:** - Static scrape targets: the `evo-region-ops` Service in dev and stage, labelled with `environment`, `world` and `region`. No service discovery, so no token and no RBAC. - 15 s scrape and evaluation, a 64-sample and 64 KB cap per scrape, and retention of 15 days or 3 GB. - A 5 Gi `local-path` volume. Metrics are not world data, so Delete reclaim is acceptable. - **Network.** The region namespaces admit port 9100 only from Prometheus. Prometheus reaches only the region operations ports and Alertmanager; Alertmanager accepts only Prometheus. Pods run as nobody, with a read-only root filesystem and no capabilities. - **Routing.** Alertmanager routes everything to a **null** receiver until a destination is chosen (docs/BACKLOG.md). Alerts remain visible in its API and UI. **Metrics added.** `evo_region_objects_available_bytes` and `evo_region_objects_reserve_bytes` make volume capacity alertable before new objects are refused. The fixed exposition is now 32 series. **Alerts** (`config/evo-region.rules.yml`, 16 rules). Thresholds come from measured behaviour: | Area | Alerts | |---|---| | Authority | Down (2 min), NotReady (2 min; a forced-deletion recovery takes about 20 s and does not fire), Quarantined, LeaseMargin (below 10 s while ready; healthy is 15–20 s), Fenced | | Progress | StoreFailures, UnknownOutcomes, CommitLag (over 256 ticks), TickDebt and CheckpointStale (only while inputs or lag wait, because an idle region always accrues debt), QueueSaturated, InputsDropped (not counting `refused`) | | Storage | ObjectsLow (below 2× the reserve), ObjectsFull (below the reserve) | | Pipeline | Watchdog (always firing, for a later heartbeat receiver), AlertmanagerDown | **Backup alerts** wait for backups (M8.7). There is nothing to alert on until a backup destination exists; see docs/BACKLOG.md. **Alert injection.** `promtool test rules` checks each alert firing on injected series and staying silent in the healthy, idle and 20 s recovery cases. The CI `deploy` stage runs it with the `promtool` and `amtool` copied from the same pinned images, together with syntax checks of the Prometheus and Alertmanager configurations. **Logs.** Container logs are rotated by the kubelet. The defaults are 10 Mi per file and 5 files per container, 50 Mi at most per container. The actual RKE2 setting has not been verified on the nodes yet. At most: - `evo-server` writes one ack line of about 60 bytes per tick, about 100 MB a day at 20 Hz, so the rotation keeps the most recent 12 hours; - Prometheus and Alertmanager log little. Nothing writes logs to a volume. Durable history lives in PostgreSQL and the object store, never in logs. **Background delivery.** The database outbox stays the delivery mechanism, and no broker is adopted: - `region_outbox` is written in the same transaction as each permanent outcome, delivery is idempotent (`acknowledge_region_outbox`), and delivery never mutates simulation state (ADR-0063). - There is no consumer of outcomes yet: no gateway, client or projection. A relay worker would deliver to nothing, so it is deferred until a consumer exists. - JetStream is not adopted. There is no demonstrated delivery workload, and a broker would add a second durability layer to operate and back up. It could also never hold authority, since its redelivery could only re-announce an already-final outcome. Redis and Kafka are not needed. **Cardinality and cost** (per 15 s scrape): | Source | Series | |---|---| | Each region | 32, plus about 5 per-target scrape series | | Two regions (dev, stage) | about 75 | | Prometheus self-scrape | about 1,000 (typical) | | Alertmanager | about 200 | | **Total** | **about 1,300** | That is about 87 samples per second. At about 1.5 bytes per compressed sample, that is about 11 MB a day, or about 170 MB over 15 days, well inside the 3 GB retention cap and the 5 Gi volume. Each added region adds about 37 series. The request is 100m CPU and 256 Mi, with a 1 Gi memory limit. Alertmanager requests 50m and 64 Mi, with a 128 Mi limit. ## Consequences - Alerts are evaluated and visible, but not delivered until a receiver is chosen. The Watchdog is ready for a dead-man's-switch receiver. - Changing a threshold is a Git change that must keep the rule tests passing. - The kubelet log rotation values should be confirmed on the nodes. - World0 monitoring (M8.8) reuses the rules with its own target. ## Alternatives rejected - **kube-prometheus-stack or the Prometheus Operator.** Cluster-wide CRDs and RBAC for a single project; not the maintainer's choice. - **Kubernetes service discovery.** It needs a token and cluster read RBAC for no gain with two static targets. - **JetStream, Kafka or Redis for outcomes.** There is no workload yet, and each would add another stateful system.