127 lines
6.2 KiB
Markdown
127 lines
6.2 KiB
Markdown
[Reviewed]
|
||
# ADR-0072: Observability, alerting and background delivery for dev and stage
|
||
|
||
## Status
|
||
Accepted for M8.5, 2026-09-30. It implements the M8 part of ADR-0015
|
||
(observability), and builds on:
|
||
- ADR-0066 (the operations endpoints);
|
||
- ADR-0070 (the object reserve);
|
||
- ADR-0071 (the region host).
|
||
|
||
The maintainer chose an AEvolution-only Prometheus, and left the alert destination
|
||
for later.
|
||
|
||
## Context
|
||
M8.5 asks for:
|
||
- tick, lease, commit, storage and backup alerts, and bounded logs;
|
||
- an evaluation of the database outbox before any optional JetStream, with no
|
||
mandatory Redis or Kafka;
|
||
- evidence: alert injection, and a telemetry cardinality and cost report.
|
||
|
||
The named risks are broker redelivery changing authority, and logs filling a disk.
|
||
|
||
The cluster has no monitoring stack beyond metrics-server (INVENTORY.md). Each
|
||
region exposes 32 fixed series. The only labels are a 6-value phase and a 4-value
|
||
drop reason.
|
||
|
||
## Decision
|
||
**Stack.** Namespace `aevolution-monitoring`, with platform-owned guardrails, and
|
||
a manually synced Argo application `aevolution-monitoring`.
|
||
- **Components.** One Prometheus and one Alertmanager:
|
||
- `quay.io/prometheus/prometheus:v3.13.3`, pinned by digest `6976aa8a…`;
|
||
- `quay.io/prometheus/alertmanager:v0.34.1`, pinned by digest `e9733baf…`.
|
||
|
||
Both are Apache-2.0, and Prometheus is a CNCF graduated project. 3.13 is a
|
||
maintained line: 3.13.3 of 2026-09-07 carries current security fixes.
|
||
- **Prometheus configuration:**
|
||
- Static scrape targets: the `evo-region-ops` Service in dev and stage, labelled
|
||
with `environment`, `world` and `region`. No service discovery, so no token
|
||
and no RBAC.
|
||
- 15 s scrape and evaluation, a 64-sample and 64 KB cap per scrape, and
|
||
retention of 15 days or 3 GB.
|
||
- A 5 Gi `local-path` volume. Metrics are not world data, so Delete reclaim is
|
||
acceptable.
|
||
- **Network.** The region namespaces admit port 9100 only from Prometheus.
|
||
Prometheus reaches only the region operations ports and Alertmanager;
|
||
Alertmanager accepts only Prometheus. Pods run as nobody, with a read-only root
|
||
filesystem and no capabilities.
|
||
- **Routing.** Alertmanager routes everything to a **null** receiver until a
|
||
destination is chosen (docs/BACKLOG.md). Alerts remain visible in its API and
|
||
UI.
|
||
|
||
**Metrics added.** `evo_region_objects_available_bytes` and
|
||
`evo_region_objects_reserve_bytes` make volume capacity alertable before new
|
||
objects are refused. The fixed exposition is now 32 series.
|
||
|
||
**Alerts** (`config/evo-region.rules.yml`, 16 rules). Thresholds come from
|
||
measured behaviour:
|
||
|
||
| Area | Alerts |
|
||
|---|---|
|
||
| Authority | Down (2 min), NotReady (2 min; a forced-deletion recovery takes about 20 s and does not fire), Quarantined, LeaseMargin (below 10 s while ready; healthy is 15–20 s), Fenced |
|
||
| Progress | StoreFailures, UnknownOutcomes, CommitLag (over 256 ticks), TickDebt and CheckpointStale (only while inputs or lag wait, because an idle region always accrues debt), QueueSaturated, InputsDropped (not counting `refused`) |
|
||
| Storage | ObjectsLow (below 2× the reserve), ObjectsFull (below the reserve) |
|
||
| Pipeline | Watchdog (always firing, for a later heartbeat receiver), AlertmanagerDown |
|
||
|
||
**Backup alerts** wait for backups (M8.7). There is nothing to alert on until a
|
||
backup destination exists; see docs/BACKLOG.md.
|
||
|
||
**Alert injection.** `promtool test rules` checks each alert firing on injected
|
||
series and staying silent in the healthy, idle and 20 s recovery cases. The CI
|
||
`deploy` stage runs it with the `promtool` and `amtool` copied from the same
|
||
pinned images, together with syntax checks of the Prometheus and Alertmanager
|
||
configurations.
|
||
|
||
**Logs.** Container logs are rotated by the kubelet. The defaults are 10 Mi per
|
||
file and 5 files per container, 50 Mi at most per container. The actual RKE2
|
||
setting has not been verified on the nodes yet. At most:
|
||
- `evo-server` writes one ack line of about 60 bytes per tick, about 100 MB a day
|
||
at 20 Hz, so the rotation keeps the most recent 12 hours;
|
||
- Prometheus and Alertmanager log little.
|
||
|
||
Nothing writes logs to a volume. Durable history lives in PostgreSQL and the
|
||
object store, never in logs.
|
||
|
||
**Background delivery.** The database outbox stays the delivery mechanism, and no
|
||
broker is adopted:
|
||
- `region_outbox` is written in the same transaction as each permanent outcome,
|
||
delivery is idempotent (`acknowledge_region_outbox`), and delivery never
|
||
mutates simulation state (ADR-0063).
|
||
- There is no consumer of outcomes yet: no gateway, client or projection. A relay
|
||
worker would deliver to nothing, so it is deferred until a consumer exists.
|
||
- JetStream is not adopted. There is no demonstrated delivery workload, and a
|
||
broker would add a second durability layer to operate and back up. It could
|
||
also never hold authority, since its redelivery could only re-announce an
|
||
already-final outcome. Redis and Kafka are not needed.
|
||
|
||
**Cardinality and cost** (per 15 s scrape):
|
||
|
||
| Source | Series |
|
||
|---|---|
|
||
| Each region | 32, plus about 5 per-target scrape series |
|
||
| Two regions (dev, stage) | about 75 |
|
||
| Prometheus self-scrape | about 1,000 (typical) |
|
||
| Alertmanager | about 200 |
|
||
| **Total** | **about 1,300** |
|
||
|
||
That is about 87 samples per second. At about 1.5 bytes per compressed sample,
|
||
that is about 11 MB a day, or about 170 MB over 15 days, well inside the 3 GB
|
||
retention cap and the 5 Gi volume. Each added region adds about 37 series. The
|
||
request is 100m CPU and 256 Mi, with a 1 Gi memory limit. Alertmanager requests
|
||
50m and 64 Mi, with a 128 Mi limit.
|
||
|
||
## Consequences
|
||
- Alerts are evaluated and visible, but not delivered until a receiver is chosen.
|
||
The Watchdog is ready for a dead-man's-switch receiver.
|
||
- Changing a threshold is a Git change that must keep the rule tests passing.
|
||
- The kubelet log rotation values should be confirmed on the nodes.
|
||
- World0 monitoring (M8.8) reuses the rules with its own target.
|
||
|
||
## Alternatives rejected
|
||
- **kube-prometheus-stack or the Prometheus Operator.** Cluster-wide CRDs and
|
||
RBAC for a single project; not the maintainer's choice.
|
||
- **Kubernetes service discovery.** It needs a token and cluster read RBAC for no
|
||
gain with two static targets.
|
||
- **JetStream, Kafka or Redis for outcomes.** There is no workload yet, and each
|
||
would add another stateful system.
|