Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by digest and delivered by Argo (ADR-0072), scrape each region's operations Service through static targets (no API access, no RBAC); the region namespaces admit port 9100 from Prometheus only. Sixteen alert rules cover authority, lease, commits, backlog and object capacity and are proven by promtool injection tests in CI with tools copied from the same pinned images. Two new gauges expose object volume free space and the reserve. Alerts route to a null receiver until a destination is chosen; no broker is adopted and the database outbox stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -139,6 +139,27 @@ record:
|
||||
- the first pod's `phase running epoch=... committed=...` line;
|
||||
- that `/readyz` stays 503 while a replacement waits out the lease.
|
||||
|
||||
## Monitoring (M8.5, ADR-0072)
|
||||
|
||||
Bootstrap once: `kubectl apply -f deploy/argocd/monitoring/`, then sync
|
||||
`aevolution-platform` (it adds the `aevolution-monitoring` namespace and its
|
||||
guardrails) and `aevolution-monitoring`. Re-sync dev and stage, which adds the
|
||||
`evo-region-scrape` policy.
|
||||
|
||||
```sh
|
||||
kubectl -n aevolution-monitoring port-forward svc/prometheus 9090 &
|
||||
curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"' # both regions up
|
||||
curl -s localhost:9090/api/v1/alerts # Watchdog firing only
|
||||
```
|
||||
|
||||
**Injection in the cluster** (dev only):
|
||||
- `kubectl -n evolution-dev scale deploy/evo-region --replicas=0` makes
|
||||
`EvoRegionDown` pending, then firing after 2 minutes, in Alertmanager (null
|
||||
receiver).
|
||||
- Scale back to 1 and it resolves.
|
||||
|
||||
Record the timestamps. The rule tests in CI already prove the thresholds.
|
||||
|
||||
## Break-glass
|
||||
|
||||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||||
|
||||
Reference in New Issue
Block a user