Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics

An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by
digest and delivered by Argo (ADR-0072), scrape each region's operations
Service through static targets (no API access, no RBAC); the region
namespaces admit port 9100 from Prometheus only. Sixteen alert rules
cover authority, lease, commits, backlog and object capacity and are
proven by promtool injection tests in CI with tools copied from the same
pinned images. Two new gauges expose object volume free space and the
reserve. Alerts route to a null receiver until a destination is chosen;
no broker is adopted and the database outbox stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 06:16:36 -04:00
co-authored by Claude Opus 5.5
parent 2c79d4748c
commit b4493b3d0f
33 changed files with 1068 additions and 8 deletions
+33
View File
@@ -2811,3 +2811,36 @@ admission policy and the stage and World0 AppProjects, then ran
- `region-drill.sh` passed `bash -n`, and refuses World0 before kubectl (also
enforced by the policy check).
- Not executed: the in-cluster region sync and the drill.
## M8.5 observability - 2026-09-30
- `promtool` 3.13.3 and `amtool` 0.34.1 came from the official release tarballs,
with SHA-256 verified against the published sums: `b349c732…` and `265b9d1e…`.
- The Prometheus configuration syntax is valid, and all 16 rules are valid.
- The alert injection tests passed, as did the Alertmanager `check-config`.
- Rule mutations: five were tried, and three were caught at first.
- **NotReady firing after 15 s:** missed, because the forced-deletion test only
evaluated after recovery. The test now also checks during the dip.
- **ObjectsFull weakened to half the reserve:** missed, because the injected
free space was below both thresholds. The test now uses 900 MB.
- After these fixes all five were caught: the lease threshold, NotReady too
eager, debt counted without inputs, a weakened ObjectsFull, and `refused`
drops counted.
- Windows MSVC CTest 59/59, including the `ops-contracts` update to 32 series.
- WSL Linux `checks` set: 59/59 in each configuration. The format check passed.
- PostgreSQL suites: Release, analysis and ASan/UBSan passed. The ops process
test checks the reserve gauge (1 GiB) and that free space is nonzero.
- `deploy-policy-check.py`: 67 objects, 0 failures. New rules:
- monitoring images must be digest-pinned;
- monitoring pods mount no token;
- no RBAC objects are allowed;
- the monitoring project is scoped.
- Kustomize renders the platform bundle (four namespaces), the monitoring app
(2 Deployments, 2 ConfigMaps, 2 Services, 2 policies, 1 claim), and dev and
stage with the scrape policy.
- The CI `ci-deploy` stage now copies `promtool` and `amtool` from the pinned
images and runs `tools/ci/monitoring-check.sh`. It has not run locally, because
Docker is not running.
- Not executed: anything in the cluster (monitoring sync, target health, the
`EvoRegionDown` injection), alert delivery (no receiver), and the node log
rotation check.