Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by digest and delivered by Argo (ADR-0072), scrape each region's operations Service through static targets (no API access, no RBAC); the region namespaces admit port 9100 from Prometheus only. Sixteen alert rules cover authority, lease, commits, backlog and object capacity and are proven by promtool injection tests in CI with tools copied from the same pinned images. Two new gauges expose object volume free space and the reserve. Alerts route to a null receiver until a destination is chosen; no broker is adopted and the database outbox stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -2811,3 +2811,36 @@ admission policy and the stage and World0 AppProjects, then ran
|
||||
- `region-drill.sh` passed `bash -n`, and refuses World0 before kubectl (also
|
||||
enforced by the policy check).
|
||||
- Not executed: the in-cluster region sync and the drill.
|
||||
|
||||
## M8.5 observability - 2026-09-30
|
||||
|
||||
- `promtool` 3.13.3 and `amtool` 0.34.1 came from the official release tarballs,
|
||||
with SHA-256 verified against the published sums: `b349c732…` and `265b9d1e…`.
|
||||
- The Prometheus configuration syntax is valid, and all 16 rules are valid.
|
||||
- The alert injection tests passed, as did the Alertmanager `check-config`.
|
||||
- Rule mutations: five were tried, and three were caught at first.
|
||||
- **NotReady firing after 15 s:** missed, because the forced-deletion test only
|
||||
evaluated after recovery. The test now also checks during the dip.
|
||||
- **ObjectsFull weakened to half the reserve:** missed, because the injected
|
||||
free space was below both thresholds. The test now uses 900 MB.
|
||||
- After these fixes all five were caught: the lease threshold, NotReady too
|
||||
eager, debt counted without inputs, a weakened ObjectsFull, and `refused`
|
||||
drops counted.
|
||||
- Windows MSVC CTest 59/59, including the `ops-contracts` update to 32 series.
|
||||
- WSL Linux `checks` set: 59/59 in each configuration. The format check passed.
|
||||
- PostgreSQL suites: Release, analysis and ASan/UBSan passed. The ops process
|
||||
test checks the reserve gauge (1 GiB) and that free space is nonzero.
|
||||
- `deploy-policy-check.py`: 67 objects, 0 failures. New rules:
|
||||
- monitoring images must be digest-pinned;
|
||||
- monitoring pods mount no token;
|
||||
- no RBAC objects are allowed;
|
||||
- the monitoring project is scoped.
|
||||
- Kustomize renders the platform bundle (four namespaces), the monitoring app
|
||||
(2 Deployments, 2 ConfigMaps, 2 Services, 2 policies, 1 claim), and dev and
|
||||
stage with the scrape policy.
|
||||
- The CI `ci-deploy` stage now copies `promtool` and `amtool` from the pinned
|
||||
images and runs `tools/ci/monitoring-check.sh`. It has not run locally, because
|
||||
Docker is not running.
|
||||
- Not executed: anything in the cluster (monitoring sync, target health, the
|
||||
`EvoRegionDown` injection), alert delivery (no receiver), and the node log
|
||||
rotation check.
|
||||
|
||||
Reference in New Issue
Block a user