Add M8.5 observability: pinned Prometheus, tested alerts, capacity metrics
An AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by digest and delivered by Argo (ADR-0072), scrape each region's operations Service through static targets (no API access, no RBAC); the region namespaces admit port 9100 from Prometheus only. Sixteen alert rules cover authority, lease, commits, backlog and object capacity and are proven by promtool injection tests in CI with tools copied from the same pinned images. Two new gauges expose object volume free space and the reserve. Alerts route to a null receiver until a destination is chosen; no broker is adopted and the database outbox stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -10,6 +10,8 @@ names it. Remove an item only when it is done, and record where it was done.
|
||||
| **Backup destination for dev/stage stores.** There are no backups yet (maintainer, 2026-09-29). The plan is to build backup tooling for the AEvolution PostgreSQL VM and each `aevolution-objects` volume up to a pluggable destination, then add the destination. It must be off the host for anything beyond disposable environments. | M8.3, 2026-09-29 | The maintainer will supply a destination later | M8.7 backup and restore; any World0 data |
|
||||
| **Secret manager for Argo-delivered configuration.** Secrets are still created by hand (ADR-0069 amendment). Choose how they reach the cluster through Git or Argo: for example Vault with the Agent Injector (as in the maintainer's other environment), or sealed or external secrets. Until then, rotate by hand. | M8.3, 2026-09-29 | The maintainer wants everything through Argo; a secret manager is not chosen | Stage and World0 credentials; M8.4 |
|
||||
| **Store Job values.** Supply the published registry digests for `aevolution-postgres-tools` and `aevolution-server` (from each CI Publish step's `Digest:` line), and the NOLOGIN runtime group role for dev (`aevolution`) and for stage (`aevolution_stage`). Then include `stores/<env>` in each environment's path. | M8.3, 2026-09-29 | Values pending | M8.3 in-cluster evidence |
|
||||
| **Alert destination.** Alertmanager routes every alert to a null receiver (ADR-0072). Choose where alerts go (Discord, email, a webhook), and optionally a heartbeat receiver for the always-firing Watchdog. Receiver credentials go in a Secret. | M8.5, 2026-09-30 | The maintainer will decide later | Any claim that alerts reach a person; M8.6 alert evidence; World0 |
|
||||
| **Kubelet log rotation check.** Confirm the RKE2 nodes' `container-log-max-size` and `container-log-max-files` (default 10Mi × 5), which bound region host logs (ADR-0072). | M8.5, 2026-09-30 | Needs node access | M8.6 |
|
||||
| **Storage after the RAID/disk rework.** Re-run `deploy/kubernetes/inventory/disk-probe-local-path.yaml` (or a new class), and update INVENTORY.md. | M8.1, 2026-09-29 | Maintainer is reorganizing disks | M8.3's final storage decision |
|
||||
|
||||
## Accepted risks (maintainer decisions)
|
||||
|
||||
@@ -635,3 +635,11 @@ M8.4 implemented and locally qualified (2026-09-29), under ADR-0071:
|
||||
- `region-drill.sh` for rollout and forced-deletion evidence.
|
||||
The in-cluster sync and drill wait on the maintainer. No director yet: assignment
|
||||
is static and lives in Git.
|
||||
|
||||
M8.5 implemented and locally qualified (2026-09-30), under ADR-0072:
|
||||
- AEvolution-only Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by digest;
|
||||
- 16 alert rules, tested by injection with promtool in CI;
|
||||
- object volume capacity metrics;
|
||||
- the database outbox kept (no broker);
|
||||
- bounded logs, and a cardinality and cost report.
|
||||
Alert delivery waits on the maintainer's choice of receiver.
|
||||
|
||||
@@ -1064,3 +1064,22 @@ the policy check enforces this. Dev is world 100 and stage world 200, both regio
|
||||
|
||||
Next: apply `deploy/argocd/dev/project.yaml`, sync, and run
|
||||
`tools/ops/region-drill.sh dev`.
|
||||
|
||||
## M8.5 observability (2026-09-30)
|
||||
|
||||
ADR-0072 adds an AEvolution-only monitoring stack in `aevolution-monitoring`:
|
||||
- **Components:** Prometheus 3.13.3 and Alertmanager 0.34.1, pinned by digest and
|
||||
delivered through Argo.
|
||||
- **Scraping:** static targets, so no Kubernetes API access. The region's port
|
||||
9100 is reachable only from Prometheus.
|
||||
- **Alerts:** 16 rules covering authority, lease, commits, backlog and object
|
||||
capacity, plus a Watchdog. Each is proven firing and silent by promtool
|
||||
injection tests in CI. They route to a null receiver until a destination is
|
||||
chosen.
|
||||
- **Metrics:** two new capacity gauges (32 fixed series).
|
||||
|
||||
No broker is adopted: the database outbox stays. Logs are bounded by kubelet
|
||||
rotation. Monitoring costs about 1,300 series, roughly 11 MB a day.
|
||||
|
||||
Next: sync the monitoring app, confirm the targets, and run the `EvoRegionDown`
|
||||
injection. M8.4's drill results are still to be recorded.
|
||||
|
||||
@@ -111,3 +111,10 @@ Tests check the per-call budget, including at the connection cap, and that an
|
||||
accepted probe is still answered under replenished pressure. A real server keeps
|
||||
its lease and readiness while a client opens and closes connections continuously
|
||||
through 60 ticks.
|
||||
|
||||
## Amendment: object volume capacity (2026-09-30, M8.5)
|
||||
Two gauges were added, `evo_region_objects_available_bytes` and
|
||||
`evo_region_objects_reserve_bytes`, so that ADR-0070's free-space reserve can be
|
||||
alerted on before new objects are refused (ADR-0072). The fixed exposition is now
|
||||
32 series. It still has only the `phase` and `reason` labels, and stays under the
|
||||
8 KiB bound.
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
# ADR-0072: Observability, alerting and background delivery for dev and stage
|
||||
|
||||
## Status
|
||||
Accepted for M8.5, 2026-09-30. It implements the M8 part of ADR-0015
|
||||
(observability), and builds on:
|
||||
- ADR-0066 (the operations endpoints);
|
||||
- ADR-0070 (the object reserve);
|
||||
- ADR-0071 (the region host).
|
||||
|
||||
The maintainer chose an AEvolution-only Prometheus, and left the alert destination
|
||||
for later.
|
||||
|
||||
## Context
|
||||
M8.5 asks for:
|
||||
- tick, lease, commit, storage and backup alerts, and bounded logs;
|
||||
- an evaluation of the database outbox before any optional JetStream, with no
|
||||
mandatory Redis or Kafka;
|
||||
- evidence: alert injection, and a telemetry cardinality and cost report.
|
||||
|
||||
The named risks are broker redelivery changing authority, and logs filling a disk.
|
||||
|
||||
The cluster has no monitoring stack beyond metrics-server (INVENTORY.md). Each
|
||||
region exposes 32 fixed series. The only labels are a 6-value phase and a 4-value
|
||||
drop reason.
|
||||
|
||||
## Decision
|
||||
**Stack.** Namespace `aevolution-monitoring`, with platform-owned guardrails, and
|
||||
a manually synced Argo application `aevolution-monitoring`.
|
||||
- **Components.** One Prometheus and one Alertmanager:
|
||||
- `quay.io/prometheus/prometheus:v3.13.3`, pinned by digest `6976aa8a…`;
|
||||
- `quay.io/prometheus/alertmanager:v0.34.1`, pinned by digest `e9733baf…`.
|
||||
|
||||
Both are Apache-2.0, and Prometheus is a CNCF graduated project. 3.13 is a
|
||||
maintained line: 3.13.3 of 2026-09-07 carries current security fixes.
|
||||
- **Prometheus configuration:**
|
||||
- Static scrape targets: the `evo-region-ops` Service in dev and stage, labelled
|
||||
with `environment`, `world` and `region`. No service discovery, so no token
|
||||
and no RBAC.
|
||||
- 15 s scrape and evaluation, a 64-sample and 64 KB cap per scrape, and
|
||||
retention of 15 days or 3 GB.
|
||||
- A 5 Gi `local-path` volume. Metrics are not world data, so Delete reclaim is
|
||||
acceptable.
|
||||
- **Network.** The region namespaces admit port 9100 only from Prometheus.
|
||||
Prometheus reaches only the region operations ports and Alertmanager;
|
||||
Alertmanager accepts only Prometheus. Pods run as nobody, with a read-only root
|
||||
filesystem and no capabilities.
|
||||
- **Routing.** Alertmanager routes everything to a **null** receiver until a
|
||||
destination is chosen (docs/BACKLOG.md). Alerts remain visible in its API and
|
||||
UI.
|
||||
|
||||
**Metrics added.** `evo_region_objects_available_bytes` and
|
||||
`evo_region_objects_reserve_bytes` make volume capacity alertable before new
|
||||
objects are refused. The fixed exposition is now 32 series.
|
||||
|
||||
**Alerts** (`config/evo-region.rules.yml`, 16 rules). Thresholds come from
|
||||
measured behaviour:
|
||||
|
||||
| Area | Alerts |
|
||||
|---|---|
|
||||
| Authority | Down (2 min), NotReady (2 min; a forced-deletion recovery takes about 20 s and does not fire), Quarantined, LeaseMargin (below 10 s while ready; healthy is 15–20 s), Fenced |
|
||||
| Progress | StoreFailures, UnknownOutcomes, CommitLag (over 256 ticks), TickDebt and CheckpointStale (only while inputs or lag wait, because an idle region always accrues debt), QueueSaturated, InputsDropped (not counting `refused`) |
|
||||
| Storage | ObjectsLow (below 2× the reserve), ObjectsFull (below the reserve) |
|
||||
| Pipeline | Watchdog (always firing, for a later heartbeat receiver), AlertmanagerDown |
|
||||
|
||||
**Backup alerts** wait for backups (M8.7). There is nothing to alert on until a
|
||||
backup destination exists; see docs/BACKLOG.md.
|
||||
|
||||
**Alert injection.** `promtool test rules` checks each alert firing on injected
|
||||
series and staying silent in the healthy, idle and 20 s recovery cases. The CI
|
||||
`deploy` stage runs it with the `promtool` and `amtool` copied from the same
|
||||
pinned images, together with syntax checks of the Prometheus and Alertmanager
|
||||
configurations.
|
||||
|
||||
**Logs.** Container logs are rotated by the kubelet. The defaults are 10 Mi per
|
||||
file and 5 files per container, 50 Mi at most per container. The actual RKE2
|
||||
setting has not been verified on the nodes yet. At most:
|
||||
- `evo-server` writes one ack line of about 60 bytes per tick, about 100 MB a day
|
||||
at 20 Hz, so the rotation keeps the most recent 12 hours;
|
||||
- Prometheus and Alertmanager log little.
|
||||
|
||||
Nothing writes logs to a volume. Durable history lives in PostgreSQL and the
|
||||
object store, never in logs.
|
||||
|
||||
**Background delivery.** The database outbox stays the delivery mechanism, and no
|
||||
broker is adopted:
|
||||
- `region_outbox` is written in the same transaction as each permanent outcome,
|
||||
delivery is idempotent (`acknowledge_region_outbox`), and delivery never
|
||||
mutates simulation state (ADR-0063).
|
||||
- There is no consumer of outcomes yet: no gateway, client or projection. A relay
|
||||
worker would deliver to nothing, so it is deferred until a consumer exists.
|
||||
- JetStream is not adopted. There is no demonstrated delivery workload, and a
|
||||
broker would add a second durability layer to operate and back up. It could
|
||||
also never hold authority, since its redelivery could only re-announce an
|
||||
already-final outcome. Redis and Kafka are not needed.
|
||||
|
||||
**Cardinality and cost** (per 15 s scrape):
|
||||
|
||||
| Source | Series |
|
||||
|---|---|
|
||||
| Each region | 32, plus about 5 per-target scrape series |
|
||||
| Two regions (dev, stage) | about 75 |
|
||||
| Prometheus self-scrape | about 1,000 (typical) |
|
||||
| Alertmanager | about 200 |
|
||||
| **Total** | **about 1,300** |
|
||||
|
||||
That is about 87 samples per second. At about 1.5 bytes per compressed sample,
|
||||
that is about 11 MB a day, or about 170 MB over 15 days, well inside the 3 GB
|
||||
retention cap and the 5 Gi volume. Each added region adds about 37 series. The
|
||||
request is 100m CPU and 256 Mi, with a 1 Gi memory limit. Alertmanager requests
|
||||
50m and 64 Mi, with a 128 Mi limit.
|
||||
|
||||
## Consequences
|
||||
- Alerts are evaluated and visible, but not delivered until a receiver is chosen.
|
||||
The Watchdog is ready for a dead-man's-switch receiver.
|
||||
- Changing a threshold is a Git change that must keep the rule tests passing.
|
||||
- The kubelet log rotation values should be confirmed on the nodes.
|
||||
- World0 monitoring (M8.8) reuses the rules with its own target.
|
||||
|
||||
## Alternatives rejected
|
||||
- **kube-prometheus-stack or the Prometheus Operator.** Cluster-wide CRDs and
|
||||
RBAC for a single project; not the maintainer's choice.
|
||||
- **Kubernetes service discovery.** It needs a token and cluster read RBAC for no
|
||||
gain with two static targets.
|
||||
- **JetStream, Kafka or Redis for outcomes.** There is no workload yet, and each
|
||||
would add another stateful system.
|
||||
@@ -82,3 +82,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0069: Environment isolation and World0 protection](0069-environment-isolation.md) - accepted for M8.2 per-environment guardrails, native World0 admission protection, scoped Argo projects and verification.
|
||||
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
|
||||
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
|
||||
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
|
||||
|
||||
@@ -2811,3 +2811,36 @@ admission policy and the stage and World0 AppProjects, then ran
|
||||
- `region-drill.sh` passed `bash -n`, and refuses World0 before kubectl (also
|
||||
enforced by the policy check).
|
||||
- Not executed: the in-cluster region sync and the drill.
|
||||
|
||||
## M8.5 observability - 2026-09-30
|
||||
|
||||
- `promtool` 3.13.3 and `amtool` 0.34.1 came from the official release tarballs,
|
||||
with SHA-256 verified against the published sums: `b349c732…` and `265b9d1e…`.
|
||||
- The Prometheus configuration syntax is valid, and all 16 rules are valid.
|
||||
- The alert injection tests passed, as did the Alertmanager `check-config`.
|
||||
- Rule mutations: five were tried, and three were caught at first.
|
||||
- **NotReady firing after 15 s:** missed, because the forced-deletion test only
|
||||
evaluated after recovery. The test now also checks during the dip.
|
||||
- **ObjectsFull weakened to half the reserve:** missed, because the injected
|
||||
free space was below both thresholds. The test now uses 900 MB.
|
||||
- After these fixes all five were caught: the lease threshold, NotReady too
|
||||
eager, debt counted without inputs, a weakened ObjectsFull, and `refused`
|
||||
drops counted.
|
||||
- Windows MSVC CTest 59/59, including the `ops-contracts` update to 32 series.
|
||||
- WSL Linux `checks` set: 59/59 in each configuration. The format check passed.
|
||||
- PostgreSQL suites: Release, analysis and ASan/UBSan passed. The ops process
|
||||
test checks the reserve gauge (1 GiB) and that free space is nonzero.
|
||||
- `deploy-policy-check.py`: 67 objects, 0 failures. New rules:
|
||||
- monitoring images must be digest-pinned;
|
||||
- monitoring pods mount no token;
|
||||
- no RBAC objects are allowed;
|
||||
- the monitoring project is scoped.
|
||||
- Kustomize renders the platform bundle (four namespaces), the monitoring app
|
||||
(2 Deployments, 2 ConfigMaps, 2 Services, 2 policies, 1 claim), and dev and
|
||||
stage with the scrape policy.
|
||||
- The CI `ci-deploy` stage now copies `promtool` and `amtool` from the pinned
|
||||
images and runs `tools/ci/monitoring-check.sh`. It has not run locally, because
|
||||
Docker is not running.
|
||||
- Not executed: anything in the cluster (monitoring sync, target health, the
|
||||
`EvoRegionDown` injection), alert delivery (no receiver), and the node log
|
||||
rotation check.
|
||||
|
||||
@@ -139,6 +139,27 @@ record:
|
||||
- the first pod's `phase running epoch=... committed=...` line;
|
||||
- that `/readyz` stays 503 while a replacement waits out the lease.
|
||||
|
||||
## Monitoring (M8.5, ADR-0072)
|
||||
|
||||
Bootstrap once: `kubectl apply -f deploy/argocd/monitoring/`, then sync
|
||||
`aevolution-platform` (it adds the `aevolution-monitoring` namespace and its
|
||||
guardrails) and `aevolution-monitoring`. Re-sync dev and stage, which adds the
|
||||
`evo-region-scrape` policy.
|
||||
|
||||
```sh
|
||||
kubectl -n aevolution-monitoring port-forward svc/prometheus 9090 &
|
||||
curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"' # both regions up
|
||||
curl -s localhost:9090/api/v1/alerts # Watchdog firing only
|
||||
```
|
||||
|
||||
**Injection in the cluster** (dev only):
|
||||
- `kubectl -n evolution-dev scale deploy/evo-region --replicas=0` makes
|
||||
`EvoRegionDown` pending, then firing after 2 minutes, in Alertmanager (null
|
||||
receiver).
|
||||
- Scale back to 1 and it resolves.
|
||||
|
||||
Record the timestamps. The rule tests in CI already prove the thresholds.
|
||||
|
||||
## Break-glass
|
||||
|
||||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||||
|
||||
Reference in New Issue
Block a user