Add M8.6 synthetic load driver and fault drill

A drill-only evo-server --synthetic-inputs development-v1 driver feeds
deterministic inputs from the committed head after every recovery, with
request IDs that match acknowledgements across owners (ADR-0073); it is
off in Git and the policy check keeps it off. fault-drill.sh loads dev
or stage and runs repeated forced kills (RTO, zero lost
acknowledgements), a full object volume (nothing acknowledged), an
empty volume (quarantine, never a fresh world) and an operator-driven
database outage (stale epoch never resumes), restoring the Git
configuration on exit. Process tests prove contiguous acknowledgement
across a SIGKILL and exact resumption after the committed head.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-30 10:52:01 -04:00
co-authored by Claude Opus 5.5
parent 4ef0fe8167
commit 3f08a1d221
16 changed files with 513 additions and 5 deletions
+7
View File
@@ -643,3 +643,10 @@ M8.5 implemented and locally qualified (2026-09-30), under ADR-0072:
- the database outbox kept (no broker);
- bounded logs, and a cardinality and cost report.
Alert delivery waits on the maintainer's choice of receiver.
M8.6 implemented and locally qualified (2026-09-30), under ADR-0073:
- a drill-only synthetic input driver;
- `fault-drill.sh`, covering repeated forced kills, a full volume, an empty volume
and an operator-driven database outage;
- RTO and zero-lost-acknowledgement checks.
The in-cluster drill results wait on the maintainer.
+22
View File
@@ -1083,3 +1083,25 @@ rotation. Monitoring costs about 1,300 series, roughly 11 MB a day.
Next: sync the monitoring app, confirm the targets, and run the `EvoRegionDown`
injection. M8.4's drill results are still to be recorded.
## M8.6 fault drills (2026-09-30)
ADR-0073 adds a drill-only synthetic driver, `evo-server --synthetic-inputs
development-v1`:
- it feeds deterministic inputs from the committed head after every recovery,
including the tick-20 death;
- its request IDs are `SYN1‖region‖tick`;
- it is off in Git, and the policy check enforces that.
`tools/ops/fault-drill.sh` puts dev or stage under load and runs these scenarios:
- **kills:** three forced kills, measuring RTO and checking that no acknowledgement
is lost;
- **full:** a full volume, which must acknowledge nothing;
- **empty:** an empty volume, which must quarantine and never start a fresh world;
- **db-outage:** an operator-blocked database, after which a stale epoch never
resumes.
It restores the Git configuration on exit.
Earlier (M8.4 follow-up): `REGION_RETAINED` keeps earlier builds recoverable
across image upgrades.
+87
View File
@@ -0,0 +1,87 @@
# ADR-0073: Synthetic load and fault drills in dev and stage
## Status
Accepted for M8.6, 2026-09-30. It builds on:
- ADR-0064/0065 (host and recovery);
- ADR-0070 (the object reserve);
- ADR-0071 (the Deployment);
- ADR-0072 (alerts).
## Context
M8.6 requires:
- dev/stage pod kills to restore the committed world automatically;
- a stale owner that can never resume;
- unavailable state that never initializes a fresh world.
The evidence asked for covers:
- repeated pod kills and a stale process;
- a database or object-store outage, a partition, and a full disk;
- measured RTO, and zero lost acknowledgements.
The named risk is a happy-path pod restart mistaken for disaster recovery.
A region in the cluster receives no inputs: there is no gateway, and a pod's stdin
cannot be fed. It therefore never commits past genesis, and an idle region cannot
show lost acknowledgements.
## Decision
**Synthetic load for drills.** `evo-server --synthetic-inputs development-v1`
replaces stdin with a deterministic driver.
- Whenever the host is running with authority, it keeps the bounded backlog fed
with `development_input(t)` (the qualification fixture's schedule, including
the consumer's death at tick 20).
- It continues from the committed head, and restarts from it after every
recovery.
- Request IDs are `SYN1 ‖ region ‖ tick`, so acknowledgements can be matched
across owners.
Like `--crash-at`, it is a test driver: a real region's inputs come from its
gateway and are never invented. In the Deployment it is off (`REGION_INPUTS=""`),
and the policy check forbids enabling it in Git. `tools/ops/fault-drill.sh` turns
it on for the drill and restores the Git values on exit, including on failure. The
driver keeps the region committing continuously, and history is never deleted (no
GC), so drills are short.
**Fault drill** (`tools/ops/fault-drill.sh <dev|stage> [scenario]`; World0 is
refused before anything touches the cluster). It streams each pod's log for
acknowledgement accounting, and reads the recovered commit from the successor's
`phase running epoch=E committed=C` line.
| Scenario | Fault | Required result |
|---|---|---|
| kills | Three forced pod deletions under load | Measured time to Ready (RTO). Every tick the dead pod acknowledged is at or below the committed head its successor recovers, and the epoch increases |
| full | `REGION_RESERVE_MIB` beyond any disk | The host recovers and simulates, but acknowledges nothing (the store refuses new objects). Restoring it resumes from the committed head with nothing lost |
| empty | The object volume swapped for an empty one | The host quarantines and never becomes Ready (missing payload), and never provisions a fresh world. Restoring it recovers a head at or past every earlier acknowledgement |
| db-outage | Operator blocks the database longer than the lease | The host stops being Ready and acknowledging. When the block lifts, it re-acquires under a newer epoch (the stale epoch never resumes) with nothing lost |
The database outage is an operator step, for example dropping port 5432 from the
store node on the database VM. A NetworkPolicy change cannot sever an established
connection under Calico (existing flows are accepted by conntrack), so it would
not partition a live session.
**RTO.** The forced-deletion RTO is expected to be lease-bound: about 20 s under the
production lease (M7.9 measured 19.6 s locally; the M8.4 drill measured 19.3 s in
dev). The 60 s target remains a proposal until M8.8 ratifies it.
**Code evidence.** `region_load_tests` runs `evo-server` under synthetic load on real
PostgreSQL:
- it acknowledges contiguously past the tick-20 death with deterministic IDs;
- after a SIGKILL, no acknowledgement exceeds the committed head;
- the successor resumes at exactly committed + 1, with no repeated death;
- stdin is ignored in synthetic mode.
## Consequences
- Every drill leaves permanent history in dev or stage: about 20 ticks a second,
with a commit about every second while loaded. Keep drills to minutes.
- Pod-kill evidence is worker-loss evidence only. Host loss and restoring from
backup are M8.7/M8.8. This is the distinction the named risk asks for.
- Argo shows the dev or stage application OutOfSync during a drill, and Synced
again once the script restores the Git values.
## Alternatives rejected
- **A network input listener (gateway-lite) now.** Premature design of the
gateway's contract.
- **A synthetic load sidecar.** A pod's stdin cannot be fed from another container
without a shared pipe that the image does not have.
- **Partitioning with NetworkPolicy.** It does not cut established connections
under Calico.
+1
View File
@@ -83,3 +83,4 @@ design. No unadopted candidate is a link dependency of the proof.
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
+22
View File
@@ -2860,3 +2860,25 @@ admission policy and the stage and World0 AppProjects, then ran
- Image upgrade to `edb39f…`: completed with `REGION_RETAINED=4af72099…`, so the
existing dev and stage heads, written by the earlier build, stay recoverable
after the upgrade.
## M8.6 synthetic load and fault drills - 2026-09-30
- `region_load_tests` (real PostgreSQL and real `evo-server`, fast lease):
- A synthetic owner acknowledged contiguously from tick 1 past tick 45, with
deterministic `SYN1` IDs. The consumer's death was emitted as an outcome.
- After a SIGKILL mid-stream, every acknowledgement was at or below the
committed head, and there was exactly one durable decision.
- The successor started at epoch 2 and acknowledged exactly committed + 1 onward,
with no repeated death.
- With synthetic inputs enabled, a line typed on stdin was ignored.
- `lineage-region-contracts`: `development_input(t)` equals the fixture's
`region_input(t, true)` for ticks 1–200.
- Mutation: resuming the driver at committed + 2 was caught by the contiguity check.
- Windows MSVC CTest 59/59. WSL Linux `checks` set: 59/59 in each configuration.
PostgreSQL suites passed in Release, analysis and ASan/UBSan. The format check
passed.
- `deploy-policy-check.py`: 67 objects, 0 failures. New rules: `REGION_INPUTS`
must be empty in Git, and `fault-drill.sh` must refuse World0 before kubectl.
- `fault-drill.sh` passed `bash -n`. It refuses `world0` and unknown scenarios
(exit 2) without calling kubectl.
- Not executed: the in-cluster drill (kills, full, empty, db-outage).
+22
View File
@@ -160,6 +160,28 @@ curl -s localhost:9090/api/v1/alerts # Watchdog
Record the timestamps. The rule tests in CI already prove the thresholds.
## Fault drills (M8.6, ADR-0073)
`bash tools/ops/fault-drill.sh dev all` enables synthetic load, then runs three
forced kills, a full volume and an empty volume. It restores the Git settings on
exit.
For the database outage, run `bash tools/ops/fault-drill.sh dev db-outage` and
follow its prompt. On `pg-forge-01`, block the store node for at least 30 s, then
lift the block:
```sh
sudo iptables -I INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP # rke2-worker-1
sleep 40
sudo iptables -D INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP
```
Record each `kill`, `full`, `empty` and `db-outage` line in VALIDATION.md. During
the drill, Prometheus should show:
- `EvoRegionObjectsFull` (full);
- `EvoRegionQuarantined` or `EvoRegionNotReady` (empty);
- `EvoRegionFenced` (db-outage).
Keep drills short: synthetic load leaves permanent history.
## Break-glass
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,