Add M8.6 synthetic load driver and fault drill
A drill-only evo-server --synthetic-inputs development-v1 driver feeds deterministic inputs from the committed head after every recovery, with request IDs that match acknowledgements across owners (ADR-0073); it is off in Git and the policy check keeps it off. fault-drill.sh loads dev or stage and runs repeated forced kills (RTO, zero lost acknowledgements), a full object volume (nothing acknowledged), an empty volume (quarantine, never a fresh world) and an operator-driven database outage (stale epoch never resumes), restoring the Git configuration on exit. Process tests prove contiguous acknowledgement across a SIGKILL and exact resumption after the committed head. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -643,3 +643,10 @@ M8.5 implemented and locally qualified (2026-09-30), under ADR-0072:
|
||||
- the database outbox kept (no broker);
|
||||
- bounded logs, and a cardinality and cost report.
|
||||
Alert delivery waits on the maintainer's choice of receiver.
|
||||
|
||||
M8.6 implemented and locally qualified (2026-09-30), under ADR-0073:
|
||||
- a drill-only synthetic input driver;
|
||||
- `fault-drill.sh`, covering repeated forced kills, a full volume, an empty volume
|
||||
and an operator-driven database outage;
|
||||
- RTO and zero-lost-acknowledgement checks.
|
||||
The in-cluster drill results wait on the maintainer.
|
||||
|
||||
@@ -1083,3 +1083,25 @@ rotation. Monitoring costs about 1,300 series, roughly 11 MB a day.
|
||||
|
||||
Next: sync the monitoring app, confirm the targets, and run the `EvoRegionDown`
|
||||
injection. M8.4's drill results are still to be recorded.
|
||||
|
||||
## M8.6 fault drills (2026-09-30)
|
||||
|
||||
ADR-0073 adds a drill-only synthetic driver, `evo-server --synthetic-inputs
|
||||
development-v1`:
|
||||
- it feeds deterministic inputs from the committed head after every recovery,
|
||||
including the tick-20 death;
|
||||
- its request IDs are `SYN1‖region‖tick`;
|
||||
- it is off in Git, and the policy check enforces that.
|
||||
|
||||
`tools/ops/fault-drill.sh` puts dev or stage under load and runs these scenarios:
|
||||
- **kills:** three forced kills, measuring RTO and checking that no acknowledgement
|
||||
is lost;
|
||||
- **full:** a full volume, which must acknowledge nothing;
|
||||
- **empty:** an empty volume, which must quarantine and never start a fresh world;
|
||||
- **db-outage:** an operator-blocked database, after which a stale epoch never
|
||||
resumes.
|
||||
|
||||
It restores the Git configuration on exit.
|
||||
|
||||
Earlier (M8.4 follow-up): `REGION_RETAINED` keeps earlier builds recoverable
|
||||
across image upgrades.
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
# ADR-0073: Synthetic load and fault drills in dev and stage
|
||||
|
||||
## Status
|
||||
Accepted for M8.6, 2026-09-30. It builds on:
|
||||
- ADR-0064/0065 (host and recovery);
|
||||
- ADR-0070 (the object reserve);
|
||||
- ADR-0071 (the Deployment);
|
||||
- ADR-0072 (alerts).
|
||||
|
||||
## Context
|
||||
M8.6 requires:
|
||||
- dev/stage pod kills to restore the committed world automatically;
|
||||
- a stale owner that can never resume;
|
||||
- unavailable state that never initializes a fresh world.
|
||||
|
||||
The evidence asked for covers:
|
||||
- repeated pod kills and a stale process;
|
||||
- a database or object-store outage, a partition, and a full disk;
|
||||
- measured RTO, and zero lost acknowledgements.
|
||||
|
||||
The named risk is a happy-path pod restart mistaken for disaster recovery.
|
||||
|
||||
A region in the cluster receives no inputs: there is no gateway, and a pod's stdin
|
||||
cannot be fed. It therefore never commits past genesis, and an idle region cannot
|
||||
show lost acknowledgements.
|
||||
|
||||
## Decision
|
||||
**Synthetic load for drills.** `evo-server --synthetic-inputs development-v1`
|
||||
replaces stdin with a deterministic driver.
|
||||
- Whenever the host is running with authority, it keeps the bounded backlog fed
|
||||
with `development_input(t)` (the qualification fixture's schedule, including
|
||||
the consumer's death at tick 20).
|
||||
- It continues from the committed head, and restarts from it after every
|
||||
recovery.
|
||||
- Request IDs are `SYN1 ‖ region ‖ tick`, so acknowledgements can be matched
|
||||
across owners.
|
||||
|
||||
Like `--crash-at`, it is a test driver: a real region's inputs come from its
|
||||
gateway and are never invented. In the Deployment it is off (`REGION_INPUTS=""`),
|
||||
and the policy check forbids enabling it in Git. `tools/ops/fault-drill.sh` turns
|
||||
it on for the drill and restores the Git values on exit, including on failure. The
|
||||
driver keeps the region committing continuously, and history is never deleted (no
|
||||
GC), so drills are short.
|
||||
|
||||
**Fault drill** (`tools/ops/fault-drill.sh <dev|stage> [scenario]`; World0 is
|
||||
refused before anything touches the cluster). It streams each pod's log for
|
||||
acknowledgement accounting, and reads the recovered commit from the successor's
|
||||
`phase running epoch=E committed=C` line.
|
||||
|
||||
| Scenario | Fault | Required result |
|
||||
|---|---|---|
|
||||
| kills | Three forced pod deletions under load | Measured time to Ready (RTO). Every tick the dead pod acknowledged is at or below the committed head its successor recovers, and the epoch increases |
|
||||
| full | `REGION_RESERVE_MIB` beyond any disk | The host recovers and simulates, but acknowledges nothing (the store refuses new objects). Restoring it resumes from the committed head with nothing lost |
|
||||
| empty | The object volume swapped for an empty one | The host quarantines and never becomes Ready (missing payload), and never provisions a fresh world. Restoring it recovers a head at or past every earlier acknowledgement |
|
||||
| db-outage | Operator blocks the database longer than the lease | The host stops being Ready and acknowledging. When the block lifts, it re-acquires under a newer epoch (the stale epoch never resumes) with nothing lost |
|
||||
|
||||
The database outage is an operator step, for example dropping port 5432 from the
|
||||
store node on the database VM. A NetworkPolicy change cannot sever an established
|
||||
connection under Calico (existing flows are accepted by conntrack), so it would
|
||||
not partition a live session.
|
||||
|
||||
**RTO.** The forced-deletion RTO is expected to be lease-bound: about 20 s under the
|
||||
production lease (M7.9 measured 19.6 s locally; the M8.4 drill measured 19.3 s in
|
||||
dev). The 60 s target remains a proposal until M8.8 ratifies it.
|
||||
|
||||
**Code evidence.** `region_load_tests` runs `evo-server` under synthetic load on real
|
||||
PostgreSQL:
|
||||
- it acknowledges contiguously past the tick-20 death with deterministic IDs;
|
||||
- after a SIGKILL, no acknowledgement exceeds the committed head;
|
||||
- the successor resumes at exactly committed + 1, with no repeated death;
|
||||
- stdin is ignored in synthetic mode.
|
||||
|
||||
## Consequences
|
||||
- Every drill leaves permanent history in dev or stage: about 20 ticks a second,
|
||||
with a commit about every second while loaded. Keep drills to minutes.
|
||||
- Pod-kill evidence is worker-loss evidence only. Host loss and restoring from
|
||||
backup are M8.7/M8.8. This is the distinction the named risk asks for.
|
||||
- Argo shows the dev or stage application OutOfSync during a drill, and Synced
|
||||
again once the script restores the Git values.
|
||||
|
||||
## Alternatives rejected
|
||||
- **A network input listener (gateway-lite) now.** Premature design of the
|
||||
gateway's contract.
|
||||
- **A synthetic load sidecar.** A pod's stdin cannot be fed from another container
|
||||
without a shared pipe that the image does not have.
|
||||
- **Partitioning with NetworkPolicy.** It does not cut established connections
|
||||
under Calico.
|
||||
@@ -83,3 +83,4 @@ design. No unadopted candidate is a link dependency of the proof.
|
||||
- [ADR-0070: Durable stores for dev and stage](0070-dev-stage-stores.md) - accepted for M8.3 external PostgreSQL per environment, Retain-class node-local object volumes, free-space reserve and in-cluster store qualification.
|
||||
- [ADR-0071: Region host deployment in dev and stage](0071-region-host-deployment.md) - accepted for M8.4 single-replica Recreate host, digest-as-build identity, idempotent provisioning hook and the region drill; no director yet.
|
||||
- [ADR-0072: Observability, alerting and background delivery](0072-observability.md) - accepted for M8.5 pinned AEvolution Prometheus/Alertmanager, 16 tested alert rules, bounded logs, database outbox over any broker, cardinality/cost report.
|
||||
- [ADR-0073: Synthetic load and fault drills](0073-fault-drills.md) - accepted for M8.6 drill-only deterministic input driver, kill/full/empty/db-outage drill with RTO and zero-lost-acknowledgement checks.
|
||||
|
||||
@@ -2860,3 +2860,25 @@ admission policy and the stage and World0 AppProjects, then ran
|
||||
- Image upgrade to `edb39f…`: completed with `REGION_RETAINED=4af72099…`, so the
|
||||
existing dev and stage heads, written by the earlier build, stay recoverable
|
||||
after the upgrade.
|
||||
|
||||
## M8.6 synthetic load and fault drills - 2026-09-30
|
||||
|
||||
- `region_load_tests` (real PostgreSQL and real `evo-server`, fast lease):
|
||||
- A synthetic owner acknowledged contiguously from tick 1 past tick 45, with
|
||||
deterministic `SYN1` IDs. The consumer's death was emitted as an outcome.
|
||||
- After a SIGKILL mid-stream, every acknowledgement was at or below the
|
||||
committed head, and there was exactly one durable decision.
|
||||
- The successor started at epoch 2 and acknowledged exactly committed + 1 onward,
|
||||
with no repeated death.
|
||||
- With synthetic inputs enabled, a line typed on stdin was ignored.
|
||||
- `lineage-region-contracts`: `development_input(t)` equals the fixture's
|
||||
`region_input(t, true)` for ticks 1–200.
|
||||
- Mutation: resuming the driver at committed + 2 was caught by the contiguity check.
|
||||
- Windows MSVC CTest 59/59. WSL Linux `checks` set: 59/59 in each configuration.
|
||||
PostgreSQL suites passed in Release, analysis and ASan/UBSan. The format check
|
||||
passed.
|
||||
- `deploy-policy-check.py`: 67 objects, 0 failures. New rules: `REGION_INPUTS`
|
||||
must be empty in Git, and `fault-drill.sh` must refuse World0 before kubectl.
|
||||
- `fault-drill.sh` passed `bash -n`. It refuses `world0` and unknown scenarios
|
||||
(exit 2) without calling kubectl.
|
||||
- Not executed: the in-cluster drill (kills, full, empty, db-outage).
|
||||
|
||||
@@ -160,6 +160,28 @@ curl -s localhost:9090/api/v1/alerts # Watchdog
|
||||
|
||||
Record the timestamps. The rule tests in CI already prove the thresholds.
|
||||
|
||||
## Fault drills (M8.6, ADR-0073)
|
||||
|
||||
`bash tools/ops/fault-drill.sh dev all` enables synthetic load, then runs three
|
||||
forced kills, a full volume and an empty volume. It restores the Git settings on
|
||||
exit.
|
||||
|
||||
For the database outage, run `bash tools/ops/fault-drill.sh dev db-outage` and
|
||||
follow its prompt. On `pg-forge-01`, block the store node for at least 30 s, then
|
||||
lift the block:
|
||||
```sh
|
||||
sudo iptables -I INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP # rke2-worker-1
|
||||
sleep 40
|
||||
sudo iptables -D INPUT -p tcp --dport 5432 -s 172.10.1.135 -j DROP
|
||||
```
|
||||
Record each `kill`, `full`, `empty` and `db-outage` line in VALIDATION.md. During
|
||||
the drill, Prometheus should show:
|
||||
- `EvoRegionObjectsFull` (full);
|
||||
- `EvoRegionQuarantined` or `EvoRegionNotReady` (empty);
|
||||
- `EvoRegionFenced` (db-outage).
|
||||
|
||||
Keep drills short: synthetic load leaves permanent history.
|
||||
|
||||
## Break-glass
|
||||
|
||||
How membership is granted is not decided yet; see docs/BACKLOG.md. Until it is,
|
||||
|
||||
Reference in New Issue
Block a user