5.7 KiB
[Reviewed]
ADR-0073: Synthetic load and fault drills in dev and stage
Status
Accepted for M8.6, 2026-09-30. It builds on:
- ADR-0064/0065 (host and recovery);
- ADR-0070 (the object reserve);
- ADR-0071 (the Deployment);
- ADR-0072 (alerts).
Context
M8.6 requires:
- dev/stage pod kills to restore the committed world automatically;
- a stale owner that can never resume;
- unavailable state that never initializes a fresh world.
The evidence asked for covers:
- repeated pod kills and a stale process;
- a database or object-store outage, a partition, and a full disk;
- measured RTO, and zero lost acknowledgements.
The named risk is a happy-path pod restart mistaken for disaster recovery.
A region in the cluster receives no inputs: there is no gateway, and a pod's stdin cannot be fed. It therefore never commits past genesis, and an idle region cannot show lost acknowledgements.
Decision
Synthetic load for drills. evo-server --synthetic-inputs development-v1
replaces stdin with a deterministic driver.
- Whenever the host is running with authority, it keeps the bounded backlog fed
with
development_input(t)(the qualification fixture's schedule, including the consumer's death at tick 20). - It continues from the committed head, and restarts from it after every recovery.
- Request IDs are
SYN1 ‖ region ‖ tick, so acknowledgements can be matched across owners.
Like --crash-at, it is a test driver: a real region's inputs come from its
gateway and are never invented. In the Deployment it is off (REGION_INPUTS=""),
and the policy check forbids enabling it in Git. tools/ops/fault-drill.sh turns
it on for the drill and restores the Git values on exit, including on failure. The
driver keeps the region committing continuously, and history is never deleted (no
GC), so drills are short.
Fault drill (tools/ops/fault-drill.sh <dev|stage> [scenario]; World0 is
refused before anything touches the cluster). It streams each pod's log for
acknowledgement accounting, and reads the recovered commit from the successor's
phase running epoch=E committed=C line.
| Scenario | Fault | Required result |
|---|---|---|
| kills | Three forced pod deletions under load | Measured time to Ready (RTO). Every tick the dead pod acknowledged is at or below the committed head its successor recovers, and the epoch increases |
| full | REGION_RESERVE_MIB beyond any disk |
The host recovers and simulates, but acknowledges nothing (the store refuses new objects). Restoring it resumes from the committed head with nothing lost |
| empty | The object volume swapped for an empty one | The host quarantines and never becomes Ready (missing payload), and never provisions a fresh world. Restoring it recovers a head at or past every earlier acknowledgement |
| db-outage | Operator blocks the database longer than the lease | The host stops being Ready and acknowledging. When the block lifts, it re-acquires under a newer epoch (the stale epoch never resumes) with nothing lost |
The database outage is an operator step, for example dropping port 5432 from the store node on the database VM. A NetworkPolicy change cannot sever an established connection under Calico (existing flows are accepted by conntrack), so it would not partition a live session.
RTO. The forced-deletion RTO is expected to be lease-bound: about 20 s under the production lease (M7.9 measured 19.6 s locally; the M8.4 drill measured 19.3 s in dev). The 60 s target remains a proposal until M8.8 ratifies it.
Code evidence. region_load_tests runs evo-server under synthetic load on real
PostgreSQL:
- it acknowledges contiguously past the tick-20 death with deterministic IDs;
- after a SIGKILL, no acknowledgement exceeds the committed head;
- the successor resumes at exactly committed + 1, with no repeated death;
- stdin is ignored in synthetic mode.
Consequences
- Every drill leaves permanent history in dev or stage: about 20 ticks a second, with a commit about every second while loaded. Keep drills to minutes.
- Pod-kill evidence is worker-loss evidence only. Host loss and restoring from backup are M8.7/M8.8. This is the distinction the named risk asks for.
- Argo shows the dev or stage application OutOfSync during a drill, and Synced again once the script restores the Git values.
- The first dev drill found that a full volume made the host retry its refused publication on every step (about 140 a second). Commit retries after a store failure now back off from 100 ms, doubling to a 2 s cap, and reset on the next confirmed commit. The cap is well inside the lease, so recovery after an outage stays prompt.
Alternatives rejected
- A network input listener (gateway-lite) now. Premature design of the gateway's contract.
- A synthetic load sidecar. A pod's stdin cannot be fed from another container without a shared pipe that the image does not have.
- Partitioning with NetworkPolicy. It does not cut established connections under Calico.
Amendment: drill verdicts (M8 review, 2026-09-30)
A missing observation no longer passes. The verdicts are in tools/ops/lib/drill.sh
and are fixture-tested by tools/ci/ops-lib-check.sh:
- Each kill must recover within
RTO_BUDGET_MS, default 60 000 (ADR-0075). - The empty scenario passes only on positive evidence: a
phase quarantinedline, and exit 4 if the container ended. Any other exit fails. No evidence is INCONCLUSIVE, which fails the drill. - A dead or drained owner's acknowledgements are read from its log after the stream has ended.
Limit: the drill reads acknowledgements from logs. A line that was written but not streamed before a forced kill is not observed, so the zero-lost-acknowledgement check covers observed acknowledgements only.