Files
aevolution/docs/adr/0064-region-host.md
T
2026-10-08 12:08:53 -04:00

120 lines
5.6 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
[Reviewed]
# ADR-0064: Bounded authoritative region host
## Status
Accepted for M7.6, 2026-09-28. It composes ADR-0060 (fencing), ADR-0061/0062
(checkpoints) and ADR-0063 (batch outcomes). It adds no gateway, network listener,
metrics endpoint or deployment; those are M7.8 and later.
## Context
The persistence APIs exist, but nothing yet owns a region over time. ADR-0058
requires:
- one simulation owner, with simulated and committed ticks both visible;
- bounded backlogs, with backpressure before admission;
- a whole-region pause at any irreversible outcome until durable confirmation;
- bounded drain, and readiness that fails closed.
Nothing may treat an uncommitted death as fact.
## Decision
**`RegionHost<State>`** (`src/server`, Linux), for `EnvironmentalCommunity` and
`LineageRegion`, is a single-threaded state machine advanced by `step(now)`:
- *acquiring*: take the fenced lease, then recover and verify the committed head.
The recovered quantum and frame must match the configuration.
- *running*: lease upkeep, bounded ticks, and commits.
- *resolving*: an ambiguous commit is resolved before anything else.
- *draining*, then *stopped*; or *quarantined*.
All I/O runs on the caller's thread within the LeasePolicy deadlines. While a
commit runs, the region is stopped.
**Admission** is provisional:
- The backlog holds at most 64 inputs and 64 KiB (88 bytes are accounted per
input). When it is full, the caller gets `full` and must hold its input.
- Duplicate request identities within the host are refused.
- Inputs are refused unless the host is ready.
An input becomes final only when the commit containing its tick is confirmed.
The host then emits `FinalAck(request, tick, commit_sequence)`; permanent
outcomes are emitted as `FinalOutcome`. Inputs that will never be acknowledged
are reported as `Dropped`, with reason refused, drained, fenced or quarantined.
After an unknown outcome, "fenced" means "unknown to this host", not "not
applied".
**Ticks**:
- Scheduling time comes from the caller, while lease deadlines use `LeaseClock`
and database time.
- The due count is (now − start) / quantum. Each step runs at most 8 catch-up
ticks, and one queued input is consumed per tick.
- An empty backlog accrues tick debt. The host never invents inputs, and debt is
measured (current and maximum) rather than skipped.
**Commits** use `RegionBatch`:
- A death seals the batch and commits in the same step, so no dependent tick runs
on an uncommitted outcome.
- Otherwise a batch flushes after `flush_interval` (default 1 s), at 64 ticks,
or on drain.
- The commit ID is the acquisition's holder token with the next sequence number
in its last 8 bytes. It is stable across retries of one batch, and unique
because holder tokens are generated per acquisition.
| Outcome | Host response |
|---|---|
| Storage failure (upload/verify/digest), transient SQL error | Stay paused and retry the same batch; `store_failures` counts it |
| Unknown | Enter *resolving*: resolve the exact prepared bytes. Committed → adopt the receipt. Absent → retry the same bytes while `may_begin_commit` holds. Conflict → quarantine |
| Fenced | Discard state, report unconfirmed inputs, re-acquire and recover from the head |
| Candidate, predecessor, head moved, conflict, receipt mismatch, unrestorable head | Quarantine, releasing the lease if still held |
**Readiness** is true only when running, recovered, not draining, and
`LeaseAuthority::may_originate` holds.
**Drain** stops admission, reports the backlog as drained, commits pending ticks,
releases the lease, then stops. A successor can therefore acquire immediately,
without waiting for expiry.
**`evo-server`** is the development process around one host:
- Tick inputs are stdin lines. It reads only while the backlog has room, so stdin
itself backpressures.
- Events are stdout lines: phase, ready, ack, outcome, dropped, stats.
- An optional readiness file mirrors readiness.
- SIGTERM or SIGINT drain within `--drain-ms`.
- Exit codes: 0 drained, 3 drain deadline passed (pending or unknown work is left
to the next owner), 4 quarantined, 2 usage.
- `--lease-profile fast` (a 4 s lease) is for development tests only.
## Alternatives Considered
- **A separate I/O thread**: would allow ticks during commits, but the contract
pauses the region at commits anyway, and extra threads violate single ownership
without a measured need.
- **Wall-clock-driven synthetic inputs**: would invent world input.
- **Acknowledging on apply**: would expose provisional state as final.
- **Waiting for lease expiry after a clean shutdown**: slows handover for no
safety benefit.
## Benefits
- One place enforces pause-at-outcome, final-only-after-commit and fail-closed
readiness.
- Every failure mode maps to a defined phase.
- The process can be killed or terminated with verifiable durable results.
## Costs
- Ticks stall during commits: about 7 ms locally per batch in Release (M7.5), and
longer if storage is slow.
- The stdin feed is development plumbing, not a network protocol.
- There are no metrics endpoints yet.
## Risks
- Duplicate request detection is per host process, not durable. Client
deduplication across owners needs the gateway protocol.
- The killed-process test measures recovery after the development 4 s lease; the
production 20 s lease bounds unavailability accordingly.
- Clock steps (ADR-0060) still apply.
## Consequences
- M7.7 kills processes at every publication and ACK boundary, and verifies
recovery before readiness.
- M7.8 exposes phase, readiness, tick debt, commit lag and lease safety as probes
and metrics, and packages `evo-server`.
- A gateway must send inputs with request identities and treat only `FinalAck` as
durable.