120 lines
5.6 KiB
Markdown
120 lines
5.6 KiB
Markdown
[Reviewed]
|
||
# ADR-0064: Bounded authoritative region host
|
||
|
||
## Status
|
||
Accepted for M7.6, 2026-09-28. It composes ADR-0060 (fencing), ADR-0061/0062
|
||
(checkpoints) and ADR-0063 (batch outcomes). It adds no gateway, network listener,
|
||
metrics endpoint or deployment; those are M7.8 and later.
|
||
|
||
## Context
|
||
The persistence APIs exist, but nothing yet owns a region over time. ADR-0058
|
||
requires:
|
||
- one simulation owner, with simulated and committed ticks both visible;
|
||
- bounded backlogs, with backpressure before admission;
|
||
- a whole-region pause at any irreversible outcome until durable confirmation;
|
||
- bounded drain, and readiness that fails closed.
|
||
|
||
Nothing may treat an uncommitted death as fact.
|
||
|
||
## Decision
|
||
**`RegionHost<State>`** (`src/server`, Linux), for `EnvironmentalCommunity` and
|
||
`LineageRegion`, is a single-threaded state machine advanced by `step(now)`:
|
||
- *acquiring*: take the fenced lease, then recover and verify the committed head.
|
||
The recovered quantum and frame must match the configuration.
|
||
- *running*: lease upkeep, bounded ticks, and commits.
|
||
- *resolving*: an ambiguous commit is resolved before anything else.
|
||
- *draining*, then *stopped*; or *quarantined*.
|
||
|
||
All I/O runs on the caller's thread within the LeasePolicy deadlines. While a
|
||
commit runs, the region is stopped.
|
||
|
||
**Admission** is provisional:
|
||
- The backlog holds at most 64 inputs and 64 KiB (88 bytes are accounted per
|
||
input). When it is full, the caller gets `full` and must hold its input.
|
||
- Duplicate request identities within the host are refused.
|
||
- Inputs are refused unless the host is ready.
|
||
|
||
An input becomes final only when the commit containing its tick is confirmed.
|
||
The host then emits `FinalAck(request, tick, commit_sequence)`; permanent
|
||
outcomes are emitted as `FinalOutcome`. Inputs that will never be acknowledged
|
||
are reported as `Dropped`, with reason refused, drained, fenced or quarantined.
|
||
After an unknown outcome, "fenced" means "unknown to this host", not "not
|
||
applied".
|
||
|
||
**Ticks**:
|
||
- Scheduling time comes from the caller, while lease deadlines use `LeaseClock`
|
||
and database time.
|
||
- The due count is (now − start) / quantum. Each step runs at most 8 catch-up
|
||
ticks, and one queued input is consumed per tick.
|
||
- An empty backlog accrues tick debt. The host never invents inputs, and debt is
|
||
measured (current and maximum) rather than skipped.
|
||
|
||
**Commits** use `RegionBatch`:
|
||
- A death seals the batch and commits in the same step, so no dependent tick runs
|
||
on an uncommitted outcome.
|
||
- Otherwise a batch flushes after `flush_interval` (default 1 s), at 64 ticks,
|
||
or on drain.
|
||
- The commit ID is the acquisition's holder token with the next sequence number
|
||
in its last 8 bytes. It is stable across retries of one batch, and unique
|
||
because holder tokens are generated per acquisition.
|
||
|
||
| Outcome | Host response |
|
||
|---|---|
|
||
| Storage failure (upload/verify/digest), transient SQL error | Stay paused and retry the same batch; `store_failures` counts it |
|
||
| Unknown | Enter *resolving*: resolve the exact prepared bytes. Committed → adopt the receipt. Absent → retry the same bytes while `may_begin_commit` holds. Conflict → quarantine |
|
||
| Fenced | Discard state, report unconfirmed inputs, re-acquire and recover from the head |
|
||
| Candidate, predecessor, head moved, conflict, receipt mismatch, unrestorable head | Quarantine, releasing the lease if still held |
|
||
|
||
**Readiness** is true only when running, recovered, not draining, and
|
||
`LeaseAuthority::may_originate` holds.
|
||
|
||
**Drain** stops admission, reports the backlog as drained, commits pending ticks,
|
||
releases the lease, then stops. A successor can therefore acquire immediately,
|
||
without waiting for expiry.
|
||
|
||
**`evo-server`** is the development process around one host:
|
||
- Tick inputs are stdin lines. It reads only while the backlog has room, so stdin
|
||
itself backpressures.
|
||
- Events are stdout lines: phase, ready, ack, outcome, dropped, stats.
|
||
- An optional readiness file mirrors readiness.
|
||
- SIGTERM or SIGINT drain within `--drain-ms`.
|
||
- Exit codes: 0 drained, 3 drain deadline passed (pending or unknown work is left
|
||
to the next owner), 4 quarantined, 2 usage.
|
||
- `--lease-profile fast` (a 4 s lease) is for development tests only.
|
||
|
||
## Alternatives Considered
|
||
- **A separate I/O thread**: would allow ticks during commits, but the contract
|
||
pauses the region at commits anyway, and extra threads violate single ownership
|
||
without a measured need.
|
||
- **Wall-clock-driven synthetic inputs**: would invent world input.
|
||
- **Acknowledging on apply**: would expose provisional state as final.
|
||
- **Waiting for lease expiry after a clean shutdown**: slows handover for no
|
||
safety benefit.
|
||
|
||
## Benefits
|
||
- One place enforces pause-at-outcome, final-only-after-commit and fail-closed
|
||
readiness.
|
||
- Every failure mode maps to a defined phase.
|
||
- The process can be killed or terminated with verifiable durable results.
|
||
|
||
## Costs
|
||
- Ticks stall during commits: about 7 ms locally per batch in Release (M7.5), and
|
||
longer if storage is slow.
|
||
- The stdin feed is development plumbing, not a network protocol.
|
||
- There are no metrics endpoints yet.
|
||
|
||
## Risks
|
||
- Duplicate request detection is per host process, not durable. Client
|
||
deduplication across owners needs the gateway protocol.
|
||
- The killed-process test measures recovery after the development 4 s lease; the
|
||
production 20 s lease bounds unavailability accordingly.
|
||
- Clock steps (ADR-0060) still apply.
|
||
|
||
## Consequences
|
||
- M7.7 kills processes at every publication and ACK boundary, and verifies
|
||
recovery before readiness.
|
||
- M7.8 exposes phase, readiness, tick debt, commit lag and lease safety as probes
|
||
and metrics, and packages `evo-server`.
|
||
- A gateway must send inputs with request identities and treat only `FinalAck` as
|
||
durable.
|