ADR-0064: RegionHost<State> (environmental and lineage profiles) is a single-owner, single-threaded state machine advanced by step(now): acquire the fenced lease and recover the verified head, run, resolve ambiguous commits, drain and stop, or quarantine. Admission is provisional and bounded (64 inputs, 64 KiB) with backpressure; FinalAck and FinalOutcome are emitted only after a confirmed commit, and inputs that will never be acknowledged are reported with a reason. Ticks follow the caller's scheduling clock with bounded catch-up and measured tick debt, never invented inputs. A death seals the batch and commits in the same step, so no dependent tick uses an uncommitted outcome. Storage outages pause and retry, unknown outcomes resolve by exact bytes, fencing discards unconfirmed work and re-acquires; readiness fails closed. Drain commits pending ticks and releases the lease. evo-server hosts one region with a bounded stdin input feed, stdout event lines, a readiness file and SIGTERM drain (exit 0/3/4). Validated: Windows CTest 58/58; PostgreSQL suites under GCC Release, clang-tidy and ASan/UBSan including real-process SIGTERM drain and SIGKILL with successor recovery; four host mutations each caught. No gateway, metrics, Docker/Gitea CI or deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
119 lines
5.6 KiB
Markdown
119 lines
5.6 KiB
Markdown
# ADR-0064: Bounded authoritative region host
|
||
|
||
## Status
|
||
Accepted for M7.6, 2026-09-28. It composes ADR-0060 (fencing), ADR-0061/0062
|
||
(checkpoints) and ADR-0063 (batch outcomes). It adds no gateway, network listener,
|
||
metrics endpoint or deployment; those are M7.8 and later.
|
||
|
||
## Context
|
||
The persistence APIs exist, but nothing yet owns a region over time. ADR-0058
|
||
requires:
|
||
- one simulation owner, with simulated and committed ticks both visible;
|
||
- bounded backlogs, with backpressure before admission;
|
||
- a whole-region pause at any irreversible outcome until durable confirmation;
|
||
- bounded drain, and readiness that fails closed.
|
||
|
||
Nothing may treat an uncommitted death as fact.
|
||
|
||
## Decision
|
||
**`RegionHost<State>`** (`src/server`, Linux), for `EnvironmentalCommunity` and
|
||
`LineageRegion`, is a single-threaded state machine advanced by `step(now)`:
|
||
- *acquiring*: take the fenced lease, then recover and verify the committed head.
|
||
The recovered quantum and frame must match the configuration.
|
||
- *running*: lease upkeep, bounded ticks, and commits.
|
||
- *resolving*: an ambiguous commit is resolved before anything else.
|
||
- *draining*, then *stopped*; or *quarantined*.
|
||
|
||
All I/O runs on the caller's thread within the LeasePolicy deadlines. While a
|
||
commit runs, the region is stopped.
|
||
|
||
**Admission** is provisional:
|
||
- The backlog holds at most 64 inputs and 64 KiB (88 bytes are accounted per
|
||
input). When it is full, the caller gets `full` and must hold its input.
|
||
- Duplicate request identities within the host are refused.
|
||
- Inputs are refused unless the host is ready.
|
||
|
||
An input becomes final only when the commit containing its tick is confirmed.
|
||
The host then emits `FinalAck(request, tick, commit_sequence)`; permanent
|
||
outcomes are emitted as `FinalOutcome`. Inputs that will never be acknowledged
|
||
are reported as `Dropped`, with reason refused, drained, fenced or quarantined.
|
||
After an unknown outcome, "fenced" means "unknown to this host", not "not
|
||
applied".
|
||
|
||
**Ticks**:
|
||
- Scheduling time comes from the caller, while lease deadlines use `LeaseClock`
|
||
and database time.
|
||
- The due count is (now − start) / quantum. Each step runs at most 8 catch-up
|
||
ticks, and one queued input is consumed per tick.
|
||
- An empty backlog accrues tick debt. The host never invents inputs, and debt is
|
||
measured (current and maximum) rather than skipped.
|
||
|
||
**Commits** use `RegionBatch`:
|
||
- A death seals the batch and commits in the same step, so no dependent tick runs
|
||
on an uncommitted outcome.
|
||
- Otherwise a batch flushes after `flush_interval` (default 1 s), at 64 ticks,
|
||
or on drain.
|
||
- The commit ID is the acquisition's holder token with the next sequence number
|
||
in its last 8 bytes. It is stable across retries of one batch, and unique
|
||
because holder tokens are generated per acquisition.
|
||
|
||
| Outcome | Host response |
|
||
|---|---|
|
||
| Storage failure (upload/verify/digest), transient SQL error | Stay paused and retry the same batch; `store_failures` counts it |
|
||
| Unknown | Enter *resolving*: resolve the exact prepared bytes. Committed → adopt the receipt. Absent → retry the same bytes while `may_begin_commit` holds. Conflict → quarantine |
|
||
| Fenced | Discard state, report unconfirmed inputs, re-acquire and recover from the head |
|
||
| Candidate, predecessor, head moved, conflict, receipt mismatch, unrestorable head | Quarantine, releasing the lease if still held |
|
||
|
||
**Readiness** is true only when running, recovered, not draining, and
|
||
`LeaseAuthority::may_originate` holds.
|
||
|
||
**Drain** stops admission, reports the backlog as drained, commits pending ticks,
|
||
releases the lease, then stops. A successor can therefore acquire immediately,
|
||
without waiting for expiry.
|
||
|
||
**`evo-server`** is the development process around one host:
|
||
- Tick inputs are stdin lines. It reads only while the backlog has room, so stdin
|
||
itself backpressures.
|
||
- Events are stdout lines: phase, ready, ack, outcome, dropped, stats.
|
||
- An optional readiness file mirrors readiness.
|
||
- SIGTERM or SIGINT drain within `--drain-ms`.
|
||
- Exit codes: 0 drained, 3 drain deadline passed (pending or unknown work is left
|
||
to the next owner), 4 quarantined, 2 usage.
|
||
- `--lease-profile fast` (a 4 s lease) is for development tests only.
|
||
|
||
## Alternatives Considered
|
||
- **A separate I/O thread**: would allow ticks during commits, but the contract
|
||
pauses the region at commits anyway, and extra threads violate single ownership
|
||
without a measured need.
|
||
- **Wall-clock-driven synthetic inputs**: would invent world input.
|
||
- **Acknowledging on apply**: would expose provisional state as final.
|
||
- **Waiting for lease expiry after a clean shutdown**: slows handover for no
|
||
safety benefit.
|
||
|
||
## Benefits
|
||
- One place enforces pause-at-outcome, final-only-after-commit and fail-closed
|
||
readiness.
|
||
- Every failure mode maps to a defined phase.
|
||
- The process can be killed or terminated with verifiable durable results.
|
||
|
||
## Costs
|
||
- Ticks stall during commits: about 7 ms locally per batch in Release (M7.5), and
|
||
longer if storage is slow.
|
||
- The stdin feed is development plumbing, not a network protocol.
|
||
- There are no metrics endpoints yet.
|
||
|
||
## Risks
|
||
- Duplicate request detection is per host process, not durable. Client
|
||
deduplication across owners needs the gateway protocol.
|
||
- The killed-process test measures recovery after the development 4 s lease; the
|
||
production 20 s lease bounds unavailability accordingly.
|
||
- Clock steps (ADR-0060) still apply.
|
||
|
||
## Consequences
|
||
- M7.7 kills processes at every publication and ACK boundary, and verifies
|
||
recovery before readiness.
|
||
- M7.8 exposes phase, readiness, tick debt, commit lag and lease safety as probes
|
||
and metrics, and packages `evo-server`.
|
||
- A gateway must send inputs with request identities and treat only `FinalAck` as
|
||
durable.
|