Files
aevolution/docs/adr/0064-region-host.md
T
2026-10-08 12:08:53 -04:00

5.6 KiB
Raw Blame History

[Reviewed]

ADR-0064: Bounded authoritative region host

Status

Accepted for M7.6, 2026-09-28. It composes ADR-0060 (fencing), ADR-0061/0062 (checkpoints) and ADR-0063 (batch outcomes). It adds no gateway, network listener, metrics endpoint or deployment; those are M7.8 and later.

Context

The persistence APIs exist, but nothing yet owns a region over time. ADR-0058 requires:

  • one simulation owner, with simulated and committed ticks both visible;
  • bounded backlogs, with backpressure before admission;
  • a whole-region pause at any irreversible outcome until durable confirmation;
  • bounded drain, and readiness that fails closed.

Nothing may treat an uncommitted death as fact.

Decision

RegionHost<State> (src/server, Linux), for EnvironmentalCommunity and LineageRegion, is a single-threaded state machine advanced by step(now):

  • acquiring: take the fenced lease, then recover and verify the committed head. The recovered quantum and frame must match the configuration.
  • running: lease upkeep, bounded ticks, and commits.
  • resolving: an ambiguous commit is resolved before anything else.
  • draining, then stopped; or quarantined.

All I/O runs on the caller's thread within the LeasePolicy deadlines. While a commit runs, the region is stopped.

Admission is provisional:

  • The backlog holds at most 64 inputs and 64 KiB (88 bytes are accounted per input). When it is full, the caller gets full and must hold its input.
  • Duplicate request identities within the host are refused.
  • Inputs are refused unless the host is ready.

An input becomes final only when the commit containing its tick is confirmed. The host then emits FinalAck(request, tick, commit_sequence); permanent outcomes are emitted as FinalOutcome. Inputs that will never be acknowledged are reported as Dropped, with reason refused, drained, fenced or quarantined. After an unknown outcome, "fenced" means "unknown to this host", not "not applied".

Ticks:

  • Scheduling time comes from the caller, while lease deadlines use LeaseClock and database time.
  • The due count is (now − start) / quantum. Each step runs at most 8 catch-up ticks, and one queued input is consumed per tick.
  • An empty backlog accrues tick debt. The host never invents inputs, and debt is measured (current and maximum) rather than skipped.

Commits use RegionBatch:

  • A death seals the batch and commits in the same step, so no dependent tick runs on an uncommitted outcome.
  • Otherwise a batch flushes after flush_interval (default 1 s), at 64 ticks, or on drain.
  • The commit ID is the acquisition's holder token with the next sequence number in its last 8 bytes. It is stable across retries of one batch, and unique because holder tokens are generated per acquisition.
Outcome Host response
Storage failure (upload/verify/digest), transient SQL error Stay paused and retry the same batch; store_failures counts it
Unknown Enter resolving: resolve the exact prepared bytes. Committed → adopt the receipt. Absent → retry the same bytes while may_begin_commit holds. Conflict → quarantine
Fenced Discard state, report unconfirmed inputs, re-acquire and recover from the head
Candidate, predecessor, head moved, conflict, receipt mismatch, unrestorable head Quarantine, releasing the lease if still held

Readiness is true only when running, recovered, not draining, and LeaseAuthority::may_originate holds.

Drain stops admission, reports the backlog as drained, commits pending ticks, releases the lease, then stops. A successor can therefore acquire immediately, without waiting for expiry.

evo-server is the development process around one host:

  • Tick inputs are stdin lines. It reads only while the backlog has room, so stdin itself backpressures.
  • Events are stdout lines: phase, ready, ack, outcome, dropped, stats.
  • An optional readiness file mirrors readiness.
  • SIGTERM or SIGINT drain within --drain-ms.
  • Exit codes: 0 drained, 3 drain deadline passed (pending or unknown work is left to the next owner), 4 quarantined, 2 usage.
  • --lease-profile fast (a 4 s lease) is for development tests only.

Alternatives Considered

  • A separate I/O thread: would allow ticks during commits, but the contract pauses the region at commits anyway, and extra threads violate single ownership without a measured need.
  • Wall-clock-driven synthetic inputs: would invent world input.
  • Acknowledging on apply: would expose provisional state as final.
  • Waiting for lease expiry after a clean shutdown: slows handover for no safety benefit.

Benefits

  • One place enforces pause-at-outcome, final-only-after-commit and fail-closed readiness.
  • Every failure mode maps to a defined phase.
  • The process can be killed or terminated with verifiable durable results.

Costs

  • Ticks stall during commits: about 7 ms locally per batch in Release (M7.5), and longer if storage is slow.
  • The stdin feed is development plumbing, not a network protocol.
  • There are no metrics endpoints yet.

Risks

  • Duplicate request detection is per host process, not durable. Client deduplication across owners needs the gateway protocol.
  • The killed-process test measures recovery after the development 4 s lease; the production 20 s lease bounds unavailability accordingly.
  • Clock steps (ADR-0060) still apply.

Consequences

  • M7.7 kills processes at every publication and ACK boundary, and verifies recovery before readiness.
  • M7.8 exposes phase, readiness, tick debt, commit lag and lease safety as probes and metrics, and packages evo-server.
  • A gateway must send inputs with request identities and treat only FinalAck as durable.