Fix grant ledger ownership and truncated-record recovery

This commit is contained in:
dingo
2026-10-02 06:24:56 -04:00
parent c588be237c
commit 7ac587743c
6 changed files with 180 additions and 9 deletions
+10 -1
View File
@@ -63,7 +63,16 @@ changes need a new version.
ledger (`--session-grant-ledger`; `/objects/dev-session-grants.v1` in dev, outside the
object namespaces and ignored by backups) and fsynced **before** Welcome; a failed
write refuses the session and keeps the grant consumed in memory. A replacement
process loads unexpired entries into admission at startup. Corrupt content, I/O
process loads unexpired entries into admission at startup. Before any read or rewrite,
an opener obtains a nonblocking exclusive OS lock on a stable sibling `.lock` file
(POSIX flock / Windows share-denied handle), held until the append handle closes.
A competing opener fails before mutation, including failed duplicate server starts.
The lock file is never removed during normal operation: replacing it could split
ownership. OS process exit releases the lock; no stale-lock deletion is needed.
Records must end with LF and contain at most 37 bytes before it. Incomplete tails
and oversized records are rejected before compaction, leaving ledger bytes intact.
The text format has no checksum: syntactically valid corruption or removal of entire
records is not detected; this is not a tamper-evident ledger. Corrupt content, I/O
failure or more than 65536 entries fail closed. The ledger does not survive a
restore-to-bundle (a rewind) or a lost volume; grants expire within an hour.
+30
View File
@@ -3564,3 +3564,33 @@ drained with `exit 0`.
Not done: the reviewer's own reproduction binaries target the removed `send` API and
were not rebuilt; equivalent regressions above replace them. The `ci-client` Docker
image and Gitea pipeline were not rerun here, and no cluster operation was performed.
## 2026-10-02 — consumed-grant ledger follow-up corrections
Resolved the two c588be2 review findings. Ledger open now acquires a nonblocking
exclusive lifetime lock on a stable sibling file before reading or compacting
(POSIX flock; Windows exclusive-sharing handle). Duplicate starts on either the
same port or a different port cannot replace the active ledger. The parser caps
records at 37 bytes before LF and rejects every nonempty incomplete final record
before compaction, preserving the original bytes. No ledger format migration or
simulation/protocol version change; valid existing files remain compatible.
Regression evidence: competing SessionServer starts fail with a ledger error;
the original server subsequently admits a grant, restarts, and rejects its replay.
Direct competing opens return busy. Every nonempty truncation of a valid final
record (including shortened numeric expiries) and an oversized line is rejected
without changing the file or creating a compaction temporary. Complete/empty files
and existing expired-entry compaction remain accepted.
Executed: Ubuntu GCC14 network CTest 1/1; Clang19 ASan/UBSan network CTest 1/1;
Windows MSVC network preset configure/build/CTest 1/1; clang-tidy19 on the changed
implementation and session tests with warnings as errors, no owned diagnostics;
clang-format19 on changed C++ files; git diff --check. The first restricted Windows
Ninja invocation stalled without compiling; its task-owned build processes were
stopped, then the same preset passed outside the restricted process environment.
Not rerun: Docker/Gitea, full PostgreSQL process suite, fuzzing, TSan, or live cluster
operations. No deployment performed. The text ledger detects malformed framing,
not syntactically valid corruption or deletion of entire records; ADR-0083 now states
that integrity limit explicitly. Client event-list bounds remain M9.8 work.
@@ -41,6 +41,12 @@ The Deployment mounts it optionally (0440). Consumed grants are recorded in
`/objects/dev-session-grants.v1` on the region's persistent volume (fsynced before
each Welcome), so a restarted pod refuses grants its predecessor used. Corrupt or
unwritable ledgers fail closed (`error session-grant-ledger`, or sessions refused).
A sibling `.lock` file enforces one ledger owner across startup and compaction.
Do not delete or replace this lock file while a server might be running; its
presence does not mean a stale lock. Process exit releases the OS lock automatically.
A competing startup fails before modifying the ledger. Incomplete final records and
oversized lines are refused without rewriting; investigate the storage failure
rather than deleting grant history to force startup.
Rotation: create new credentials, replace the Secret, restart evo-region and
redistribute `dev-ca.pem`. Old grants stop verifying with the new key.