Fix grant ledger ownership and truncated-record recovery

This commit is contained in:
dingo
2026-10-02 06:24:56 -04:00
parent c588be237c
commit 7ac587743c
6 changed files with 180 additions and 9 deletions
+10 -1
View File
@@ -63,7 +63,16 @@ changes need a new version.
ledger (`--session-grant-ledger`; `/objects/dev-session-grants.v1` in dev, outside the
object namespaces and ignored by backups) and fsynced **before** Welcome; a failed
write refuses the session and keeps the grant consumed in memory. A replacement
process loads unexpired entries into admission at startup. Corrupt content, I/O
process loads unexpired entries into admission at startup. Before any read or rewrite,
an opener obtains a nonblocking exclusive OS lock on a stable sibling `.lock` file
(POSIX flock / Windows share-denied handle), held until the append handle closes.
A competing opener fails before mutation, including failed duplicate server starts.
The lock file is never removed during normal operation: replacing it could split
ownership. OS process exit releases the lock; no stale-lock deletion is needed.
Records must end with LF and contain at most 37 bytes before it. Incomplete tails
and oversized records are rejected before compaction, leaving ledger bytes intact.
The text format has no checksum: syntactically valid corruption or removal of entire
records is not detected; this is not a tamper-evident ledger. Corrupt content, I/O
failure or more than 65536 entries fail closed. The ledger does not survive a
restore-to-bundle (a rewind) or a lost volume; grants expire within an hour.
+30
View File
@@ -3564,3 +3564,33 @@ drained with `exit 0`.
Not done: the reviewer's own reproduction binaries target the removed `send` API and
were not rebuilt; equivalent regressions above replace them. The `ci-client` Docker
image and Gitea pipeline were not rerun here, and no cluster operation was performed.
## 2026-10-02 — consumed-grant ledger follow-up corrections
Resolved the two c588be2 review findings. Ledger open now acquires a nonblocking
exclusive lifetime lock on a stable sibling file before reading or compacting
(POSIX flock; Windows exclusive-sharing handle). Duplicate starts on either the
same port or a different port cannot replace the active ledger. The parser caps
records at 37 bytes before LF and rejects every nonempty incomplete final record
before compaction, preserving the original bytes. No ledger format migration or
simulation/protocol version change; valid existing files remain compatible.
Regression evidence: competing SessionServer starts fail with a ledger error;
the original server subsequently admits a grant, restarts, and rejects its replay.
Direct competing opens return busy. Every nonempty truncation of a valid final
record (including shortened numeric expiries) and an oversized line is rejected
without changing the file or creating a compaction temporary. Complete/empty files
and existing expired-entry compaction remain accepted.
Executed: Ubuntu GCC14 network CTest 1/1; Clang19 ASan/UBSan network CTest 1/1;
Windows MSVC network preset configure/build/CTest 1/1; clang-tidy19 on the changed
implementation and session tests with warnings as errors, no owned diagnostics;
clang-format19 on changed C++ files; git diff --check. The first restricted Windows
Ninja invocation stalled without compiling; its task-owned build processes were
stopped, then the same preset passed outside the restricted process environment.
Not rerun: Docker/Gitea, full PostgreSQL process suite, fuzzing, TSan, or live cluster
operations. No deployment performed. The text ledger detects malformed framing,
not syntactically valid corruption or deletion of entire records; ADR-0083 now states
that integrity limit explicitly. Client event-list bounds remain M9.8 work.
@@ -41,6 +41,12 @@ The Deployment mounts it optionally (0440). Consumed grants are recorded in
`/objects/dev-session-grants.v1` on the region's persistent volume (fsynced before
each Welcome), so a restarted pod refuses grants its predecessor used. Corrupt or
unwritable ledgers fail closed (`error session-grant-ledger`, or sessions refused).
A sibling `.lock` file enforces one ledger owner across startup and compaction.
Do not delete or replace this lock file while a server might be running; its
presence does not mean a stale lock. Process exit releases the OS lock automatically.
A competing startup fails before modifying the ledger. Incomplete final records and
oversized lines are refused without rewriting; investigate the storage failure
rather than deleting grant history to force startup.
Rotation: create new credentials, replace the Secret, restart evo-region and
redistribute `dev-ca.pem`. Old grants stop verifying with the new key.
+6 -4
View File
@@ -14,9 +14,10 @@
// refuses grants its predecessor already used. Append-only text lines
// "<grant id hex> <expires_at>", flushed and fsynced before the session is welcomed.
// Opening drops expired entries by an atomic rewrite. Corrupt content, I/O failure or
// a full ledger fail closed. Single owner; not a general database.
// a full ledger fail closed. Exclusive lifetime ownership via a stable .lock file; not a general
// database.
namespace evo::network {
enum class LedgerError : std::uint8_t { io, corrupt, full };
enum class LedgerError : std::uint8_t { io, corrupt, full, busy };
class GrantLedger {
public:
static constexpr std::size_t max_entries = 65536;
@@ -33,8 +34,9 @@ class GrantLedger {
[[nodiscard]] std::expected<void, LedgerError> record(ConsumedGrant grant);
private:
GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded)
: file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {}
struct Lock;
GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded, std::unique_ptr<Lock> lock);
std::unique_ptr<Lock> lock_; // Outlives the append handle; never unlink its pathname.
std::FILE* file_;
std::vector<ConsumedGrant> loaded_;
std::size_t entries_;
+57 -4
View File
@@ -1,4 +1,5 @@
#include "evo/network/grant_ledger.hpp"
#include <cerrno>
#include <charconv>
#include <fstream>
#include <string>
@@ -7,8 +8,13 @@
#ifdef _WIN32
#include <io.h>
#ifndef NOMINMAX
#define NOMINMAX
#endif
#include <windows.h>
#else
#include <fcntl.h>
#include <sys/file.h>
#include <unistd.h>
#endif
@@ -50,7 +56,7 @@ std::string line(ConsumedGrant g) {
out[static_cast<std::size_t>(i)] = digits[g.grant_id & 0x0FU];
return out + ' ' + std::to_string(g.expires_at) + '\n';
}
bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line.
bool parse(const std::string& text, ConsumedGrant& out) { // One complete bounded line.
if (text.size() < 18 || text[16] != ' ')
return false;
const auto [a, e1] = std::from_chars(text.data(), text.data() + 16, out.grant_id, 16);
@@ -61,8 +67,44 @@ bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line.
}
} // namespace
// A separate, stable file: locking the ledger inode would not survive compaction.
struct GrantLedger::Lock {
#ifdef _WIN32
HANDLE handle{INVALID_HANDLE_VALUE};
~Lock() {
if (handle != INVALID_HANDLE_VALUE)
CloseHandle(handle);
}
#else
int descriptor{-1};
~Lock() {
if (descriptor >= 0)
::close(descriptor);
}
#endif
};
GrantLedger::GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded,
std::unique_ptr<Lock> lock)
: lock_{std::move(lock)}, file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {}
std::expected<std::unique_ptr<GrantLedger>, LedgerError>
GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) {
auto lock = std::make_unique<Lock>();
const auto lock_path = std::filesystem::path{path}.concat(".lock");
#ifdef _WIN32
lock->handle = CreateFileW(lock_path.c_str(), GENERIC_READ | GENERIC_WRITE, 0, nullptr,
OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr);
if (lock->handle == INVALID_HANDLE_VALUE)
return std::unexpected(GetLastError() == ERROR_SHARING_VIOLATION ? LedgerError::busy
: LedgerError::io);
#else
lock->descriptor = ::open(lock_path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0600);
if (lock->descriptor < 0)
return std::unexpected(LedgerError::io);
if (::flock(lock->descriptor, LOCK_EX | LOCK_NB) != 0)
return std::unexpected(errno == EWOULDBLOCK ? LedgerError::busy : LedgerError::io);
#endif
std::vector<ConsumedGrant> live;
std::error_code error;
if (std::filesystem::exists(path, error)) {
@@ -71,7 +113,15 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
return std::unexpected(LedgerError::io);
std::string text;
std::size_t total = 0;
while (std::getline(in, text)) {
constexpr std::size_t max_line_bytes = 16 + 1 + 20; // hex id, space, u64 expiry.
char byte{};
while (in.get(byte)) {
if (byte != '\n') {
if (text.size() == max_line_bytes)
return std::unexpected(LedgerError::corrupt);
text.push_back(byte);
continue;
}
ConsumedGrant g;
if (!parse(text, g))
return std::unexpected(LedgerError::corrupt); // Never guess: fail closed.
@@ -79,9 +129,12 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
return std::unexpected(LedgerError::full);
if (g.expires_at > now_seconds)
live.push_back(g);
text.clear();
}
if (!in.eof())
if (in.bad() || !in.eof())
return std::unexpected(LedgerError::io);
if (!text.empty())
return std::unexpected(LedgerError::corrupt); // Unterminated record: never compact.
} else if (error) {
return std::unexpected(LedgerError::io);
}
@@ -104,7 +157,7 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
std::FILE* file = open_file(path, "ab");
if (file == nullptr)
return std::unexpected(LedgerError::io);
return std::unique_ptr<GrantLedger>(new GrantLedger(file, std::move(live)));
return std::unique_ptr<GrantLedger>(new GrantLedger(file, std::move(live), std::move(lock)));
}
GrantLedger::~GrantLedger() {
+71
View File
@@ -81,6 +81,7 @@ struct Harness {
server.reset();
std::error_code ignored;
std::filesystem::remove(ledger, ignored);
std::filesystem::remove(std::filesystem::path{ledger}.concat(".lock"), ignored);
}
Harness(const Harness&) = delete;
Harness& operator=(const Harness&) = delete;
@@ -519,6 +520,74 @@ void grants_stay_consumed_across_restarts() {
std::filesystem::remove(path, ignored);
}
// Follow-up review: a duplicate startup must not replace an active owner's ledger.
void ledger_exclusive_owner() {
Harness h;
h.tick();
SessionServerConfig duplicate;
duplicate.credentials = h.pki.server;
duplicate.admission.world = identity;
duplicate.admission.region = region;
duplicate.grant_ledger = h.ledger;
for (const auto port : {h.port, std::uint16_t{0}}) {
duplicate.listen = "127.0.0.1:" + std::to_string(port);
CHECK(refused(SessionServer::create(duplicate, grant_key(), now_seconds),
SessionServerError::ledger));
}
CHECK(refused(GrantLedger::open(h.ledger, now_seconds), LedgerError::busy));
// Written AFTER the failed duplicate starts, then recovered through real TLS.
auto first = h.client(grants(7, 1, 100));
CHECK(h.until([&] { return first->phase() == SessionPhase::established; }));
h.clients.clear();
first.reset();
h.server.reset();
h.facts.epoch = 2;
h.server = h.make_server(h.port, 4, std::chrono::milliseconds{3000});
h.tick();
auto replay = h.client(grants(7, 1, 100));
std::vector<ClientEvent> events;
CHECK(h.until([&] {
h.collect(*replay, events);
return has(events, ClientEvent::Kind::closed);
}));
CHECK(last(events, ClientEvent::Kind::closed)->reason == CloseReason::unauthenticated);
}
void ledger_incomplete_records() {
const auto path = fresh_ledger();
const std::string prefix = "0000000000000002 1000\n";
const std::string record = "0000000000000001 1000\n";
const auto reject_unchanged = [&](const std::string& bytes) {
{
std::ofstream out{path, std::ios::binary | std::ios::trunc};
out << bytes;
CHECK(out.good());
}
CHECK(refused(GrantLedger::open(path, 100), LedgerError::corrupt));
std::ifstream in{path, std::ios::binary};
const std::string after{std::istreambuf_iterator<char>{in}, {}};
CHECK(after == bytes);
CHECK(!std::filesystem::exists(std::filesystem::path{path}.concat(".tmp")));
};
// All nonempty partial tails, including a plausible but expired decimal prefix.
for (std::size_t length = 1; length < record.size(); ++length)
reject_unchanged(prefix + record.substr(0, length));
reject_unchanged(prefix + std::string(4096, '1') + "\n");
// A complete file (including empty) remains valid; failed opens released the lock.
for (const auto& bytes : {std::string{}, prefix, prefix + record}) {
{
std::ofstream out{path, std::ios::binary | std::ios::trunc};
out << bytes;
}
const auto ledger = GrantLedger::open(path, 100);
CHECK(ledger);
CHECK((*ledger)->loaded().size() == (bytes.empty() ? 0 : bytes == prefix ? 1 : 2));
}
std::error_code ignored;
std::filesystem::remove(path, ignored);
std::filesystem::remove(std::filesystem::path{path}.concat(".lock"), ignored);
}
// Review M9.4-M9.7 #4: liveness after Welcome, both directions, without spurious drops.
void liveness() {
const auto fast_server = [](SessionServerConfig& c) {
@@ -596,6 +665,8 @@ void liveness() {
} // namespace
void session_tests() {
ledger_exclusive_owner();
ledger_incomplete_records();
connect_view_intents_and_outcomes();
duplicate_capacity_and_authentication();
stale_epoch_restart_and_reconnect();