Fix grant ledger ownership and truncated-record recovery
This commit is contained in:
@@ -63,7 +63,16 @@ changes need a new version.
|
||||
ledger (`--session-grant-ledger`; `/objects/dev-session-grants.v1` in dev, outside the
|
||||
object namespaces and ignored by backups) and fsynced **before** Welcome; a failed
|
||||
write refuses the session and keeps the grant consumed in memory. A replacement
|
||||
process loads unexpired entries into admission at startup. Corrupt content, I/O
|
||||
process loads unexpired entries into admission at startup. Before any read or rewrite,
|
||||
an opener obtains a nonblocking exclusive OS lock on a stable sibling `.lock` file
|
||||
(POSIX flock / Windows share-denied handle), held until the append handle closes.
|
||||
A competing opener fails before mutation, including failed duplicate server starts.
|
||||
The lock file is never removed during normal operation: replacing it could split
|
||||
ownership. OS process exit releases the lock; no stale-lock deletion is needed.
|
||||
Records must end with LF and contain at most 37 bytes before it. Incomplete tails
|
||||
and oversized records are rejected before compaction, leaving ledger bytes intact.
|
||||
The text format has no checksum: syntactically valid corruption or removal of entire
|
||||
records is not detected; this is not a tamper-evident ledger. Corrupt content, I/O
|
||||
failure or more than 65536 entries fail closed. The ledger does not survive a
|
||||
restore-to-bundle (a rewind) or a lost volume; grants expire within an hour.
|
||||
|
||||
|
||||
@@ -3564,3 +3564,33 @@ drained with `exit 0`.
|
||||
Not done: the reviewer's own reproduction binaries target the removed `send` API and
|
||||
were not rebuilt; equivalent regressions above replace them. The `ci-client` Docker
|
||||
image and Gitea pipeline were not rerun here, and no cluster operation was performed.
|
||||
|
||||
|
||||
## 2026-10-02 — consumed-grant ledger follow-up corrections
|
||||
|
||||
Resolved the two c588be2 review findings. Ledger open now acquires a nonblocking
|
||||
exclusive lifetime lock on a stable sibling file before reading or compacting
|
||||
(POSIX flock; Windows exclusive-sharing handle). Duplicate starts on either the
|
||||
same port or a different port cannot replace the active ledger. The parser caps
|
||||
records at 37 bytes before LF and rejects every nonempty incomplete final record
|
||||
before compaction, preserving the original bytes. No ledger format migration or
|
||||
simulation/protocol version change; valid existing files remain compatible.
|
||||
|
||||
Regression evidence: competing SessionServer starts fail with a ledger error;
|
||||
the original server subsequently admits a grant, restarts, and rejects its replay.
|
||||
Direct competing opens return busy. Every nonempty truncation of a valid final
|
||||
record (including shortened numeric expiries) and an oversized line is rejected
|
||||
without changing the file or creating a compaction temporary. Complete/empty files
|
||||
and existing expired-entry compaction remain accepted.
|
||||
|
||||
Executed: Ubuntu GCC14 network CTest 1/1; Clang19 ASan/UBSan network CTest 1/1;
|
||||
Windows MSVC network preset configure/build/CTest 1/1; clang-tidy19 on the changed
|
||||
implementation and session tests with warnings as errors, no owned diagnostics;
|
||||
clang-format19 on changed C++ files; git diff --check. The first restricted Windows
|
||||
Ninja invocation stalled without compiling; its task-owned build processes were
|
||||
stopped, then the same preset passed outside the restricted process environment.
|
||||
|
||||
Not rerun: Docker/Gitea, full PostgreSQL process suite, fuzzing, TSan, or live cluster
|
||||
operations. No deployment performed. The text ledger detects malformed framing,
|
||||
not syntactically valid corruption or deletion of entire records; ADR-0083 now states
|
||||
that integrity limit explicitly. Client event-list bounds remain M9.8 work.
|
||||
|
||||
@@ -41,6 +41,12 @@ The Deployment mounts it optionally (0440). Consumed grants are recorded in
|
||||
`/objects/dev-session-grants.v1` on the region's persistent volume (fsynced before
|
||||
each Welcome), so a restarted pod refuses grants its predecessor used. Corrupt or
|
||||
unwritable ledgers fail closed (`error session-grant-ledger`, or sessions refused).
|
||||
A sibling `.lock` file enforces one ledger owner across startup and compaction.
|
||||
Do not delete or replace this lock file while a server might be running; its
|
||||
presence does not mean a stale lock. Process exit releases the OS lock automatically.
|
||||
A competing startup fails before modifying the ledger. Incomplete final records and
|
||||
oversized lines are refused without rewriting; investigate the storage failure
|
||||
rather than deleting grant history to force startup.
|
||||
|
||||
Rotation: create new credentials, replace the Secret, restart evo-region and
|
||||
redistribute `dev-ca.pem`. Old grants stop verifying with the new key.
|
||||
|
||||
@@ -14,9 +14,10 @@
|
||||
// refuses grants its predecessor already used. Append-only text lines
|
||||
// "<grant id hex> <expires_at>", flushed and fsynced before the session is welcomed.
|
||||
// Opening drops expired entries by an atomic rewrite. Corrupt content, I/O failure or
|
||||
// a full ledger fail closed. Single owner; not a general database.
|
||||
// a full ledger fail closed. Exclusive lifetime ownership via a stable .lock file; not a general
|
||||
// database.
|
||||
namespace evo::network {
|
||||
enum class LedgerError : std::uint8_t { io, corrupt, full };
|
||||
enum class LedgerError : std::uint8_t { io, corrupt, full, busy };
|
||||
class GrantLedger {
|
||||
public:
|
||||
static constexpr std::size_t max_entries = 65536;
|
||||
@@ -33,8 +34,9 @@ class GrantLedger {
|
||||
[[nodiscard]] std::expected<void, LedgerError> record(ConsumedGrant grant);
|
||||
|
||||
private:
|
||||
GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded)
|
||||
: file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {}
|
||||
struct Lock;
|
||||
GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded, std::unique_ptr<Lock> lock);
|
||||
std::unique_ptr<Lock> lock_; // Outlives the append handle; never unlink its pathname.
|
||||
std::FILE* file_;
|
||||
std::vector<ConsumedGrant> loaded_;
|
||||
std::size_t entries_;
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
#include "evo/network/grant_ledger.hpp"
|
||||
#include <cerrno>
|
||||
#include <charconv>
|
||||
#include <fstream>
|
||||
#include <string>
|
||||
@@ -7,8 +8,13 @@
|
||||
|
||||
#ifdef _WIN32
|
||||
#include <io.h>
|
||||
#ifndef NOMINMAX
|
||||
#define NOMINMAX
|
||||
#endif
|
||||
#include <windows.h>
|
||||
#else
|
||||
#include <fcntl.h>
|
||||
#include <sys/file.h>
|
||||
#include <unistd.h>
|
||||
#endif
|
||||
|
||||
@@ -50,7 +56,7 @@ std::string line(ConsumedGrant g) {
|
||||
out[static_cast<std::size_t>(i)] = digits[g.grant_id & 0x0FU];
|
||||
return out + ' ' + std::to_string(g.expires_at) + '\n';
|
||||
}
|
||||
bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line.
|
||||
bool parse(const std::string& text, ConsumedGrant& out) { // One complete bounded line.
|
||||
if (text.size() < 18 || text[16] != ' ')
|
||||
return false;
|
||||
const auto [a, e1] = std::from_chars(text.data(), text.data() + 16, out.grant_id, 16);
|
||||
@@ -61,8 +67,44 @@ bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line.
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// A separate, stable file: locking the ledger inode would not survive compaction.
|
||||
struct GrantLedger::Lock {
|
||||
#ifdef _WIN32
|
||||
HANDLE handle{INVALID_HANDLE_VALUE};
|
||||
~Lock() {
|
||||
if (handle != INVALID_HANDLE_VALUE)
|
||||
CloseHandle(handle);
|
||||
}
|
||||
#else
|
||||
int descriptor{-1};
|
||||
~Lock() {
|
||||
if (descriptor >= 0)
|
||||
::close(descriptor);
|
||||
}
|
||||
#endif
|
||||
};
|
||||
|
||||
GrantLedger::GrantLedger(std::FILE* file, std::vector<ConsumedGrant> loaded,
|
||||
std::unique_ptr<Lock> lock)
|
||||
: lock_{std::move(lock)}, file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {}
|
||||
|
||||
std::expected<std::unique_ptr<GrantLedger>, LedgerError>
|
||||
GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) {
|
||||
auto lock = std::make_unique<Lock>();
|
||||
const auto lock_path = std::filesystem::path{path}.concat(".lock");
|
||||
#ifdef _WIN32
|
||||
lock->handle = CreateFileW(lock_path.c_str(), GENERIC_READ | GENERIC_WRITE, 0, nullptr,
|
||||
OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr);
|
||||
if (lock->handle == INVALID_HANDLE_VALUE)
|
||||
return std::unexpected(GetLastError() == ERROR_SHARING_VIOLATION ? LedgerError::busy
|
||||
: LedgerError::io);
|
||||
#else
|
||||
lock->descriptor = ::open(lock_path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0600);
|
||||
if (lock->descriptor < 0)
|
||||
return std::unexpected(LedgerError::io);
|
||||
if (::flock(lock->descriptor, LOCK_EX | LOCK_NB) != 0)
|
||||
return std::unexpected(errno == EWOULDBLOCK ? LedgerError::busy : LedgerError::io);
|
||||
#endif
|
||||
std::vector<ConsumedGrant> live;
|
||||
std::error_code error;
|
||||
if (std::filesystem::exists(path, error)) {
|
||||
@@ -71,7 +113,15 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
|
||||
return std::unexpected(LedgerError::io);
|
||||
std::string text;
|
||||
std::size_t total = 0;
|
||||
while (std::getline(in, text)) {
|
||||
constexpr std::size_t max_line_bytes = 16 + 1 + 20; // hex id, space, u64 expiry.
|
||||
char byte{};
|
||||
while (in.get(byte)) {
|
||||
if (byte != '\n') {
|
||||
if (text.size() == max_line_bytes)
|
||||
return std::unexpected(LedgerError::corrupt);
|
||||
text.push_back(byte);
|
||||
continue;
|
||||
}
|
||||
ConsumedGrant g;
|
||||
if (!parse(text, g))
|
||||
return std::unexpected(LedgerError::corrupt); // Never guess: fail closed.
|
||||
@@ -79,9 +129,12 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
|
||||
return std::unexpected(LedgerError::full);
|
||||
if (g.expires_at > now_seconds)
|
||||
live.push_back(g);
|
||||
text.clear();
|
||||
}
|
||||
if (!in.eof())
|
||||
if (in.bad() || !in.eof())
|
||||
return std::unexpected(LedgerError::io);
|
||||
if (!text.empty())
|
||||
return std::unexpected(LedgerError::corrupt); // Unterminated record: never compact.
|
||||
} else if (error) {
|
||||
return std::unexpected(LedgerError::io);
|
||||
}
|
||||
@@ -104,7 +157,7 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds)
|
||||
std::FILE* file = open_file(path, "ab");
|
||||
if (file == nullptr)
|
||||
return std::unexpected(LedgerError::io);
|
||||
return std::unique_ptr<GrantLedger>(new GrantLedger(file, std::move(live)));
|
||||
return std::unique_ptr<GrantLedger>(new GrantLedger(file, std::move(live), std::move(lock)));
|
||||
}
|
||||
|
||||
GrantLedger::~GrantLedger() {
|
||||
|
||||
@@ -81,6 +81,7 @@ struct Harness {
|
||||
server.reset();
|
||||
std::error_code ignored;
|
||||
std::filesystem::remove(ledger, ignored);
|
||||
std::filesystem::remove(std::filesystem::path{ledger}.concat(".lock"), ignored);
|
||||
}
|
||||
Harness(const Harness&) = delete;
|
||||
Harness& operator=(const Harness&) = delete;
|
||||
@@ -519,6 +520,74 @@ void grants_stay_consumed_across_restarts() {
|
||||
std::filesystem::remove(path, ignored);
|
||||
}
|
||||
|
||||
// Follow-up review: a duplicate startup must not replace an active owner's ledger.
|
||||
void ledger_exclusive_owner() {
|
||||
Harness h;
|
||||
h.tick();
|
||||
SessionServerConfig duplicate;
|
||||
duplicate.credentials = h.pki.server;
|
||||
duplicate.admission.world = identity;
|
||||
duplicate.admission.region = region;
|
||||
duplicate.grant_ledger = h.ledger;
|
||||
for (const auto port : {h.port, std::uint16_t{0}}) {
|
||||
duplicate.listen = "127.0.0.1:" + std::to_string(port);
|
||||
CHECK(refused(SessionServer::create(duplicate, grant_key(), now_seconds),
|
||||
SessionServerError::ledger));
|
||||
}
|
||||
CHECK(refused(GrantLedger::open(h.ledger, now_seconds), LedgerError::busy));
|
||||
// Written AFTER the failed duplicate starts, then recovered through real TLS.
|
||||
auto first = h.client(grants(7, 1, 100));
|
||||
CHECK(h.until([&] { return first->phase() == SessionPhase::established; }));
|
||||
h.clients.clear();
|
||||
first.reset();
|
||||
h.server.reset();
|
||||
h.facts.epoch = 2;
|
||||
h.server = h.make_server(h.port, 4, std::chrono::milliseconds{3000});
|
||||
h.tick();
|
||||
auto replay = h.client(grants(7, 1, 100));
|
||||
std::vector<ClientEvent> events;
|
||||
CHECK(h.until([&] {
|
||||
h.collect(*replay, events);
|
||||
return has(events, ClientEvent::Kind::closed);
|
||||
}));
|
||||
CHECK(last(events, ClientEvent::Kind::closed)->reason == CloseReason::unauthenticated);
|
||||
}
|
||||
|
||||
void ledger_incomplete_records() {
|
||||
const auto path = fresh_ledger();
|
||||
const std::string prefix = "0000000000000002 1000\n";
|
||||
const std::string record = "0000000000000001 1000\n";
|
||||
const auto reject_unchanged = [&](const std::string& bytes) {
|
||||
{
|
||||
std::ofstream out{path, std::ios::binary | std::ios::trunc};
|
||||
out << bytes;
|
||||
CHECK(out.good());
|
||||
}
|
||||
CHECK(refused(GrantLedger::open(path, 100), LedgerError::corrupt));
|
||||
std::ifstream in{path, std::ios::binary};
|
||||
const std::string after{std::istreambuf_iterator<char>{in}, {}};
|
||||
CHECK(after == bytes);
|
||||
CHECK(!std::filesystem::exists(std::filesystem::path{path}.concat(".tmp")));
|
||||
};
|
||||
// All nonempty partial tails, including a plausible but expired decimal prefix.
|
||||
for (std::size_t length = 1; length < record.size(); ++length)
|
||||
reject_unchanged(prefix + record.substr(0, length));
|
||||
reject_unchanged(prefix + std::string(4096, '1') + "\n");
|
||||
// A complete file (including empty) remains valid; failed opens released the lock.
|
||||
for (const auto& bytes : {std::string{}, prefix, prefix + record}) {
|
||||
{
|
||||
std::ofstream out{path, std::ios::binary | std::ios::trunc};
|
||||
out << bytes;
|
||||
}
|
||||
const auto ledger = GrantLedger::open(path, 100);
|
||||
CHECK(ledger);
|
||||
CHECK((*ledger)->loaded().size() == (bytes.empty() ? 0 : bytes == prefix ? 1 : 2));
|
||||
}
|
||||
std::error_code ignored;
|
||||
std::filesystem::remove(path, ignored);
|
||||
std::filesystem::remove(std::filesystem::path{path}.concat(".lock"), ignored);
|
||||
}
|
||||
|
||||
// Review M9.4-M9.7 #4: liveness after Welcome, both directions, without spurious drops.
|
||||
void liveness() {
|
||||
const auto fast_server = [](SessionServerConfig& c) {
|
||||
@@ -596,6 +665,8 @@ void liveness() {
|
||||
} // namespace
|
||||
|
||||
void session_tests() {
|
||||
ledger_exclusive_owner();
|
||||
ledger_incomplete_records();
|
||||
connect_view_intents_and_outcomes();
|
||||
duplicate_capacity_and_authentication();
|
||||
stale_epoch_restart_and_reconnect();
|
||||
|
||||
Reference in New Issue
Block a user