diff --git a/docs/adr/0083-dev-region-sessions.md b/docs/adr/0083-dev-region-sessions.md index 0a19537..81dcbb8 100644 --- a/docs/adr/0083-dev-region-sessions.md +++ b/docs/adr/0083-dev-region-sessions.md @@ -63,7 +63,16 @@ changes need a new version. ledger (`--session-grant-ledger`; `/objects/dev-session-grants.v1` in dev, outside the object namespaces and ignored by backups) and fsynced **before** Welcome; a failed write refuses the session and keeps the grant consumed in memory. A replacement -process loads unexpired entries into admission at startup. Corrupt content, I/O +process loads unexpired entries into admission at startup. Before any read or rewrite, +an opener obtains a nonblocking exclusive OS lock on a stable sibling `.lock` file +(POSIX flock / Windows share-denied handle), held until the append handle closes. +A competing opener fails before mutation, including failed duplicate server starts. +The lock file is never removed during normal operation: replacing it could split +ownership. OS process exit releases the lock; no stale-lock deletion is needed. +Records must end with LF and contain at most 37 bytes before it. Incomplete tails +and oversized records are rejected before compaction, leaving ledger bytes intact. +The text format has no checksum: syntactically valid corruption or removal of entire +records is not detected; this is not a tamper-evident ledger. Corrupt content, I/O failure or more than 65536 entries fail closed. The ledger does not survive a restore-to-bundle (a rewind) or a lost volume; grants expire within an hour. diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 059050a..a0d23b3 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -3564,3 +3564,33 @@ drained with `exit 0`. Not done: the reviewer's own reproduction binaries target the removed `send` API and were not rebuilt; equivalent regressions above replace them. The `ci-client` Docker image and Gitea pipeline were not rerun here, and no cluster operation was performed. + + +## 2026-10-02 — consumed-grant ledger follow-up corrections + +Resolved the two c588be2 review findings. Ledger open now acquires a nonblocking +exclusive lifetime lock on a stable sibling file before reading or compacting +(POSIX flock; Windows exclusive-sharing handle). Duplicate starts on either the +same port or a different port cannot replace the active ledger. The parser caps +records at 37 bytes before LF and rejects every nonempty incomplete final record +before compaction, preserving the original bytes. No ledger format migration or +simulation/protocol version change; valid existing files remain compatible. + +Regression evidence: competing SessionServer starts fail with a ledger error; +the original server subsequently admits a grant, restarts, and rejects its replay. +Direct competing opens return busy. Every nonempty truncation of a valid final +record (including shortened numeric expiries) and an oversized line is rejected +without changing the file or creating a compaction temporary. Complete/empty files +and existing expired-entry compaction remain accepted. + +Executed: Ubuntu GCC14 network CTest 1/1; Clang19 ASan/UBSan network CTest 1/1; +Windows MSVC network preset configure/build/CTest 1/1; clang-tidy19 on the changed +implementation and session tests with warnings as errors, no owned diagnostics; +clang-format19 on changed C++ files; git diff --check. The first restricted Windows +Ninja invocation stalled without compiling; its task-owned build processes were +stopped, then the same preset passed outside the restricted process environment. + +Not rerun: Docker/Gitea, full PostgreSQL process suite, fuzzing, TSan, or live cluster +operations. No deployment performed. The text ledger detects malformed framing, +not syntactically valid corruption or deletion of entire records; ADR-0083 now states +that integrity limit explicitly. Client event-list bounds remain M9.8 work. diff --git a/docs/networking/DEV_SESSION_OPERATIONS.md b/docs/networking/DEV_SESSION_OPERATIONS.md index 5981ffe..268e671 100644 --- a/docs/networking/DEV_SESSION_OPERATIONS.md +++ b/docs/networking/DEV_SESSION_OPERATIONS.md @@ -41,6 +41,12 @@ The Deployment mounts it optionally (0440). Consumed grants are recorded in `/objects/dev-session-grants.v1` on the region's persistent volume (fsynced before each Welcome), so a restarted pod refuses grants its predecessor used. Corrupt or unwritable ledgers fail closed (`error session-grant-ledger`, or sessions refused). +A sibling `.lock` file enforces one ledger owner across startup and compaction. +Do not delete or replace this lock file while a server might be running; its +presence does not mean a stale lock. Process exit releases the OS lock automatically. +A competing startup fails before modifying the ledger. Incomplete final records and +oversized lines are refused without rewriting; investigate the storage failure +rather than deleting grant history to force startup. Rotation: create new credentials, replace the Secret, restart evo-region and redistribute `dev-ca.pem`. Old grants stop verifying with the new key. diff --git a/include/evo/network/grant_ledger.hpp b/include/evo/network/grant_ledger.hpp index 277c85e..1a3a1d4 100644 --- a/include/evo/network/grant_ledger.hpp +++ b/include/evo/network/grant_ledger.hpp @@ -14,9 +14,10 @@ // refuses grants its predecessor already used. Append-only text lines // " ", flushed and fsynced before the session is welcomed. // Opening drops expired entries by an atomic rewrite. Corrupt content, I/O failure or -// a full ledger fail closed. Single owner; not a general database. +// a full ledger fail closed. Exclusive lifetime ownership via a stable .lock file; not a general +// database. namespace evo::network { -enum class LedgerError : std::uint8_t { io, corrupt, full }; +enum class LedgerError : std::uint8_t { io, corrupt, full, busy }; class GrantLedger { public: static constexpr std::size_t max_entries = 65536; @@ -33,8 +34,9 @@ class GrantLedger { [[nodiscard]] std::expected record(ConsumedGrant grant); private: - GrantLedger(std::FILE* file, std::vector loaded) - : file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {} + struct Lock; + GrantLedger(std::FILE* file, std::vector loaded, std::unique_ptr lock); + std::unique_ptr lock_; // Outlives the append handle; never unlink its pathname. std::FILE* file_; std::vector loaded_; std::size_t entries_; diff --git a/src/network/session/grant_ledger.cpp b/src/network/session/grant_ledger.cpp index 68d95d6..09d8b33 100644 --- a/src/network/session/grant_ledger.cpp +++ b/src/network/session/grant_ledger.cpp @@ -1,4 +1,5 @@ #include "evo/network/grant_ledger.hpp" +#include #include #include #include @@ -7,8 +8,13 @@ #ifdef _WIN32 #include +#ifndef NOMINMAX +#define NOMINMAX +#endif +#include #else #include +#include #include #endif @@ -50,7 +56,7 @@ std::string line(ConsumedGrant g) { out[static_cast(i)] = digits[g.grant_id & 0x0FU]; return out + ' ' + std::to_string(g.expires_at) + '\n'; } -bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line. +bool parse(const std::string& text, ConsumedGrant& out) { // One complete bounded line. if (text.size() < 18 || text[16] != ' ') return false; const auto [a, e1] = std::from_chars(text.data(), text.data() + 16, out.grant_id, 16); @@ -61,8 +67,44 @@ bool parse(const std::string& text, ConsumedGrant& out) { // One getline() line. } } // namespace +// A separate, stable file: locking the ledger inode would not survive compaction. +struct GrantLedger::Lock { +#ifdef _WIN32 + HANDLE handle{INVALID_HANDLE_VALUE}; + ~Lock() { + if (handle != INVALID_HANDLE_VALUE) + CloseHandle(handle); + } +#else + int descriptor{-1}; + ~Lock() { + if (descriptor >= 0) + ::close(descriptor); + } +#endif +}; + +GrantLedger::GrantLedger(std::FILE* file, std::vector loaded, + std::unique_ptr lock) + : lock_{std::move(lock)}, file_{file}, loaded_{std::move(loaded)}, entries_{loaded_.size()} {} + std::expected, LedgerError> GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) { + auto lock = std::make_unique(); + const auto lock_path = std::filesystem::path{path}.concat(".lock"); +#ifdef _WIN32 + lock->handle = CreateFileW(lock_path.c_str(), GENERIC_READ | GENERIC_WRITE, 0, nullptr, + OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, nullptr); + if (lock->handle == INVALID_HANDLE_VALUE) + return std::unexpected(GetLastError() == ERROR_SHARING_VIOLATION ? LedgerError::busy + : LedgerError::io); +#else + lock->descriptor = ::open(lock_path.c_str(), O_RDWR | O_CREAT | O_CLOEXEC, 0600); + if (lock->descriptor < 0) + return std::unexpected(LedgerError::io); + if (::flock(lock->descriptor, LOCK_EX | LOCK_NB) != 0) + return std::unexpected(errno == EWOULDBLOCK ? LedgerError::busy : LedgerError::io); +#endif std::vector live; std::error_code error; if (std::filesystem::exists(path, error)) { @@ -71,7 +113,15 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) return std::unexpected(LedgerError::io); std::string text; std::size_t total = 0; - while (std::getline(in, text)) { + constexpr std::size_t max_line_bytes = 16 + 1 + 20; // hex id, space, u64 expiry. + char byte{}; + while (in.get(byte)) { + if (byte != '\n') { + if (text.size() == max_line_bytes) + return std::unexpected(LedgerError::corrupt); + text.push_back(byte); + continue; + } ConsumedGrant g; if (!parse(text, g)) return std::unexpected(LedgerError::corrupt); // Never guess: fail closed. @@ -79,9 +129,12 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) return std::unexpected(LedgerError::full); if (g.expires_at > now_seconds) live.push_back(g); + text.clear(); } - if (!in.eof()) + if (in.bad() || !in.eof()) return std::unexpected(LedgerError::io); + if (!text.empty()) + return std::unexpected(LedgerError::corrupt); // Unterminated record: never compact. } else if (error) { return std::unexpected(LedgerError::io); } @@ -104,7 +157,7 @@ GrantLedger::open(const std::filesystem::path& path, std::uint64_t now_seconds) std::FILE* file = open_file(path, "ab"); if (file == nullptr) return std::unexpected(LedgerError::io); - return std::unique_ptr(new GrantLedger(file, std::move(live))); + return std::unique_ptr(new GrantLedger(file, std::move(live), std::move(lock))); } GrantLedger::~GrantLedger() { diff --git a/tests/network_session.cpp b/tests/network_session.cpp index 81bd090..5dbe6ad 100644 --- a/tests/network_session.cpp +++ b/tests/network_session.cpp @@ -81,6 +81,7 @@ struct Harness { server.reset(); std::error_code ignored; std::filesystem::remove(ledger, ignored); + std::filesystem::remove(std::filesystem::path{ledger}.concat(".lock"), ignored); } Harness(const Harness&) = delete; Harness& operator=(const Harness&) = delete; @@ -519,6 +520,74 @@ void grants_stay_consumed_across_restarts() { std::filesystem::remove(path, ignored); } +// Follow-up review: a duplicate startup must not replace an active owner's ledger. +void ledger_exclusive_owner() { + Harness h; + h.tick(); + SessionServerConfig duplicate; + duplicate.credentials = h.pki.server; + duplicate.admission.world = identity; + duplicate.admission.region = region; + duplicate.grant_ledger = h.ledger; + for (const auto port : {h.port, std::uint16_t{0}}) { + duplicate.listen = "127.0.0.1:" + std::to_string(port); + CHECK(refused(SessionServer::create(duplicate, grant_key(), now_seconds), + SessionServerError::ledger)); + } + CHECK(refused(GrantLedger::open(h.ledger, now_seconds), LedgerError::busy)); + // Written AFTER the failed duplicate starts, then recovered through real TLS. + auto first = h.client(grants(7, 1, 100)); + CHECK(h.until([&] { return first->phase() == SessionPhase::established; })); + h.clients.clear(); + first.reset(); + h.server.reset(); + h.facts.epoch = 2; + h.server = h.make_server(h.port, 4, std::chrono::milliseconds{3000}); + h.tick(); + auto replay = h.client(grants(7, 1, 100)); + std::vector events; + CHECK(h.until([&] { + h.collect(*replay, events); + return has(events, ClientEvent::Kind::closed); + })); + CHECK(last(events, ClientEvent::Kind::closed)->reason == CloseReason::unauthenticated); +} + +void ledger_incomplete_records() { + const auto path = fresh_ledger(); + const std::string prefix = "0000000000000002 1000\n"; + const std::string record = "0000000000000001 1000\n"; + const auto reject_unchanged = [&](const std::string& bytes) { + { + std::ofstream out{path, std::ios::binary | std::ios::trunc}; + out << bytes; + CHECK(out.good()); + } + CHECK(refused(GrantLedger::open(path, 100), LedgerError::corrupt)); + std::ifstream in{path, std::ios::binary}; + const std::string after{std::istreambuf_iterator{in}, {}}; + CHECK(after == bytes); + CHECK(!std::filesystem::exists(std::filesystem::path{path}.concat(".tmp"))); + }; + // All nonempty partial tails, including a plausible but expired decimal prefix. + for (std::size_t length = 1; length < record.size(); ++length) + reject_unchanged(prefix + record.substr(0, length)); + reject_unchanged(prefix + std::string(4096, '1') + "\n"); + // A complete file (including empty) remains valid; failed opens released the lock. + for (const auto& bytes : {std::string{}, prefix, prefix + record}) { + { + std::ofstream out{path, std::ios::binary | std::ios::trunc}; + out << bytes; + } + const auto ledger = GrantLedger::open(path, 100); + CHECK(ledger); + CHECK((*ledger)->loaded().size() == (bytes.empty() ? 0 : bytes == prefix ? 1 : 2)); + } + std::error_code ignored; + std::filesystem::remove(path, ignored); + std::filesystem::remove(std::filesystem::path{path}.concat(".lock"), ignored); +} + // Review M9.4-M9.7 #4: liveness after Welcome, both directions, without spurious drops. void liveness() { const auto fast_server = [](SessionServerConfig& c) { @@ -596,6 +665,8 @@ void liveness() { } // namespace void session_tests() { + ledger_exclusive_owner(); + ledger_incomplete_records(); connect_view_intents_and_outcomes(); duplicate_capacity_and_authentication(); stale_epoch_restart_and_reconnect();