Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 4s
Foundation / checks (tsan) (push) Successful in 4s
Foundation / checks (postgres) (push) Successful in 4s
Foundation / checks (deploy) (push) Successful in 4m28s
Foundation / postgres-tools (push) Successful in 28s
Foundation / server-image (push) Successful in 4m14s
Foundation / image (push) Successful in 4m40s
With commits every flush interval the session layer still read the committed head as the simulated tick, so clients saw time advance 20 ticks at once and scheduled their actions for ticks the server had already simulated, which were refused (poor movement). Sessions now get the open batch's working tick, so welcome ticks, receipts, the simulated watermark and action admission follow what has actually been simulated. Refreshes the DEV session window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1321 lines
62 KiB
C++
1321 lines
62 KiB
C++
// M7.6 region server process (ADR-0064). Development host for one region: tick
|
|
// inputs arrive as lines on stdin (the network gateway is later work); events are
|
|
// written as lines on stdout. Linux only; requires the PostgreSQL build.
|
|
//
|
|
// Input: input <request:32 hex> <temperature_mK> <source_enabled> <bite0..4> <terminal0..1>
|
|
// Output: phase <name> epoch=<e> committed=<t> simulated=<t>
|
|
// ready <0|1>
|
|
// rejected <request> <reason> | dropped <request> <reason>
|
|
// ack <request> <tick> <commit_sequence>
|
|
// outcome <tick> <actor> <organism> <commit_sequence>
|
|
// stats max_queued=<n> max_tick_debt=<n> commits=<n> store_failures=<n>
|
|
// Exit: 0 drained, 2 usage, 3 drain deadline passed, 4 quarantined.
|
|
//
|
|
// --pg-service <file> (M8.3) takes host, hostaddr, port, database, user and CA from
|
|
// the [evolution] section of a libpq service file; anything else in it is refused.
|
|
//
|
|
// --objects-reserve-mib <n> (M8.3, default 1024) refuses new objects that would leave
|
|
// less free space on the object volume: node-local volumes do not enforce their
|
|
// requested size. A refused upload is a store failure; nothing is acknowledged.
|
|
//
|
|
// --ops-listen <numeric address:port> (M7.8) serves GET /livez (the loop is alive; says
|
|
// nothing about authority), /readyz (200 only with current authority) and /metrics
|
|
// (Prometheus text). It prints "ops listening <port>" once bound.
|
|
//
|
|
// --synthetic-inputs development-v1 (dev/stage drills only, M8.6, ADR-0073) replaces
|
|
// stdin with a deterministic driver: whenever the host is running it keeps the
|
|
// backlog fed with development_input(t) for the ticks after the committed head,
|
|
// restarting from the committed tick after every recovery. Request IDs are
|
|
// "SYN1" + region + tick, so acknowledgements can be checked across owners.
|
|
//
|
|
// Upgrade rehearsal on clones only (M8.9, ADR-0076), both requiring
|
|
// --synthetic-inputs: --synthetic-until <tick> stops the driver after that tick, so
|
|
// the region commits exactly up to it and then idles (inputs are never invented);
|
|
// --accelerate <2..1000> runs the region's scheduling clock that many times faster
|
|
// than real time (lease deadlines stay real). Never in a deployed environment.
|
|
//
|
|
// Dev sessions (M9.7, ADR-0083; builds with EVO_BUILD_NETWORK only): --session-listen
|
|
// <numeric address:port> --session-cert F --session-key F --session-grant-key F
|
|
// --session-grant-ledger F --session-environment dev serve the TLS 1.3 dev session
|
|
// protocol. The ledger durably records consumed grants (on a persistent volume), so a
|
|
// restarted process refuses them. They require
|
|
// --synthetic-inputs: an admitted intention rides the next synthetic input (content
|
|
// unchanged), so its final outcome is the input's durable acknowledgement, with no
|
|
// biological effect. The driver then looks ahead 8 ticks instead of 64. Prints
|
|
// "session listening <port>" and "session opened|closed|refused ...". Any environment
|
|
// other than dev is refused: the dev identity cannot exist elsewhere.
|
|
//
|
|
// --crash-at <point>:<n> (tests only, M7.7) SIGKILLs this process at the n-th
|
|
// occurrence of before-upload, before-transaction, after-commit or after-ack (right
|
|
// after a step's acknowledgements are written and flushed).
|
|
// --profile primitive (M11.2d) starts the creation-only profile-3 host. It requires
|
|
// an owner-provisioned checkpoint and becomes ready only after recovery. Optional
|
|
// --primitive-* flags enable the v5 new-life listener; biology ticks and genesis
|
|
// provisioning remain disabled.
|
|
#include "evo/biology/founder.hpp"
|
|
#include "evo/processes/development_opening.hpp"
|
|
#include "evo/server/cellular_session_batch.hpp"
|
|
#include "evo/server/lifecycle/cellular_schedule.hpp"
|
|
#include "evo/server/lifecycle/primitive_lives.hpp"
|
|
#include "evo/server/lifecycle/wildlife_controller.hpp"
|
|
#include "evo/server/ops_http.hpp"
|
|
#include "evo/server/primitive_region_host.hpp"
|
|
#include "evo/server/region_host.hpp"
|
|
#include "m10_bridge.hpp"
|
|
#include "session_bridge.hpp"
|
|
#ifdef EVO_SERVER_M10
|
|
#include "cellular_bridge.hpp"
|
|
#include "evo/server/pg_slot.hpp"
|
|
#include "evo/server/postgres_control_ledger.hpp"
|
|
#include "evo/server/region_sessions.hpp"
|
|
#endif
|
|
|
|
#include <algorithm>
|
|
#include <array>
|
|
#include <cerrno>
|
|
#include <charconv>
|
|
#include <chrono>
|
|
#include <csignal>
|
|
#include <cstdio>
|
|
#include <filesystem>
|
|
#include <fstream>
|
|
#include <iterator>
|
|
#include <optional>
|
|
#include <poll.h>
|
|
#include <signal.h>
|
|
#include <string>
|
|
#include <string_view>
|
|
#include <thread>
|
|
#include <unistd.h>
|
|
#include <utility>
|
|
#include <vector>
|
|
|
|
namespace {
|
|
using namespace evo;
|
|
using namespace std::chrono_literals;
|
|
volatile std::sig_atomic_t stop_requested = 0;
|
|
extern "C" void on_stop(int) {
|
|
stop_requested = 1;
|
|
}
|
|
|
|
struct Options {
|
|
std::string profile{"lineage"};
|
|
persistence::PgOptions pg;
|
|
std::filesystem::path objects, ready_file;
|
|
server::HostConfig host;
|
|
std::chrono::milliseconds drain{10000};
|
|
std::string crash_point;
|
|
std::uint64_t crash_ordinal{};
|
|
std::string ops_listen;
|
|
bool synthetic{};
|
|
std::uint64_t synthetic_until{}; // 0: unbounded.
|
|
std::uint64_t accelerate{1};
|
|
std::uint64_t objects_reserve_mib{1024};
|
|
server_app::SessionOptions session;
|
|
server_app::M10Options m10;
|
|
bool cellular_populations{};
|
|
struct PrimitiveSessionOptions {
|
|
std::string listen, certificate, key, issuer, environment;
|
|
[[nodiscard]] bool any() const noexcept {
|
|
return !listen.empty() || !certificate.empty() || !key.empty() || !issuer.empty() ||
|
|
!environment.empty();
|
|
}
|
|
[[nodiscard]] bool complete() const noexcept {
|
|
return !listen.empty() && !certificate.empty() && !key.empty() && !issuer.empty() &&
|
|
!environment.empty();
|
|
}
|
|
} primitive_session;
|
|
};
|
|
|
|
// Kills this process without any cleanup: the real SIGKILL, at a chosen boundary.
|
|
[[noreturn]] void crash_now() {
|
|
std::fflush(stdout);
|
|
::kill(::getpid(), SIGKILL);
|
|
for (;;)
|
|
::pause();
|
|
}
|
|
|
|
std::optional<std::uint64_t> number(std::string_view text) {
|
|
std::uint64_t value{};
|
|
const auto [end, error] = std::from_chars(text.data(), text.data() + text.size(), value);
|
|
if (error != std::errc{} || end != text.data() + text.size())
|
|
return std::nullopt;
|
|
return value;
|
|
}
|
|
template <std::size_t N> std::optional<std::array<std::uint8_t, N>> hex(std::string_view text) {
|
|
if (text.size() != N * 2)
|
|
return std::nullopt;
|
|
std::array<std::uint8_t, N> out{};
|
|
for (std::size_t i = 0; i < N; ++i) {
|
|
unsigned value{};
|
|
const auto [end, error] =
|
|
std::from_chars(text.data() + (2 * i), text.data() + (2 * i) + 2, value, 16);
|
|
if (error != std::errc{} || end != text.data() + (2 * i) + 2)
|
|
return std::nullopt;
|
|
out[i] = static_cast<std::uint8_t>(value);
|
|
}
|
|
return out;
|
|
}
|
|
template <std::size_t N> std::string text(const std::array<std::uint8_t, N>& bytes) {
|
|
constexpr char digits[] = "0123456789abcdef";
|
|
std::string out;
|
|
for (const auto b : bytes) {
|
|
out.push_back(digits[b >> 4U]);
|
|
out.push_back(digits[b & 0x0FU]);
|
|
}
|
|
return out;
|
|
}
|
|
|
|
std::optional<Options> parse(int argc, char** argv) {
|
|
Options o;
|
|
o.host.retained_builds.clear();
|
|
for (int i = 1; i < argc; ++i) {
|
|
const std::string_view flag = argv[i];
|
|
if (i + 1 >= argc)
|
|
return std::nullopt;
|
|
const std::string_view value = argv[++i];
|
|
const auto n = number(value);
|
|
if (flag == "--profile" && (value == "lineage" || value == "environmental" ||
|
|
value == "primitive" || value == "cellular"))
|
|
o.profile = value;
|
|
else if (flag == "--world" && n)
|
|
o.host.key.world = *n;
|
|
else if (flag == "--region" && n)
|
|
o.host.key.region = *n;
|
|
else if (flag == "--frame" && n)
|
|
o.host.frame = *n;
|
|
else if (flag == "--pg-host")
|
|
o.pg.host = value;
|
|
else if (flag == "--pg-hostaddr")
|
|
o.pg.hostaddr = value;
|
|
else if (flag == "--pg-port")
|
|
o.pg.port = value;
|
|
else if (flag == "--pg-database")
|
|
o.pg.database = value;
|
|
else if (flag == "--pg-user")
|
|
o.pg.user = value;
|
|
else if (flag == "--pg-passfile")
|
|
o.pg.password_file = value;
|
|
else if (flag == "--pg-service") {
|
|
// Connection from the [evolution] section of a libpq service file (M8.3);
|
|
// the password file stays separate (--pg-passfile).
|
|
auto service = persistence::read_pg_service(std::string(value), "evolution");
|
|
if (!service)
|
|
return std::nullopt;
|
|
service->password_file = o.pg.password_file;
|
|
o.pg = *service;
|
|
} else if (flag == "--pg-ca")
|
|
o.pg.ca_file = value;
|
|
else if (flag == "--objects")
|
|
o.objects = value;
|
|
else if (flag == "--ready-file")
|
|
o.ready_file = value;
|
|
else if (flag == "--build" || flag == "--retained") {
|
|
const auto digest = hex<32>(value);
|
|
if (!digest)
|
|
return std::nullopt;
|
|
if (flag == "--build")
|
|
o.host.build = *digest;
|
|
else
|
|
o.host.retained_builds.push_back(*digest);
|
|
} else if (flag == "--flush-ms" && n && *n > 0 && *n <= 60000)
|
|
o.host.flush_interval = std::chrono::milliseconds{*n};
|
|
else if (flag == "--drain-ms" && n && *n > 0 && *n <= 600000)
|
|
o.drain = std::chrono::milliseconds{*n};
|
|
else if (flag == "--lease-profile" && value == "fast") // Development/test only.
|
|
o.host.lease = {4000ms, 800ms, 400ms, 400ms, 1000ms};
|
|
else if (flag == "--lease-profile" && value == "default")
|
|
o.host.lease = {};
|
|
else if (flag == "--ops-listen")
|
|
o.ops_listen = value;
|
|
else if (flag == "--synthetic-inputs" && value == "development-v1")
|
|
o.synthetic = true;
|
|
else if (flag == "--synthetic-until" && n && *n > 0)
|
|
o.synthetic_until = *n;
|
|
else if (flag == "--accelerate" && n && *n >= 2 && *n <= 1000)
|
|
o.accelerate = *n;
|
|
else if (flag == "--objects-reserve-mib" && n && *n <= 1048576)
|
|
o.objects_reserve_mib = *n;
|
|
else if (flag == "--session-listen")
|
|
o.session.listen = value;
|
|
else if (flag == "--session-cert")
|
|
o.session.certificate = value;
|
|
else if (flag == "--session-key")
|
|
o.session.key = value;
|
|
else if (flag == "--session-grant-key")
|
|
o.session.grant_key = value;
|
|
else if (flag == "--session-grant-ledger")
|
|
o.session.grant_ledger = value;
|
|
else if (flag == "--session-environment" &&
|
|
(value == "dev" || value == "stage" || value == "world0"))
|
|
o.session.environment = value;
|
|
else if (flag == "--m10-listen")
|
|
o.m10.listen = value;
|
|
else if (flag == "--m10-cert")
|
|
o.m10.certificate = value;
|
|
else if (flag == "--m10-key")
|
|
o.m10.key = value;
|
|
else if (flag == "--m10-issuer")
|
|
o.m10.issuer = value;
|
|
else if (flag == "--m10-environment" &&
|
|
(value == "dev" || value == "stage" || value == "world0"))
|
|
o.m10.environment = value;
|
|
else if (flag == "--session-window")
|
|
o.m10.session_window = value;
|
|
else if (flag == "--m10-max-grant-lifetime" && n && *n >= 60 && *n <= 14400)
|
|
o.m10.max_grant_lifetime = *n;
|
|
else if (flag == "--m10-area")
|
|
o.m10.area = value;
|
|
else if (flag == "--m10-handoff-key")
|
|
o.m10.handoff_key = value;
|
|
else if (flag == "--m10-neighbour" && o.m10.neighbours.size() < 8)
|
|
o.m10.neighbours.emplace_back(value);
|
|
else if (flag == "--life-route" && o.m10.life_routes.size() < 4096)
|
|
o.m10.life_routes.emplace_back(value);
|
|
else if (flag == "--m10-load-governor" && (value == "on" || value == "off"))
|
|
o.m10.load_governor = value == "on";
|
|
else if (flag == "--cellular-populations" && (value == "on" || value == "off"))
|
|
o.cellular_populations = value == "on";
|
|
else if (flag == "--primitive-listen")
|
|
o.primitive_session.listen = value;
|
|
else if (flag == "--primitive-cert")
|
|
o.primitive_session.certificate = value;
|
|
else if (flag == "--primitive-key")
|
|
o.primitive_session.key = value;
|
|
else if (flag == "--primitive-issuer")
|
|
o.primitive_session.issuer = value;
|
|
else if (flag == "--primitive-environment" &&
|
|
(value == "dev" || value == "stage" || value == "world0"))
|
|
o.primitive_session.environment = value;
|
|
else if (flag == "--crash-at") {
|
|
const auto colon = value.find(':');
|
|
const auto ordinal =
|
|
colon == std::string_view::npos ? std::nullopt : number(value.substr(colon + 1));
|
|
const auto point = value.substr(0, colon);
|
|
if (!ordinal || *ordinal == 0 ||
|
|
(point != "before-upload" && point != "before-transaction" &&
|
|
point != "after-commit" && point != "after-ack"))
|
|
return std::nullopt;
|
|
o.crash_point = point;
|
|
o.crash_ordinal = *ordinal;
|
|
} else
|
|
return std::nullopt;
|
|
}
|
|
// Every identity and connection setting is required: there is no default
|
|
// region, database, object root or build to fall back on.
|
|
if (o.host.key.world == 0 || o.host.key.region == 0 || o.host.frame == 0 || o.pg.host.empty() ||
|
|
o.pg.port.empty() || o.pg.database.empty() || o.pg.user.empty() || o.objects.empty() ||
|
|
o.host.build == persistence::Sha256Digest{})
|
|
return std::nullopt;
|
|
if (o.cellular_populations && o.profile != "cellular")
|
|
return std::nullopt;
|
|
// Rehearsal controls exist only for the synthetic driver (clones, drills).
|
|
if ((o.synthetic_until != 0 || o.accelerate != 1) && !o.synthetic)
|
|
return std::nullopt;
|
|
// Sessions need every setting, the network build, and the synthetic carrier.
|
|
if (o.session.any() &&
|
|
(!server_app::sessions_available || !o.session.complete() || !o.synthetic))
|
|
return std::nullopt;
|
|
// M10 sessions (ADR-0091/0092) need every setting and the QUIC build.
|
|
if (o.m10.session_any() && (!server_app::m10_available || !o.m10.complete()))
|
|
return std::nullopt;
|
|
if (!o.m10.life_routes.empty() && o.profile != "primitive" && o.profile != "cellular")
|
|
return std::nullopt;
|
|
if (!o.m10.life_routes.empty() && o.profile == "primitive" && !o.primitive_session.any())
|
|
return std::nullopt;
|
|
// Primitive profile serves only the creation-only host. Its optional listener
|
|
// uses a separate v5 contract and never enables the M9/M10 action path.
|
|
if (o.primitive_session.any() &&
|
|
(!server_app::m10_available || !o.primitive_session.complete() || o.profile != "primitive"))
|
|
return std::nullopt;
|
|
if (o.profile == "primitive" &&
|
|
(o.synthetic || !o.crash_point.empty() || o.session.any() || o.m10.session_any() ||
|
|
o.host.flush_interval != 1000ms || o.drain != 10000ms))
|
|
return std::nullopt;
|
|
if (o.profile == "cellular" &&
|
|
(!server_app::m10_available || !o.m10.complete() || o.primitive_session.any() ||
|
|
o.synthetic || !o.crash_point.empty() || o.session.any() || !o.m10.area.empty() ||
|
|
!o.m10.handoff_key.empty() || !o.m10.neighbours.empty() || !o.m10.load_governor ||
|
|
o.host.flush_interval != 1000ms || o.drain != 10000ms))
|
|
return std::nullopt;
|
|
if (std::ranges::find(o.host.retained_builds, o.host.build) == o.host.retained_builds.end())
|
|
o.host.retained_builds.push_back(o.host.build);
|
|
return o;
|
|
}
|
|
|
|
std::optional<server::TickInput> parse_input(std::string_view line) {
|
|
std::array<std::string_view, 11> fields{};
|
|
std::size_t count = 0;
|
|
while (!line.empty() && count < fields.size() + 1) {
|
|
const auto start = line.find_first_not_of(' ');
|
|
if (start == std::string_view::npos)
|
|
break;
|
|
line.remove_prefix(start);
|
|
const auto end = line.find(' ');
|
|
if (count == fields.size())
|
|
return std::nullopt; // Too many fields.
|
|
fields[count++] = line.substr(0, end);
|
|
line.remove_prefix(end == std::string_view::npos ? line.size() : end);
|
|
}
|
|
if (count != fields.size() || fields[0] != "input")
|
|
return std::nullopt;
|
|
auto request = hex<16>(fields[1]);
|
|
if (!request)
|
|
return std::nullopt;
|
|
std::array<std::uint64_t, 9> v{};
|
|
for (std::size_t i = 0; i < v.size(); ++i) {
|
|
const auto n = number(fields[i + 2]);
|
|
if (!n)
|
|
return std::nullopt;
|
|
v[i] = *n;
|
|
}
|
|
server::TickInput in{*request, {}};
|
|
in.input.temperature_mK = v[0];
|
|
in.input.source_enabled = v[1];
|
|
for (std::size_t k = 0; k < 5; ++k)
|
|
in.input.bite[k] = v[2 + k];
|
|
in.input.terminal = {v[7], v[8]};
|
|
return in;
|
|
}
|
|
|
|
const char* admission_name(server::AdmissionError e) {
|
|
switch (e) {
|
|
case server::AdmissionError::not_ready:
|
|
return "not_ready";
|
|
case server::AdmissionError::draining:
|
|
return "draining";
|
|
case server::AdmissionError::full:
|
|
return "full";
|
|
case server::AdmissionError::duplicate:
|
|
return "duplicate";
|
|
case server::AdmissionError::invalid:
|
|
return "invalid";
|
|
}
|
|
return "unknown";
|
|
}
|
|
// Reads complete, bounded lines from stdin without blocking.
|
|
class LineReader {
|
|
public:
|
|
// False on EOF/error.
|
|
bool fill() {
|
|
std::array<char, 4096> chunk{};
|
|
const auto n = ::read(STDIN_FILENO, chunk.data(), chunk.size());
|
|
if (n <= 0)
|
|
return n < 0 && errno == EINTR;
|
|
buffer_.append(chunk.data(), static_cast<std::size_t>(n));
|
|
return true;
|
|
}
|
|
std::optional<std::string> next() {
|
|
const auto end = buffer_.find('\n');
|
|
if (end == std::string::npos) {
|
|
if (buffer_.size() > 512)
|
|
buffer_.clear(); // Overlong line: discarded, never partially parsed.
|
|
return std::nullopt;
|
|
}
|
|
auto line = buffer_.substr(0, end);
|
|
buffer_.erase(0, end + 1);
|
|
return line;
|
|
}
|
|
|
|
private:
|
|
std::string buffer_;
|
|
};
|
|
|
|
template <typename State> int serve(Options& o) {
|
|
if (!o.crash_point.empty() && o.crash_point != "after-ack") {
|
|
o.host.crash_hook = [point = o.crash_point, ordinal = o.crash_ordinal,
|
|
seen = std::uint64_t{0}](server::CrashPoint at) mutable {
|
|
const char* name = at == server::CrashPoint::before_upload ? "before-upload"
|
|
: at == server::CrashPoint::before_transaction ? "before-transaction"
|
|
: "after-commit";
|
|
if (point == name && ++seen == ordinal)
|
|
crash_now();
|
|
};
|
|
}
|
|
std::uint64_t ack_steps = 0;
|
|
auto objects = persistence::FilesystemObjectStore::open(
|
|
o.objects, o.host.key, persistence::max_lineage_payload_bytes, nullptr,
|
|
o.objects_reserve_mib * 1024 * 1024);
|
|
if (!objects) {
|
|
std::puts("error objects");
|
|
return 2;
|
|
}
|
|
const auto pg = o.pg;
|
|
auto made = server::RegionHost<State>::create(
|
|
o.host, [pg] { return persistence::Postgres::connect(pg, 2s); }, std::move(*objects));
|
|
if (!made) {
|
|
std::puts("error config");
|
|
return 2;
|
|
}
|
|
auto& host = **made;
|
|
std::unique_ptr<server::OpsServer> ops;
|
|
if (!o.ops_listen.empty()) {
|
|
auto listening = server::OpsServer::listen(o.ops_listen);
|
|
if (!listening) {
|
|
std::puts("error ops-listen");
|
|
return 2;
|
|
}
|
|
ops = std::move(*listening);
|
|
std::printf("ops listening %u\n", static_cast<unsigned>(ops->port()));
|
|
std::fflush(stdout);
|
|
}
|
|
auto sessions = server_app::SessionBridge::open(o.session, o.host.key, o.host.frame);
|
|
if (!sessions)
|
|
return 2;
|
|
auto m10 = server_app::M10Bridge<server::RegionHost<State>>::open(o.m10, host, o.host.key,
|
|
o.host.frame, o.pg);
|
|
if (!m10)
|
|
return 2;
|
|
std::fflush(stdout);
|
|
const std::uint64_t lookahead = sessions->enabled() ? server_app::session_lookahead_ticks : 64;
|
|
server::OpsCounters counters;
|
|
counters.queue_capacity_inputs = o.host.max_queued_inputs;
|
|
counters.queue_capacity_bytes = o.host.max_queued_bytes;
|
|
counters.objects_reserve_bytes = o.objects_reserve_mib * 1024 * 1024;
|
|
std::uint64_t last_sequence = 0;
|
|
auto last_checkpoint = std::chrono::steady_clock::now();
|
|
bool recovered = false;
|
|
const auto handler = [&](std::string_view path) -> server::OpsResponse {
|
|
const auto& s = host.status();
|
|
if (path == "/livez")
|
|
return {200, "live\n"};
|
|
if (path == "/readyz")
|
|
return s.ready ? server::OpsResponse{200, "ready\n"}
|
|
: server::OpsResponse{503, std::string("not ready: ") +
|
|
server::phase_name(s.phase) + "\n"};
|
|
if (path == "/metrics") {
|
|
counters.checkpoint_age_ms =
|
|
static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::milliseconds>(
|
|
std::chrono::steady_clock::now() - last_checkpoint)
|
|
.count());
|
|
counters.objects_available_bytes = host.objects().available_bytes().value_or(0);
|
|
m10->metrics(counters);
|
|
return {200, server::render_metrics(s, counters),
|
|
"text/plain; version=0.0.4; charset=utf-8"};
|
|
}
|
|
return {404, "not found\n"};
|
|
};
|
|
// Scheduling time; --accelerate scales it from process start (clones only).
|
|
const auto started = std::chrono::steady_clock::now();
|
|
const auto scheduling_now = [&] {
|
|
const auto real = std::chrono::steady_clock::now();
|
|
return started + (real - started) * static_cast<std::int64_t>(o.accelerate);
|
|
};
|
|
LineReader reader;
|
|
bool input_open = !o.synthetic, ready = false;
|
|
// Synthetic driver state: the next tick to submit, reset at every recovery.
|
|
std::optional<std::uint64_t> synthetic_next;
|
|
bool synthetic_running = false;
|
|
const auto synthetic_request = [&](std::uint64_t tick) {
|
|
persistence::Uuid id{'S', 'Y', 'N', '1'};
|
|
for (unsigned i = 0; i < 4; ++i)
|
|
id[4 + i] = static_cast<std::uint8_t>((o.host.key.region >> (8U * (3U - i))) & 0xFFU);
|
|
for (unsigned i = 0; i < 8; ++i)
|
|
id[8 + i] = static_cast<std::uint8_t>((tick >> (8U * (7U - i))) & 0xFFU);
|
|
return id;
|
|
};
|
|
std::optional<std::string> pending;
|
|
std::optional<server::HostPhase> phase;
|
|
std::size_t max_queued = 0;
|
|
std::optional<std::chrono::steady_clock::time_point> drain_deadline;
|
|
const auto set_ready = [&](bool now) {
|
|
if (now == ready)
|
|
return;
|
|
ready = now;
|
|
std::printf("ready %d\n", ready ? 1 : 0);
|
|
if (!o.ready_file.empty()) {
|
|
std::error_code ignored;
|
|
if (ready)
|
|
std::ofstream(o.ready_file) << "ready\n";
|
|
else
|
|
std::filesystem::remove(o.ready_file, ignored);
|
|
}
|
|
};
|
|
for (;;) {
|
|
if (stop_requested && !drain_deadline) {
|
|
m10->shutdown();
|
|
host.drain();
|
|
drain_deadline = std::chrono::steady_clock::now() + o.drain;
|
|
}
|
|
// Read only while the bounded backlog has room: stdin itself backpressures.
|
|
const bool room = host.status().queued < o.host.max_queued_inputs && !pending;
|
|
pollfd fd{STDIN_FILENO, POLLIN, 0};
|
|
if (input_open && room && !drain_deadline && ::poll(&fd, 1, 5) > 0)
|
|
input_open = reader.fill();
|
|
else if (!input_open || !room || drain_deadline)
|
|
::usleep(2000);
|
|
while (!drain_deadline) {
|
|
if (!pending)
|
|
pending = reader.next();
|
|
if (!pending)
|
|
break;
|
|
auto in = parse_input(*pending);
|
|
if (!in) {
|
|
std::puts("rejected - malformed");
|
|
pending.reset();
|
|
continue;
|
|
}
|
|
auto admitted = host.submit(*in);
|
|
if (!admitted && admitted.error() == server::AdmissionError::full)
|
|
break; // Keep the line and retry after ticks drain the backlog.
|
|
if (!admitted)
|
|
std::printf("rejected %s %s\n", text(in->request).c_str(),
|
|
admission_name(admitted.error()));
|
|
pending.reset();
|
|
}
|
|
const bool m10_accepting = m10->accepting(
|
|
static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(
|
|
std::chrono::system_clock::now().time_since_epoch())
|
|
.count()));
|
|
if (o.synthetic && !drain_deadline && (!m10->enabled() || m10_accepting)) {
|
|
const auto& now_status = host.status();
|
|
const bool running = now_status.phase == server::HostPhase::running && now_status.ready;
|
|
if (running && !synthetic_running)
|
|
synthetic_next = now_status.simulated_tick + 1; // Resume after the head.
|
|
synthetic_running = running;
|
|
while (running && synthetic_next &&
|
|
*synthetic_next <= host.status().simulated_tick + lookahead &&
|
|
(o.synthetic_until == 0 || *synthetic_next <= o.synthetic_until)) {
|
|
// An admitted session intention may ride this input; content is unchanged.
|
|
const auto carried = sessions->carry(*synthetic_next);
|
|
const server::TickInput in{carried.value_or(synthetic_request(*synthetic_next)),
|
|
processes::development_input(*synthetic_next)};
|
|
if (!host.submit(in)) {
|
|
if (carried)
|
|
sessions->not_committed(*carried);
|
|
break; // Full or no longer ready: continue next iteration.
|
|
}
|
|
++*synthetic_next;
|
|
}
|
|
}
|
|
const auto step_started = std::chrono::steady_clock::now();
|
|
const auto& s = host.step(scheduling_now());
|
|
const auto host_work = std::chrono::steady_clock::now() - step_started; // ADR-0097.
|
|
max_queued = std::max(max_queued, s.queued);
|
|
// Checkpoint age restarts at recovery and at every confirmed commit.
|
|
if ((!recovered && s.phase == server::HostPhase::running) ||
|
|
s.committed_sequence != last_sequence) {
|
|
recovered = recovered || s.phase == server::HostPhase::running;
|
|
last_sequence = s.committed_sequence;
|
|
last_checkpoint = std::chrono::steady_clock::now();
|
|
}
|
|
if (s.phase == server::HostPhase::acquiring)
|
|
recovered = false;
|
|
if (!phase || *phase != s.phase) {
|
|
phase = s.phase;
|
|
std::printf("phase %s epoch=%llu committed=%llu simulated=%llu\n",
|
|
server::phase_name(s.phase), static_cast<unsigned long long>(s.epoch),
|
|
static_cast<unsigned long long>(s.committed_tick),
|
|
static_cast<unsigned long long>(s.simulated_tick));
|
|
}
|
|
set_ready(s.ready);
|
|
const auto acknowledged = host.take_acks();
|
|
counters.acks += acknowledged.size();
|
|
for (const auto& a : acknowledged) {
|
|
std::printf("ack %s %llu %llu\n", text(a.request).c_str(),
|
|
static_cast<unsigned long long>(a.tick),
|
|
static_cast<unsigned long long>(a.commit_sequence));
|
|
sessions->committed(a.request, a.tick, a.commit_sequence);
|
|
}
|
|
const auto outcomes = host.take_outcomes();
|
|
counters.outcomes += outcomes.size();
|
|
for (const auto& x : outcomes)
|
|
std::printf("outcome %llu %u %llu %llu\n",
|
|
static_cast<unsigned long long>(x.decision.tick),
|
|
static_cast<unsigned>(x.decision.actor),
|
|
static_cast<unsigned long long>(x.decision.organism),
|
|
static_cast<unsigned long long>(x.commit_sequence));
|
|
const auto dropped = host.take_dropped();
|
|
for (const auto& d : dropped)
|
|
++counters.dropped[static_cast<std::size_t>(d.reason)];
|
|
for (const auto& d : dropped) {
|
|
std::printf("dropped %s %s\n", text(d.input.request).c_str(),
|
|
server::unacknowledged_name(d.reason));
|
|
sessions->not_committed(d.input.request);
|
|
}
|
|
std::fflush(stdout);
|
|
if (ops)
|
|
ops->serve(handler);
|
|
sessions->step(host.state(), host.status());
|
|
m10->after_step(host_work);
|
|
std::fflush(stdout);
|
|
if (o.crash_point == "after-ack" && !acknowledged.empty() && ++ack_steps == o.crash_ordinal)
|
|
crash_now();
|
|
const auto finish = [&](int code) {
|
|
set_ready(false);
|
|
std::printf(
|
|
"stats max_queued=%zu max_tick_debt=%llu commits=%llu store_failures=%llu\n",
|
|
max_queued, static_cast<unsigned long long>(s.max_tick_debt),
|
|
static_cast<unsigned long long>(s.commits),
|
|
static_cast<unsigned long long>(s.store_failures));
|
|
std::printf("exit %d\n", code);
|
|
std::fflush(stdout);
|
|
return code;
|
|
};
|
|
if (s.phase == server::HostPhase::stopped)
|
|
return finish(0);
|
|
if (s.phase == server::HostPhase::quarantined)
|
|
return finish(4);
|
|
if (drain_deadline && std::chrono::steady_clock::now() > *drain_deadline)
|
|
return finish(3); // Pending/unknown work is left for the next owner to reconcile.
|
|
}
|
|
}
|
|
|
|
#ifdef EVO_SERVER_M10
|
|
namespace {
|
|
using PrimitiveWireOutcome = network::PrimitiveLifeOutcome;
|
|
|
|
PrimitiveWireOutcome primitive_wire_outcome(std::uint64_t request,
|
|
const server::PrimitiveHostReply& reply) {
|
|
using network::PrimitiveLifeResult;
|
|
PrimitiveLifeResult result = PrimitiveLifeResult::unknown;
|
|
std::uint64_t identity = 0, sequence = 0;
|
|
if (reply.kind == server::PrimitiveHostReplyKind::born && reply.published) {
|
|
result = PrimitiveLifeResult::born;
|
|
identity = reply.published->identity;
|
|
sequence = reply.published->commit_sequence;
|
|
} else if (reply.kind == server::PrimitiveHostReplyKind::pending) {
|
|
result = PrimitiveLifeResult::pending;
|
|
} else if (reply.kind == server::PrimitiveHostReplyKind::refused) {
|
|
switch (reply.reservation.refusal) {
|
|
case persistence::LifeRefusal::insufficient_funding:
|
|
result = PrimitiveLifeResult::insufficient_funding;
|
|
break;
|
|
case persistence::LifeRefusal::no_placement:
|
|
result = PrimitiveLifeResult::no_placement;
|
|
break;
|
|
case persistence::LifeRefusal::cancelled:
|
|
result = PrimitiveLifeResult::cancelled;
|
|
break;
|
|
case persistence::LifeRefusal::existing_life:
|
|
result = PrimitiveLifeResult::existing_life;
|
|
break;
|
|
case persistence::LifeRefusal::none:
|
|
result = PrimitiveLifeResult::unknown;
|
|
break;
|
|
}
|
|
}
|
|
// A reservation owned by another region is never asserted as a birth here. The current
|
|
// session can query again after it reaches the owner; region routing stays explicit.
|
|
return {request, result, identity, sequence};
|
|
}
|
|
|
|
struct PrimitiveSessionBridge {
|
|
std::unique_ptr<server::PgSlot> slot;
|
|
std::unique_ptr<server::PostgresControlLedger> ledger;
|
|
std::unique_ptr<server::RegionSessions> sessions;
|
|
};
|
|
|
|
std::optional<PrimitiveSessionBridge>
|
|
open_primitive_sessions(const Options::PrimitiveSessionOptions& options,
|
|
const server_app::M10Options& routing, server::PrimitiveRegionHost& host,
|
|
persistence::RegionKey key, std::uint64_t frame,
|
|
const persistence::PgOptions& pg) {
|
|
PrimitiveSessionBridge bridge;
|
|
if (!options.any())
|
|
return bridge;
|
|
const auto colon = options.listen.rfind(':');
|
|
std::uint16_t port{};
|
|
if (colon == std::string::npos ||
|
|
std::from_chars(options.listen.data() + colon + 1,
|
|
options.listen.data() + options.listen.size(), port)
|
|
.ec != std::errc{}) {
|
|
std::puts("error primitive-listen");
|
|
return std::nullopt;
|
|
}
|
|
std::ifstream issuer_file(options.issuer, std::ios::binary);
|
|
std::vector<char> issuer((std::istreambuf_iterator<char>(issuer_file)), {});
|
|
if (issuer.size() != network::grant_public_key_bytes) {
|
|
std::puts("error primitive-issuer");
|
|
return std::nullopt;
|
|
}
|
|
server::RegionSessionsConfig config;
|
|
config.transport.listen_address = options.listen.substr(0, colon);
|
|
config.transport.port = port;
|
|
config.transport.certificate_pem_file = options.certificate;
|
|
config.transport.key_pem_file = options.key;
|
|
config.control.environment = options.environment == "dev" ? network::DeploymentEnvironment::dev
|
|
: options.environment == "stage"
|
|
? network::DeploymentEnvironment::stage
|
|
: network::DeploymentEnvironment::world0;
|
|
config.control.world = {key.world, frame};
|
|
config.control.region = key.region;
|
|
config.control.epoch = 1;
|
|
config.control.max_sessions = 512;
|
|
config.transport.limits.max_connections = 576;
|
|
const auto life_routes = server_app::parse_life_owner_routes(routing);
|
|
if (!life_routes) {
|
|
std::puts("error primitive-life-routes");
|
|
return std::nullopt;
|
|
}
|
|
config.life_owner_routes = *life_routes;
|
|
network::GrantPublicKey issuer_key{};
|
|
std::copy(issuer.begin(), issuer.end(), issuer_key.begin());
|
|
config.issuer_keys.push_back({1, issuer_key});
|
|
bridge.slot = std::make_unique<server::PgSlot>(pg, std::chrono::seconds{2});
|
|
bridge.ledger = std::make_unique<server::PostgresControlLedger>(
|
|
*bridge.slot, key, [&host] { return host.authority(); });
|
|
config.ledger = bridge.ledger.get();
|
|
config.create_primitive_life = [&host](const network::AccountId& account,
|
|
std::uint64_t generation, std::uint64_t request) {
|
|
const auto result =
|
|
host.new_life(account.bytes, generation, request, biology::founder_life_rules);
|
|
return primitive_wire_outcome(request, result);
|
|
};
|
|
config.query_primitive_life = [&host](const network::AccountId& account,
|
|
std::uint64_t generation, std::uint64_t request) {
|
|
return primitive_wire_outcome(request, host.query_life(account.bytes, generation, request));
|
|
};
|
|
auto sessions = server::RegionSessions::create(std::move(config));
|
|
if (!sessions) {
|
|
std::puts(sessions.error() == server::RegionSessionsError::transport
|
|
? "error primitive-listen"
|
|
: "error primitive-session");
|
|
return std::nullopt;
|
|
}
|
|
bridge.sessions = std::move(*sessions);
|
|
std::printf("primitive listening %u\n", static_cast<unsigned>(bridge.sessions->port()));
|
|
return bridge;
|
|
}
|
|
} // namespace
|
|
#endif
|
|
|
|
int serve_primitive(Options& o) {
|
|
auto objects = persistence::FilesystemObjectStore::open(
|
|
o.objects, o.host.key, server::max_primitive_lives_bytes, nullptr,
|
|
o.objects_reserve_mib * 1024 * 1024);
|
|
if (!objects) {
|
|
std::puts("error objects");
|
|
return 2;
|
|
}
|
|
// The unique_ptr moves into the host, but the store object itself stays put.
|
|
const auto* objects_view = objects->get();
|
|
const auto pg = o.pg;
|
|
auto made = server::PrimitiveRegionHost::create(
|
|
{o.host.key, o.host.frame, o.host.lease, o.host.build, o.host.retained_builds},
|
|
[pg] { return persistence::Postgres::connect(pg, 2s); }, std::move(*objects));
|
|
if (!made) {
|
|
std::puts("error config");
|
|
return 2;
|
|
}
|
|
auto& host = **made;
|
|
#ifdef EVO_SERVER_M10
|
|
auto primitive_sessions =
|
|
open_primitive_sessions(o.primitive_session, o.m10, host, o.host.key, o.host.frame, o.pg);
|
|
if (!primitive_sessions)
|
|
return 2;
|
|
#endif
|
|
std::unique_ptr<server::OpsServer> ops;
|
|
if (!o.ops_listen.empty()) {
|
|
auto listening = server::OpsServer::listen(o.ops_listen);
|
|
if (!listening) {
|
|
std::puts("error ops-listen");
|
|
return 2;
|
|
}
|
|
ops = std::move(*listening);
|
|
std::printf("ops listening %u\n", static_cast<unsigned>(ops->port()));
|
|
}
|
|
server::OpsCounters counters;
|
|
counters.objects_reserve_bytes = o.objects_reserve_mib * 1024 * 1024;
|
|
bool ready = false;
|
|
std::optional<server::HostPhase> phase;
|
|
const auto set_ready = [&](bool now) {
|
|
if (ready != now) {
|
|
ready = now;
|
|
std::printf("ready %d\n", ready ? 1 : 0);
|
|
}
|
|
if (!o.ready_file.empty()) {
|
|
std::error_code ignored;
|
|
if (ready)
|
|
std::ofstream(o.ready_file) << "ready\n";
|
|
else
|
|
std::filesystem::remove(o.ready_file, ignored);
|
|
}
|
|
};
|
|
const auto finish = [&](int code) {
|
|
set_ready(false);
|
|
const auto& status = host.status();
|
|
std::printf("phase %s epoch=%llu committed=%llu simulated=%llu\n",
|
|
server::phase_name(status.phase), static_cast<unsigned long long>(status.epoch),
|
|
static_cast<unsigned long long>(status.committed_tick),
|
|
static_cast<unsigned long long>(status.simulated_tick));
|
|
std::printf("stats commits=%llu store_failures=%llu\n",
|
|
static_cast<unsigned long long>(status.commits),
|
|
static_cast<unsigned long long>(status.store_failures));
|
|
std::printf("exit %d\n", code);
|
|
std::fflush(stdout);
|
|
return code;
|
|
};
|
|
for (;;) {
|
|
if (stop_requested) {
|
|
#ifdef EVO_SERVER_M10
|
|
if (primitive_sessions->sessions)
|
|
primitive_sessions->sessions->shutdown(std::chrono::steady_clock::now());
|
|
#endif
|
|
host.drain();
|
|
return finish(0);
|
|
}
|
|
const auto& status = host.step();
|
|
#ifdef EVO_SERVER_M10
|
|
if (primitive_sessions->sessions)
|
|
primitive_sessions->sessions->poll(
|
|
{status.ready, status.epoch, status.simulated_tick},
|
|
static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(
|
|
std::chrono::system_clock::now().time_since_epoch())
|
|
.count()),
|
|
std::chrono::steady_clock::now());
|
|
#endif
|
|
set_ready(status.ready);
|
|
if (!phase || *phase != status.phase) {
|
|
phase = status.phase;
|
|
std::printf("phase %s epoch=%llu committed=%llu simulated=%llu\n",
|
|
server::phase_name(status.phase),
|
|
static_cast<unsigned long long>(status.epoch),
|
|
static_cast<unsigned long long>(status.committed_tick),
|
|
static_cast<unsigned long long>(status.simulated_tick));
|
|
}
|
|
if (ops) {
|
|
const auto handler = [&](std::string_view path) -> server::OpsResponse {
|
|
const auto& current = host.status();
|
|
if (path == "/livez")
|
|
return {200, "live\n"};
|
|
if (path == "/readyz")
|
|
return current.ready ? server::OpsResponse{200, "ready\n"}
|
|
: server::OpsResponse{
|
|
503, std::string("not ready: ") +
|
|
server::phase_name(current.phase) + "\n"};
|
|
if (path == "/metrics") {
|
|
counters.objects_available_bytes = objects_view->available_bytes().value_or(0);
|
|
return {200, server::render_metrics(current, counters),
|
|
"text/plain; version=0.0.4; charset=utf-8"};
|
|
}
|
|
return {404, "not found\n"};
|
|
};
|
|
ops->serve(handler);
|
|
}
|
|
std::fflush(stdout);
|
|
if (status.phase == server::HostPhase::quarantined)
|
|
return finish(4);
|
|
std::this_thread::sleep_for(5ms);
|
|
}
|
|
}
|
|
|
|
#ifdef EVO_SERVER_M10
|
|
int serve_cellular(Options& o) {
|
|
auto objects = persistence::FilesystemObjectStore::open(
|
|
o.objects, o.host.key, server::max_cellular_population_bytes, nullptr,
|
|
o.objects_reserve_mib * 1024 * 1024);
|
|
if (!objects) {
|
|
std::puts("error objects");
|
|
return 2;
|
|
}
|
|
const auto pg = o.pg;
|
|
auto made = server::CellularRegionHost::create(
|
|
{o.host.key, o.host.frame, o.host.lease, o.host.build, o.host.retained_builds,
|
|
o.cellular_populations ? std::optional{server::CellularTierPolicy{true, 1200, 1800}}
|
|
: std::nullopt},
|
|
[pg] { return persistence::Postgres::connect(pg, 2s); }, std::move(*objects));
|
|
if (!made) {
|
|
std::puts("error config");
|
|
return 2;
|
|
}
|
|
auto& host = **made;
|
|
auto bridge = server_app::CellularBridge::open(o.m10, host, o.host.key, o.host.frame, o.pg);
|
|
if (!bridge)
|
|
return 2;
|
|
std::unique_ptr<server::OpsServer> ops;
|
|
if (!o.ops_listen.empty()) {
|
|
auto listening = server::OpsServer::listen(o.ops_listen);
|
|
if (!listening) {
|
|
std::puts("error ops-listen");
|
|
return 2;
|
|
}
|
|
ops = std::move(*listening);
|
|
std::printf("ops listening %u\n", static_cast<unsigned>(ops->port()));
|
|
}
|
|
bool ready = false;
|
|
std::optional<server::HostPhase> phase;
|
|
server::CellularTickSchedule schedule;
|
|
// The open session batch between commits, the actions it carries, and the monotone pace
|
|
// head per epoch (rules-12 regions commit every flush interval, not every tick).
|
|
std::optional<server::CellularSessionBatchBuilder> batch_builder;
|
|
std::vector<network::CarriedAction> batch_actions;
|
|
std::uint64_t pace_epoch = 0, pace_tick = 0;
|
|
std::vector<network::CarriedAction> awaiting_outcomes;
|
|
// A tick that cannot be built is retried every 50 ms; name each distinct cause once.
|
|
std::optional<std::pair<std::string_view, server::CellularSessionBatchError>> stalled;
|
|
const auto report_stall = [&](std::string_view where, server::CellularSessionBatchError error,
|
|
std::uint64_t tick, int lives_error = -1) {
|
|
if (stalled && stalled->first == where && stalled->second == error)
|
|
return;
|
|
stalled.emplace(where, error);
|
|
std::printf("cellular tick stalled tick=%llu at=%.*s error=%u lives=%d\n",
|
|
static_cast<unsigned long long>(tick), static_cast<int>(where.size()),
|
|
where.data(), static_cast<unsigned>(error), lives_error);
|
|
};
|
|
const auto set_ready = [&](bool now) {
|
|
if (ready != now) {
|
|
ready = now;
|
|
std::printf("ready %d\n", ready ? 1 : 0);
|
|
}
|
|
if (!o.ready_file.empty()) {
|
|
std::error_code ignored;
|
|
if (ready)
|
|
std::ofstream(o.ready_file) << "ready\n";
|
|
else
|
|
std::filesystem::remove(o.ready_file, ignored);
|
|
}
|
|
};
|
|
const auto deliver_finalized = [&] {
|
|
for (const auto& finalized : host.take_finalized_actions()) {
|
|
const auto pending = std::ranges::find_if(awaiting_outcomes, [&](const auto& action) {
|
|
return action.account.bytes == finalized.action.account &&
|
|
action.action_id == finalized.action.action_id;
|
|
});
|
|
if (pending == awaiting_outcomes.end())
|
|
continue;
|
|
const std::array committed{*pending};
|
|
bridge->committed(committed, finalized.commit_sequence);
|
|
awaiting_outcomes.erase(pending);
|
|
}
|
|
for (const auto& mutation : host.take_finalized_mutations()) {
|
|
network::AccountId account;
|
|
account.bytes = mutation.account;
|
|
std::printf("mutation committed tick=%llu result=%u\n",
|
|
static_cast<unsigned long long>(mutation.tick),
|
|
static_cast<unsigned>(mutation.result));
|
|
static_cast<void>(bridge->sessions().send_mutation_outcome(
|
|
account, server::action_outcome(mutation)));
|
|
}
|
|
};
|
|
const auto finish = [&](int code) {
|
|
set_ready(false);
|
|
const auto& status = host.status();
|
|
std::printf("phase %s epoch=%llu committed=%llu simulated=%llu\n",
|
|
server::phase_name(status.phase), static_cast<unsigned long long>(status.epoch),
|
|
static_cast<unsigned long long>(status.committed_tick),
|
|
static_cast<unsigned long long>(status.simulated_tick));
|
|
std::printf("stats commits=%llu store_failures=%llu\n",
|
|
static_cast<unsigned long long>(status.commits),
|
|
static_cast<unsigned long long>(status.store_failures));
|
|
std::printf("exit %d\n", code);
|
|
std::fflush(stdout);
|
|
return code;
|
|
};
|
|
for (;;) {
|
|
if (stop_requested) {
|
|
bridge->shutdown();
|
|
host.drain();
|
|
return finish(0);
|
|
}
|
|
const auto& status = host.step();
|
|
const auto wall =
|
|
static_cast<std::uint64_t>(std::chrono::duration_cast<std::chrono::seconds>(
|
|
std::chrono::system_clock::now().time_since_epoch())
|
|
.count());
|
|
static_cast<void>(bridge->accepting(wall));
|
|
// Sessions schedule and admit against what has been simulated: the open batch's
|
|
// working tick runs up to a flush interval ahead of the committed head.
|
|
auto session_status = status;
|
|
if (batch_builder)
|
|
session_status.simulated_tick =
|
|
std::max(session_status.simulated_tick, batch_builder->candidate().tick());
|
|
bridge->poll(session_status, wall);
|
|
deliver_finalized();
|
|
set_ready(status.ready);
|
|
if (!phase || *phase != status.phase) {
|
|
phase = status.phase;
|
|
std::printf("phase %s epoch=%llu committed=%llu simulated=%llu\n",
|
|
server::phase_name(status.phase),
|
|
static_cast<unsigned long long>(status.epoch),
|
|
static_cast<unsigned long long>(status.committed_tick),
|
|
static_cast<unsigned long long>(status.simulated_tick));
|
|
}
|
|
if (ops) {
|
|
const auto handler = [&](std::string_view path) -> server::OpsResponse {
|
|
const auto& current = host.status();
|
|
if (path == "/livez")
|
|
return {200, "live\n"};
|
|
if (path == "/readyz")
|
|
return current.ready ? server::OpsResponse{200, "ready\n"}
|
|
: server::OpsResponse{
|
|
503, std::string("not ready: ") +
|
|
server::phase_name(current.phase) + "\n"};
|
|
if (path == "/metrics") {
|
|
auto paced = current;
|
|
paced.tick_debt = schedule.status().debt;
|
|
paced.max_tick_debt = schedule.status().max_debt;
|
|
server::OpsCounters counters;
|
|
counters.checkpoint_age_ms = schedule.status().head_age_us / 1000;
|
|
return {200, server::render_metrics(paced, counters)};
|
|
}
|
|
return {404, "not found\n"};
|
|
};
|
|
ops->serve(handler);
|
|
}
|
|
if (status.phase == server::HostPhase::quarantined)
|
|
return finish(4);
|
|
|
|
const auto* state = host.state();
|
|
bool player_present = false;
|
|
if (status.ready && state) {
|
|
for (const auto& life : state->lives()) {
|
|
network::AccountId account;
|
|
account.bytes = life.account;
|
|
player_present = player_present || bridge->sessions().holder(account);
|
|
}
|
|
}
|
|
// A queued birth publishes only while its account holds a session here: the
|
|
// publisher checks current account control, and a departed account's birth would
|
|
// fail every batch with the lease-fence code, so the region would fence itself in a
|
|
// loop. The birth stays queued and resumes when the account returns.
|
|
std::vector<server::CellularTickIntents::Birth> held_births;
|
|
for (const auto& birth : host.queued_births()) {
|
|
network::AccountId account;
|
|
account.bytes = birth.account;
|
|
if (held_births.size() < biology::max_cellular_tick_lives &&
|
|
bridge->sessions().holder(account))
|
|
held_births.push_back(birth);
|
|
}
|
|
const bool birth_waiting = !held_births.empty();
|
|
const bool want_population = o.cellular_populations && state &&
|
|
server::population_interval_safe(*state) && !player_present &&
|
|
!birth_waiting && host.queued_mutations().empty() &&
|
|
std::ranges::any_of(state->lives(), [](const auto& life) {
|
|
return life.account == server::LifeAccount{};
|
|
});
|
|
if (status.ready && state && want_population != host.population_mode()) {
|
|
[[maybe_unused]] const auto changed = host.set_population_mode(want_population);
|
|
state = host.state();
|
|
}
|
|
const bool needed = state && (player_present || birth_waiting ||
|
|
server::cellular_unattended_tick_needed(*state) ||
|
|
status.ecological_pending_ticks != 0);
|
|
const auto monotonic_us = [] {
|
|
return static_cast<std::uint64_t>(
|
|
std::chrono::duration_cast<std::chrono::microseconds>(
|
|
std::chrono::steady_clock::now().time_since_epoch())
|
|
.count());
|
|
};
|
|
// Pacing follows the simulated tick (the open batch's working state), never backwards
|
|
// within an epoch: a discarded batch must not look like a head regression.
|
|
const auto simulated_tick = batch_builder ? batch_builder->candidate().tick()
|
|
: state ? state->tick()
|
|
: status.committed_tick;
|
|
if (status.epoch != pace_epoch) {
|
|
pace_epoch = status.epoch;
|
|
pace_tick = simulated_tick;
|
|
}
|
|
pace_tick = std::max(pace_tick, simulated_tick);
|
|
const auto paced = schedule.observe(monotonic_us(), status.epoch, pace_tick, needed,
|
|
status.ready && state, status.ecological_pending_ticks);
|
|
if (!paced) {
|
|
std::puts("error cellular-schedule");
|
|
return finish(4);
|
|
}
|
|
const auto flush_ticks = std::clamp<std::uint64_t>(
|
|
static_cast<std::uint64_t>(o.host.flush_interval.count()) / 50, 1, 60);
|
|
const auto flush = [&] {
|
|
if (!batch_builder || batch_builder->batch().ticks.empty()) {
|
|
batch_builder.reset();
|
|
return;
|
|
}
|
|
auto published = host.publish(batch_builder->batch(), batch_builder->actions(),
|
|
batch_builder->mutation_actions());
|
|
if (published)
|
|
host.refuse_births(batch_builder->refused_births());
|
|
if (!published && host.status().phase != server::HostPhase::resolving) {
|
|
bridge->dropped(batch_actions, true);
|
|
std::erase_if(awaiting_outcomes, [&](const auto& action) {
|
|
return std::ranges::find(batch_actions, action) != batch_actions.end();
|
|
});
|
|
}
|
|
batch_builder.reset();
|
|
batch_actions.clear();
|
|
deliver_finalized();
|
|
state = host.state(); // Publication replaced the committed state.
|
|
};
|
|
// A batch cannot outlive the authority or the raw mode it was built under; when demand
|
|
// ends, what it holds is committed.
|
|
if (batch_builder && (!status.ready || !state || host.population_mode())) {
|
|
bridge->dropped(batch_actions, true);
|
|
std::erase_if(awaiting_outcomes, [&](const auto& action) {
|
|
return std::ranges::find(batch_actions, action) != batch_actions.end();
|
|
});
|
|
batch_builder.reset();
|
|
batch_actions.clear();
|
|
} else if (batch_builder && !needed) {
|
|
flush();
|
|
}
|
|
if (status.ready && state && (needed || host.population_mode())) {
|
|
if (host.population_mode()) {
|
|
if (paced->retry_ready && (paced->due || status.ecological_pending_ticks != 0)) {
|
|
const auto published = host.advance_unattended(paced->due ? paced->debt : 0);
|
|
if (!published)
|
|
schedule.defer(monotonic_us());
|
|
}
|
|
} else {
|
|
// Rules 12 regions hold thousands of deposits, so their committed payload is
|
|
// large: commit every flush interval (ADR-0089: 1 s), not every tick. Players
|
|
// see the batch's working state at once; outcomes become final at its commit.
|
|
// Births and mutation choices commit promptly, in their own fresh batch.
|
|
if (batch_builder && (!held_births.empty() || !host.queued_mutations().empty()))
|
|
flush();
|
|
const auto* current = batch_builder ? &batch_builder->candidate() : state;
|
|
// A failed commit can leave the host without state until it recovers.
|
|
if (current && host.committed() && paced->due && needed &&
|
|
current->tick() != UINT64_MAX) {
|
|
const auto tick = current->tick() + 1;
|
|
std::vector<server::CellularLifeIntent> baseline;
|
|
baseline.reserve(current->lives().size());
|
|
// ADR-0128: an owned life whose account holds no session here is frozen
|
|
// (rules 5); the trace records the boundary, and its carried actions are
|
|
// refused below.
|
|
const bool presence_rules =
|
|
current->rules_version() >= server::cellular_lives_presence_rules;
|
|
std::vector<server::LifeAccount> offline;
|
|
for (const auto& life : current->lives()) {
|
|
server::CellularLifeIntent intent{life.identity.life, {}};
|
|
network::AccountId account;
|
|
account.bytes = life.account;
|
|
if (presence_rules && life.account != server::LifeAccount{} &&
|
|
!bridge->sessions().holder(account)) {
|
|
intent.intent.frozen = true;
|
|
offline.push_back(life.account);
|
|
}
|
|
baseline.push_back(intent);
|
|
}
|
|
std::ranges::sort(baseline, {}, &server::CellularLifeIntent::life_id);
|
|
// ADR-0125: wildlife follow the deterministic controller; its intentions
|
|
// ride the trace like a player's, and the owner validates them.
|
|
if (const auto wild = server::wildlife_intents(*current, tick); wild) {
|
|
for (const auto& intent : *wild) {
|
|
const auto found = std::ranges::lower_bound(
|
|
baseline, intent.life_id, {}, &server::CellularLifeIntent::life_id);
|
|
if (found != baseline.end() && found->life_id == intent.life_id)
|
|
*found = intent;
|
|
}
|
|
}
|
|
auto actions = bridge->carry(tick);
|
|
if (!offline.empty()) {
|
|
std::vector<network::CarriedAction> departed;
|
|
std::erase_if(actions, [&](const network::CarriedAction& action) {
|
|
if (std::ranges::find(offline, action.account.bytes) == offline.end())
|
|
return false;
|
|
departed.push_back(action);
|
|
return true;
|
|
});
|
|
bridge->dropped(departed, true);
|
|
}
|
|
if (!batch_builder) {
|
|
auto created = server::CellularSessionBatchBuilder::create(
|
|
*state, host.committed()->head);
|
|
if (created)
|
|
batch_builder.emplace(std::move(*created));
|
|
else
|
|
report_stall("create", created.error(), tick);
|
|
}
|
|
if (!batch_builder) {
|
|
bridge->dropped(actions, true);
|
|
} else {
|
|
// Births and mutation choices only ever enter a fresh batch.
|
|
const bool fresh = batch_builder->batch().ticks.empty();
|
|
const std::span<const server::CellularTickIntents::Birth> births =
|
|
fresh ? std::span<const server::CellularTickIntents::Birth>{held_births}
|
|
: std::span<const server::CellularTickIntents::Birth>{};
|
|
const std::span<const persistence::CellularMutationReservation> mutations =
|
|
fresh ? host.queued_mutations()
|
|
: std::span<const persistence::CellularMutationReservation>{};
|
|
auto appended = batch_builder->append_with_birth_refusals(
|
|
tick, baseline, actions, births, mutations);
|
|
if (!appended && !actions.empty()) {
|
|
// ADR-0127: one account's unusable action (its life died, or it
|
|
// names an unperceived patch) refuses only that action, never the
|
|
// tick for every other player.
|
|
std::vector<network::CarriedAction> usable, refused;
|
|
for (const auto& action : actions)
|
|
(server::prepare_cellular_session_tick(*current, tick, baseline,
|
|
std::span{&action, 1})
|
|
? usable
|
|
: refused)
|
|
.push_back(action);
|
|
bridge->dropped(refused, true);
|
|
actions = std::move(usable);
|
|
appended = batch_builder->append_with_birth_refusals(
|
|
tick, baseline, actions, births, mutations);
|
|
}
|
|
if (!appended) {
|
|
const auto lives = batch_builder->last_lives_error();
|
|
report_stall("append", appended.error(), tick,
|
|
lives ? static_cast<int>(*lives) : -1);
|
|
bridge->dropped(actions, true);
|
|
flush(); // Commit what is good; the next tick starts afresh.
|
|
} else {
|
|
stalled.reset();
|
|
awaiting_outcomes.insert(awaiting_outcomes.end(), actions.begin(),
|
|
actions.end());
|
|
batch_actions.insert(batch_actions.end(), actions.begin(),
|
|
actions.end());
|
|
const bool prompt = !births.empty() || !mutations.empty();
|
|
if (prompt || batch_builder->batch().ticks.size() >= flush_ticks)
|
|
flush();
|
|
}
|
|
}
|
|
// Unappended work keeps its pacing debt; retry with bounded frequency.
|
|
const auto* after = batch_builder ? &batch_builder->candidate() : host.state();
|
|
if (!after || after->tick() < tick)
|
|
schedule.defer(monotonic_us());
|
|
}
|
|
}
|
|
}
|
|
if (const auto* committed_state = host.state(); committed_state)
|
|
bridge->replicate(batch_builder ? batch_builder->candidate() : *committed_state,
|
|
committed_state);
|
|
std::fflush(stdout);
|
|
std::this_thread::sleep_for(2ms);
|
|
}
|
|
}
|
|
#endif
|
|
} // namespace
|
|
|
|
int main(int argc, char** argv) {
|
|
auto options = parse(argc, argv);
|
|
if (!options) {
|
|
std::fputs("usage: evo-server --profile lineage|environmental|primitive|"
|
|
"cellular --world N --region N "
|
|
"--frame N --pg-host H --pg-port P --pg-database D --pg-user U "
|
|
"--objects DIR --build HEX64 [--retained HEX64]... [--pg-hostaddr A "
|
|
"--pg-passfile F --pg-ca F] [--flush-ms N] [--drain-ms N] "
|
|
"[--lease-profile default|fast] [--ready-file PATH] "
|
|
"[--primitive-listen ADDRESS:PORT --primitive-cert F --primitive-key F "
|
|
"--primitive-issuer F --primitive-environment dev|stage|world0] "
|
|
"[--ops-listen ADDRESS:PORT] [--synthetic-inputs development-v1 "
|
|
"[--synthetic-until TICK] [--accelerate N] [--session-listen ADDRESS:PORT "
|
|
"--session-cert F --session-key F --session-grant-key F "
|
|
"--session-grant-ledger F --session-environment dev]] [--m10-listen "
|
|
"ADDRESS:PORT --m10-cert F --m10-key F --m10-issuer F "
|
|
"--m10-environment dev|stage|world0 --session-window ACTIVATION:UTC_EXPIRY "
|
|
"[--m10-max-grant-lifetime 60..14400] [--m10-area X0,Y0,X1,Y1] "
|
|
"[--m10-handoff-key F [--m10-neighbour "
|
|
"REGION=X0,Y0,X1,Y1=HOST:PORT=NAME=PUBFILE]...] "
|
|
"[--life-route REGION=HOST:PORT=TLS_NAME ...] "
|
|
"[--m10-load-governor on|off]] [--cellular-populations on|off]\\n",
|
|
stderr);
|
|
return 2;
|
|
}
|
|
struct sigaction action{};
|
|
action.sa_handler = on_stop;
|
|
sigemptyset(&action.sa_mask);
|
|
sigaction(SIGTERM, &action, nullptr);
|
|
sigaction(SIGINT, &action, nullptr);
|
|
std::signal(SIGPIPE, SIG_IGN);
|
|
if (options->profile == "primitive")
|
|
return serve_primitive(*options);
|
|
#ifdef EVO_SERVER_M10
|
|
if (options->profile == "cellular")
|
|
return serve_cellular(*options);
|
|
#endif
|
|
return options->profile == "lineage" ? serve<processes::LineageRegion>(*options)
|
|
: serve<processes::EnvironmentalCommunity>(*options);
|
|
}
|