diff --git a/docs/adr/0113-cellular-replay-trace.md b/docs/adr/0113-cellular-replay-trace.md index 8d21941..1ecbfff 100644 --- a/docs/adr/0113-cellular-replay-trace.md +++ b/docs/adr/0113-cellular-replay-trace.md @@ -45,3 +45,12 @@ deterministic and bounded, while the complete post-tick checkpoint remains the authoritative recovery state. Profile 1–3 trace formats and manifest size rules remain unchanged. Process-level uncertain-commit and restart tests are still required before M11.3d can close. + +ADR-0114 extends the profile-4 trace to format 2. It retains the 112-byte +header and per-tick life intentions, then records a bounded birth set sorted by +account and request. Each birth record is 40 bytes: account (16 bytes), request +ID, control generation, and allocator-issued identity (u64 LE each). The birth +is replayed after that tick's existing-life step. Format-1 traces remain +decodable for already committed checkpoints. The new maximum is +`112 + ticks * (8 + 256 * (24 + 40))` bytes; migration 22 adjusts the profile-4 +manifest bound without changing other profiles. diff --git a/docs/engineering/M11_3_DELIVERY_PLAN.md b/docs/engineering/M11_3_DELIVERY_PLAN.md index 74e91ab..201c2c7 100644 --- a/docs/engineering/M11_3_DELIVERY_PLAN.md +++ b/docs/engineering/M11_3_DELIVERY_PLAN.md @@ -310,6 +310,14 @@ scoped change before starting the next segment. following tick. This remains in-memory: canonical birth trace, durable publication/recovery and session scheduling remain open, so 11.3e is not playable yet. +- 11.3d-7h: profile-4 replay trace format 2 adds canonically ordered bounded + birth records after each tick's existing-life intentions. Replay applies each + birth at the end tick, derives all biology and placement from the predecessor, + and verifies that request identity, generation and allocator identity are + nonzero and unique. The decoder still accepts format-1 traces. C++ manifest + validation and migration 22 raise the profile-4 trace ceiling; PostgreSQL and + end-to-end birth publication tests are still pending, so this is not yet a + durable birth path or a playable 11.3e slice. ## Exit boundary diff --git a/docs/engineering/VALIDATION.md b/docs/engineering/VALIDATION.md index 5800924..b950188 100644 --- a/docs/engineering/VALIDATION.md +++ b/docs/engineering/VALIDATION.md @@ -6398,3 +6398,22 @@ jobs remain stalled without output and do not count as validation. The current Windows environment also lacks `clang-format`. Durable birth trace and server integration remain unimplemented, so this does not make the cellular stage playable. + +## 2026-10-11 — M11.3d-7h cellular birth replay trace (database pending) + +Profile-4 cellular trace format 2 now records a bounded birth set after each +tick's existing-life intentions. Recovery deterministically applies births at +the end tick, and the decoder retains format-1 support for old committed traces. +The C++ manifest bound was expanded; migration 22 updates the PostgreSQL +profile-4 trace ceiling and the migration runner/CI history checks. + +The focused lifecycle test passes under GCC 13.2 in debug and `-DNDEBUG` builds, +including trace encode/decode, version-1 decoding, funded end-of-tick birth +replay, and malformed generation rejection. Strict-warning syntax compilation +of `src/persistence/region_checkpoint.cpp` and `git diff --check` pass. This +Windows environment has no `psql`, so migration execution and PostgreSQL +integration are pending CI. CTest, sanitizer checks, clang-format and CI have +not run for this segment. The Windows `bash.exe` shim is access-denied, and the +migration test translation unit cannot compile here because its process fixture +requires POSIX `poll.h`. The previously noted unrelated initializer warning in +`cellular_region_step.cpp:29` is demoted only for the manual focused test build. diff --git a/include/evo/server/lifecycle/cellular_batch.hpp b/include/evo/server/lifecycle/cellular_batch.hpp index 9d663d3..a29598a 100644 --- a/include/evo/server/lifecycle/cellular_batch.hpp +++ b/include/evo/server/lifecycle/cellular_batch.hpp @@ -21,6 +21,14 @@ struct CellularBatchHeader { struct CellularTickIntents { // One intention for each life alive at the start of this tick, sorted by life_id. std::vector lives; + // Owner-authorized births are sorted by (account, request). Biology and + // placement are derived during replay after the existing-life step. + struct Birth { + LifeAccount account{}; + std::uint64_t request{}, generation{}, identity{}; + friend bool operator==(const Birth&, const Birth&) = default; + }; + std::vector births; friend bool operator==(const CellularTickIntents&, const CellularTickIntents&) = default; }; struct CellularBatch { @@ -35,14 +43,17 @@ struct CellularActionRecord { friend bool operator==(const CellularActionRecord&, const CellularActionRecord&) = default; }; -inline constexpr std::uint32_t cellular_batch_format = 1; +inline constexpr std::uint32_t cellular_batch_format = 2; +inline constexpr std::uint32_t cellular_batch_legacy_format = 1; inline constexpr std::size_t cellular_batch_header_bytes = 112; -inline constexpr std::size_t cellular_batch_tick_header_bytes = 4; +inline constexpr std::size_t cellular_batch_tick_header_bytes = 8; inline constexpr std::size_t cellular_batch_intent_bytes = 24; +inline constexpr std::size_t cellular_batch_birth_bytes = 40; inline constexpr std::size_t max_cellular_batch_ticks = 64; inline constexpr std::size_t max_cellular_batch_bytes = cellular_batch_header_bytes + max_cellular_batch_ticks * (cellular_batch_tick_header_bytes + - biology::max_cellular_tick_lives * cellular_batch_intent_bytes); + biology::max_cellular_tick_lives * + (cellular_batch_intent_bytes + cellular_batch_birth_bytes)); enum class CellularBatchError : std::uint8_t { format, version, scope, state }; inline constexpr std::size_t cellular_action_record_bytes = 48; inline constexpr std::size_t max_cellular_action_records = 1'024; diff --git a/migrations/0022_cellular_birth_trace_bound.sql b/migrations/0022_cellular_birth_trace_bound.sql new file mode 100644 index 0000000..7af3c4b --- /dev/null +++ b/migrations/0022_cellular_birth_trace_bound.sql @@ -0,0 +1,104 @@ +-- M11.3d-7h, ADR-0114: allow canonical cellular birth records in bounded traces. +CREATE OR REPLACE FUNCTION evolution.parse_checkpoint_manifest(m bytea) +RETURNS evolution.checkpoint_manifest LANGUAGE plpgsql IMMUTABLE +SET search_path = pg_catalog, pg_temp AS $$ +DECLARE + r evolution.checkpoint_manifest; + zero16 constant bytea := '\x00000000000000000000000000000000'; + zero32 constant bytea := decode(repeat('00', 32), 'hex'); + history numeric; + cap numeric; + ticks numeric; +BEGIN + IF m IS NULL OR octet_length(m) < 12 OR substring(m FROM 1 FOR 8) <> '\x4145564f52434d31' THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + IF evolution.manifest_uint(m, 8, 4) <> 2 THEN + RAISE EXCEPTION 'unsupported checkpoint manifest' USING ERRCODE = 'EVL06'; + END IF; + IF octet_length(m) <> 360 THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + r.profile := evolution.manifest_uint(m, 12, 4); + history := evolution.manifest_uint(m, 232, 4); + IF NOT ( + (((r.profile = 1 AND history = 0) OR (r.profile = 2 AND history = 1)) + AND substring(m FROM 161 FOR 36) = '\x020000000200000001000000010000000100000001000000010000000100000001000000') + OR (r.profile = 3 AND history = 2 + AND substring(m FROM 161 FOR 36) = '\x010000000100000001000000000000000000000001000000010000000100000001000000') + OR (r.profile = 4 AND history = 3 + AND substring(m FROM 161 FOR 36) = '\x010000000100000001000000010000000100000002000000010000000100000000000000') + ) + OR evolution.manifest_uint(m, 228, 4) <> 0 OR evolution.manifest_uint(m, 236, 4) <> 1 THEN + RAISE EXCEPTION 'unsupported checkpoint manifest' USING ERRCODE = 'EVL06'; + END IF; + cap := CASE r.profile WHEN 1 THEN 1048576 WHEN 2 THEN 4268144 + WHEN 4 THEN 16777216 ELSE 3266800 END; + r.world_id := evolution.manifest_uint(m, 16, 8); + r.region_id := evolution.manifest_uint(m, 24, 8); + r.frame_id := evolution.manifest_uint(m, 32, 8); + r.commit_id := encode(substring(m FROM 41 FOR 16), 'hex')::uuid; + r.commit_sequence := evolution.manifest_uint(m, 56, 8); + r.predecessor_commit := encode(substring(m FROM 65 FOR 16), 'hex')::uuid; + r.predecessor_digest := substring(m FROM 81 FOR 32); + r.begin_tick := evolution.manifest_uint(m, 112, 8); + r.end_tick := evolution.manifest_uint(m, 120, 8); + r.holder := encode(substring(m FROM 129 FOR 16), 'hex')::uuid; + r.epoch := evolution.manifest_uint(m, 144, 8); + r.tick_microseconds := evolution.manifest_uint(m, 152, 8); + r.versions := substring(m FROM 161 FOR 36); + r.build_identity := substring(m FROM 197 FOR 32); + r.payload_bytes := evolution.manifest_uint(m, 240, 8); + r.payload_sha256 := substring(m FROM 249 FOR 32); + r.trace_bytes := evolution.manifest_uint(m, 280, 8); + r.trace_sha256 := substring(m FROM 289 FOR 32); + r.decision_count := evolution.manifest_uint(m, 320, 4); + r.decisions_sha256 := substring(m FROM 329 FOR 32); + IF r.world_id = 0 OR r.region_id = 0 OR r.frame_id = 0 + OR substring(m FROM 41 FOR 16) = zero16 OR r.tick_microseconds = 0 + OR r.build_identity = zero32 OR r.payload_bytes < 1 OR r.payload_bytes > cap + OR r.payload_sha256 = zero32 OR evolution.manifest_uint(m, 324, 4) <> 0 THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + IF r.profile IN (3, 4) AND (r.tick_microseconds <> 50000 OR + (r.profile = 3 AND (r.decision_count <> 0 OR r.decisions_sha256 <> zero32)) OR + (r.commit_sequence = 0 AND + (substring(m FROM 65 FOR 16) <> zero16 OR r.predecessor_digest <> zero32))) THEN + RAISE EXCEPTION 'invalid lifecycle checkpoint' USING ERRCODE = '22023'; + END IF; + IF r.commit_sequence = 0 THEN + IF substring(m FROM 129 FOR 16) <> zero16 OR r.epoch <> 0 OR r.begin_tick <> r.end_tick + OR r.trace_bytes <> 0 OR r.trace_sha256 <> zero32 THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + IF substring(m FROM 65 FOR 16) = zero16 AND r.predecessor_digest = zero32 THEN + IF r.end_tick <> 0 OR r.decision_count <> 0 OR r.decisions_sha256 <> zero32 THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + r.predecessor_commit := NULL; + ELSIF substring(m FROM 65 FOR 16) = zero16 OR r.predecessor_digest = zero32 + OR r.predecessor_commit = r.commit_id OR r.profile IN (3, 4) + OR r.decision_count > 8 + OR (r.decision_count = 0) <> (r.decisions_sha256 = zero32) THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + r.holder := NULL; + ELSE + ticks := r.end_tick - r.begin_tick; + IF substring(m FROM 65 FOR 16) = zero16 OR r.predecessor_commit = r.commit_id + OR r.predecessor_digest = zero32 OR substring(m FROM 129 FOR 16) = zero16 + OR r.epoch = 0 OR r.end_tick <= r.begin_tick OR ticks > 64 + OR (r.profile IN (1, 2) AND r.trace_bytes <> 96 + 304 * ticks) + OR (r.profile = 3 AND (r.trace_bytes < 112 OR r.trace_bytes > 12400 + OR mod(r.trace_bytes - 112, 48) <> 0)) + OR (r.profile = 4 AND (r.trace_bytes < 112 + 4 * ticks + OR r.trace_bytes > 112 + 16400 * ticks)) + OR r.trace_sha256 = zero32 OR r.decision_count > 8 + OR (r.decision_count = 0) <> (r.decisions_sha256 = zero32) THEN + RAISE EXCEPTION 'malformed checkpoint manifest' USING ERRCODE = '22023'; + END IF; + END IF; + RETURN r; +END $$; + +INSERT INTO evolution.schema_migrations(version, sha256) VALUES (22, :'migration_sha_22'); diff --git a/migrations/apply.sql b/migrations/apply.sql index 12e4a99..42ebb18 100644 --- a/migrations/apply.sql +++ b/migrations/apply.sql @@ -95,27 +95,35 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') :'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2' || ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' || ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6' || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' || ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10' || ',11:' || :'migration_sha_11' || ',12:' || :'migration_sha_12' || ',13:' || :'migration_sha_13' || ',14:' || :'migration_sha_14' || ',15:' || :'migration_sha_15' || ',16:' || :'migration_sha_16' || ',17:' || :'migration_sha_17' || ',18:' || :'migration_sha_18' || ',19:' || :'migration_sha_19' AS at_v19, :'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2' || ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' || ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6' || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' || ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10' || ',11:' || :'migration_sha_11' || ',12:' || :'migration_sha_12' || ',13:' || :'migration_sha_13' || ',14:' || :'migration_sha_14' || ',15:' || :'migration_sha_15' || ',16:' || :'migration_sha_16' || ',17:' || :'migration_sha_17' || ',18:' || :'migration_sha_18' || ',19:' || :'migration_sha_19' || ',20:' || :'migration_sha_20' AS at_v20, :'history' = '1:' || :'migration_sha' || ',2:' || :'migration_sha_2' || ',3:' || :'migration_sha_3' || ',4:' || :'migration_sha_4' || ',5:' || :'migration_sha_5' || ',6:' || :'migration_sha_6' || ',7:' || :'migration_sha_7' || ',8:' || :'migration_sha_8' || ',9:' || :'migration_sha_9' || ',10:' || :'migration_sha_10' || ',11:' || :'migration_sha_11' || ',12:' || :'migration_sha_12' || ',13:' || :'migration_sha_13' || ',14:' || :'migration_sha_14' || ',15:' || :'migration_sha_15' || ',16:' || :'migration_sha_16' || ',17:' || :'migration_sha_17' || ',18:' || :'migration_sha_18' || ',19:' || :'migration_sha_19' || ',20:' || :'migration_sha_20' || ',21:' || :'migration_sha_21' AS at_v21 \gset - \if :at_v21 + SELECT right(:'history', length(',22:' || :'migration_sha_22')) = ',22:' || :'migration_sha_22' AS at_v22 \gset + \if :at_v22 + \elif :at_v21 + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v20 \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v19 \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v18 \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v17 \ir 0018_cellular_profile_admission.sql \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v16 \ir 0017_life_request_outcomes.sql \ir 0018_cellular_profile_admission.sql \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v15 \ir 0016_primitive_publication.sql \ir 0017_life_request_outcomes.sql @@ -123,6 +131,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v14 \ir 0015_life_reservations.sql \ir 0016_primitive_publication.sql @@ -131,6 +140,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v13 \ir 0014_probe_overdue.sql \ir 0015_life_reservations.sql @@ -140,6 +150,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v12 \ir 0013_probe_incoming.sql \ir 0014_probe_overdue.sql @@ -150,6 +161,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v11 \ir 0012_probe_handoff.sql \ir 0013_probe_incoming.sql @@ -161,6 +173,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v10 \ir 0011_region_actions.sql \ir 0012_probe_handoff.sql @@ -173,6 +186,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v9 \ir 0010_account_control.sql \ir 0011_region_actions.sql @@ -186,6 +200,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v8 \ir 0009_serialized_authority.sql \ir 0010_account_control.sql @@ -200,6 +215,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v7 \ir 0008_upgrade_barrier.sql \ir 0009_serialized_authority.sql @@ -215,6 +231,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v6 \ir 0007_region_clones.sql \ir 0008_upgrade_barrier.sql @@ -231,6 +248,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v5 \ir 0006_commit_lookup.sql \ir 0007_region_clones.sql @@ -248,6 +266,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v4 \ir pre_v5_guard.sql \ir 0005_batch_decisions.sql @@ -267,6 +286,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v3 \ir 0004_lineage_profile.sql \ir pre_v5_guard.sql @@ -287,6 +307,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v2 \ir 0003_region_checkpoints.sql \ir 0004_lineage_profile.sql @@ -308,6 +329,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \elif :at_v1 \ir 0002_region_fencing.sql \ir 0003_region_checkpoints.sql @@ -330,6 +352,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \else \echo 'Unknown or changed migration history' DO $$ BEGIN RAISE EXCEPTION 'unknown or changed migration history'; END $$; @@ -357,6 +380,7 @@ SELECT EXISTS (SELECT 1 FROM pg_catalog.pg_namespace WHERE nspname='evolution') \ir 0019_cellular_action_records.sql \ir 0020_cellular_publication.sql \ir 0021_cellular_trace_and_recovery.sql + \ir 0022_cellular_birth_trace_bound.sql \endif -- Effective-privilege audit on every run, including no-ops (ADR-0060). Attributes -- alone are insufficient: predefined roles such as pg_write_all_data, PUBLIC diff --git a/src/persistence/region_checkpoint.cpp b/src/persistence/region_checkpoint.cpp index 48f6f10..d8d0b96 100644 --- a/src/persistence/region_checkpoint.cpp +++ b/src/persistence/region_checkpoint.cpp @@ -94,12 +94,12 @@ bool structural(const CheckpointManifest& m) noexcept { if (m.manifest_version == 1) return link && no_batch; const auto ticks = m.end_tick - m.begin_tick; - const auto cellular_min = server::cellular_batch_header_bytes + - ticks * server::cellular_batch_tick_header_bytes; + const auto cellular_min = server::cellular_batch_header_bytes + ticks * 4; const auto cellular_max = server::cellular_batch_header_bytes + ticks * (server::cellular_batch_tick_header_bytes + biology::max_cellular_tick_lives * - server::cellular_batch_intent_bytes); + (server::cellular_batch_intent_bytes + + server::cellular_batch_birth_bytes)); const bool trace_shape = primitive ? b.trace_bytes >= server::primitive_batch_header_bytes && b.trace_bytes <= server::max_primitive_batch_bytes && diff --git a/src/server/lifecycle/cellular_batch.cpp b/src/server/lifecycle/cellular_batch.cpp index 8424b1e..0d2eb08 100644 --- a/src/server/lifecycle/cellular_batch.cpp +++ b/src/server/lifecycle/cellular_batch.cpp @@ -5,6 +5,7 @@ #include #include #include +#include #include #include @@ -39,8 +40,11 @@ bool valid_intent(const CellularLifeIntent& life) { step.movement.y <= movement_limit; } bool valid_ticks(std::span ticks) { + std::set identities; + std::set accounts; for (const auto& tick : ticks) { - if (tick.lives.size() > biology::max_cellular_tick_lives) + if (tick.lives.size() > biology::max_cellular_tick_lives || + tick.births.size() > biology::max_cellular_tick_lives) return false; for (std::size_t i = 0; i < tick.lives.size(); ++i) { const auto& life = tick.lives[i]; @@ -48,6 +52,17 @@ bool valid_ticks(std::span ticks) { (i > 0 && tick.lives[i - 1].life_id >= life.life_id)) return false; } + for (std::size_t i = 0; i < tick.births.size(); ++i) { + const auto& birth = tick.births[i]; + if (all_zero(birth.account) || birth.request == 0 || birth.generation == 0 || + birth.identity == 0 || + (i > 0 && std::tie(tick.births[i - 1].account, + tick.births[i - 1].request) >= + std::tie(birth.account, birth.request))) + return false; + if (!identities.insert(birth.identity).second || !accounts.insert(birth.account).second) + return false; + } } return true; } @@ -107,7 +122,8 @@ encode_cellular_batch(const CellularBatch& batch) { return std::unexpected{CellularBatchError::format}; std::size_t size = cellular_batch_header_bytes; for (const auto& tick : batch.ticks) { - const auto records = tick.lives.size() * cellular_batch_intent_bytes; + const auto records = tick.lives.size() * cellular_batch_intent_bytes + + tick.births.size() * cellular_batch_birth_bytes; if (records > max_cellular_batch_bytes - size - cellular_batch_tick_header_bytes) return std::unexpected{CellularBatchError::format}; size += cellular_batch_tick_header_bytes + records; @@ -129,8 +145,15 @@ encode_cellular_batch(const CellularBatch& batch) { put(out, 0, 4); for (const auto& tick : batch.ticks) { put(out, tick.lives.size(), 4); + put(out, tick.births.size(), 4); for (const auto& life : tick.lives) put_intent(out, life); + for (const auto& birth : tick.births) { + out.insert(out.end(), birth.account.begin(), birth.account.end()); + put(out, birth.request, 8); + put(out, birth.generation, 8); + put(out, birth.identity, 8); + } } if (out.size() != size) return std::unexpected{CellularBatchError::format}; @@ -142,7 +165,9 @@ decode_cellular_batch(std::span bytes) { if (bytes.size() < cellular_batch_header_bytes || bytes.size() > max_cellular_batch_bytes || !std::ranges::equal(bytes.first(magic.size()), magic)) return std::unexpected{CellularBatchError::format}; - if (get(bytes, 8, 4) != cellular_batch_format || get(bytes, 12, 4) != 4) + const auto format = get(bytes, 8, 4); + if ((format != cellular_batch_format && format != cellular_batch_legacy_format) || + get(bytes, 12, 4) != 4) return std::unexpected{CellularBatchError::version}; if (get(bytes, 108, 4) != 0) return std::unexpected{CellularBatchError::format}; @@ -166,12 +191,21 @@ decode_cellular_batch(std::span bytes) { std::size_t offset = cellular_batch_header_bytes; out.ticks.reserve(static_cast(tick_count)); for (std::uint64_t tick_index = 0; tick_index < tick_count; ++tick_index) { - if (bytes.size() - offset < cellular_batch_tick_header_bytes) + const auto tick_header_bytes = format == cellular_batch_format + ? cellular_batch_tick_header_bytes + : sizeof(std::uint32_t); + if (bytes.size() - offset < tick_header_bytes) return std::unexpected{CellularBatchError::format}; const auto count = get(bytes, offset, 4); - offset += cellular_batch_tick_header_bytes; + offset += 4; + std::uint64_t birth_count{}; + if (format == cellular_batch_format) { + birth_count = get(bytes, offset, 4); + offset += 4; + } if (count > biology::max_cellular_tick_lives || - count > (bytes.size() - offset) / cellular_batch_intent_bytes) + count > (bytes.size() - offset) / cellular_batch_intent_bytes || + birth_count > biology::max_cellular_tick_lives) return std::unexpected{CellularBatchError::format}; CellularTickIntents step; step.lives.reserve(static_cast(count)); @@ -198,6 +232,25 @@ decode_cellular_batch(std::span bytes) { step.lives.push_back(life); offset += cellular_batch_intent_bytes; } + if (birth_count > (bytes.size() - offset) / cellular_batch_birth_bytes) + return std::unexpected{CellularBatchError::format}; + step.births.reserve(static_cast(birth_count)); + for (std::uint64_t i = 0; i < birth_count; ++i) { + const auto record = bytes.subspan(offset, cellular_batch_birth_bytes); + CellularTickIntents::Birth birth; + std::ranges::copy(record.first(16), birth.account.begin()); + birth.request = get(record, 16, 8); + birth.generation = get(record, 24, 8); + birth.identity = get(record, 32, 8); + if (all_zero(birth.account) || birth.request == 0 || birth.generation == 0 || + birth.identity == 0 || + (!step.births.empty() && + std::tie(step.births.back().account, step.births.back().request) >= + std::tie(birth.account, birth.request))) + return std::unexpected{CellularBatchError::format}; + step.births.push_back(birth); + offset += cellular_batch_birth_bytes; + } out.ticks.push_back(std::move(step)); } if (offset != bytes.size()) @@ -217,9 +270,19 @@ replay_cellular_batch(const CellularLives& base, const CellularBatch& batch) { base.tick()); if (!replay) return std::unexpected{CellularBatchError::state}; - for (const auto& tick : batch.ticks) + for (const auto& tick : batch.ticks) { if (!(*replay)->advance_tick(tick.lives)) return std::unexpected{CellularBatchError::state}; + for (const auto& birth : tick.births) { + const LifeRequest request{birth.account, birth.request}; + const LifeAllocation allocation{birth.identity, birth.identity, birth.identity, + birth.identity, {(*replay)->tick()}}; + auto born = (*replay)->birth_candidate(request, allocation); + if (!born) + return std::unexpected{CellularBatchError::state}; + *replay = std::move(*born); + } + } return std::move(*replay); } diff --git a/tests/cellular_lives_tests.cpp b/tests/cellular_lives_tests.cpp index 6bffc43..bf7caa6 100644 --- a/tests/cellular_lives_tests.cpp +++ b/tests/cellular_lives_tests.cpp @@ -94,6 +94,38 @@ void reserved_birth_is_funded_and_detached() { CHECK((*born)->advance_tick(next_tick) && (*born)->tick() == 2 && (*born)->lives().size() == 1); + auto trace_base = empty_cellular_owner(founder_price(1), {{0, 0}}); + CellularBatch birth_batch; + birth_batch.header = {trace_base->world(), trace_base->region(), trace_base->frame(), {}, {}, + trace_base->tick(), trace_base->tick() + 1}; + birth_batch.header.predecessor.fill(3); + birth_batch.header.predecessor_digest.fill(4); + CellularTickIntents birth_tick; + birth_tick.births.push_back({request.account, request.action, 1, 801}); + birth_batch.ticks.push_back(birth_tick); + const auto birth_bytes = encode_cellular_batch(birth_batch); + CHECK(birth_bytes); + const auto decoded_births = decode_cellular_batch(*birth_bytes); + CHECK(decoded_births && *decoded_births == birth_batch); + const auto replayed_birth = replay_cellular_batch(*trace_base, *decoded_births); + CHECK(replayed_birth && (*replayed_birth)->tick() == 1 && + (*replayed_birth)->lives().size() == 1 && + (*replayed_birth)->lives()[0].identity.life == 801 && + (*replayed_birth)->lives()[0].account == request.account && + (*replayed_birth)->archive().recorded() == 1 && + (*replayed_birth)->current_total() == (*replayed_birth)->opening_total()); + auto legacy_batch = birth_batch; + legacy_batch.ticks[0].births.clear(); + auto legacy_bytes = encode_cellular_batch(legacy_batch); + CHECK(legacy_bytes); + (*legacy_bytes)[8] = static_cast(cellular_batch_legacy_format); + legacy_bytes->erase(legacy_bytes->begin() + 116, legacy_bytes->begin() + 120); + const auto decoded_legacy = decode_cellular_batch(*legacy_bytes); + CHECK(decoded_legacy && decoded_legacy->ticks[0].births.empty()); + auto invalid_birth_batch = birth_batch; + invalid_birth_batch.ticks[0].births[0].generation = 0; + CHECK(encode_cellular_batch(invalid_birth_batch).error() == CellularBatchError::format); + auto unfunded = empty_cellular_owner({}, {{0, 0}}); CHECK(unfunded->advance_tick(no_lives)); const auto unfunded_snapshot = unfunded->snapshot(); diff --git a/tests/region_migration_tests.cpp b/tests/region_migration_tests.cpp index 319ceb5..9889e61 100644 --- a/tests/region_migration_tests.cpp +++ b/tests/region_migration_tests.cpp @@ -187,6 +187,7 @@ void upgrade_across_migration_and_barrier() { " -v migration_sha_19=$(s 0019_cellular_action_records.sql)" " -v migration_sha_20=$(s 0020_cellular_publication.sql)" " -v migration_sha_21=$(s 0021_cellular_trace_and_recovery.sql)" + " -v migration_sha_22=$(s 0022_cellular_birth_trace_bound.sql)" " -v runtime_role=evo_runtime_test -f migrations/apply.sql" " > /dev/null 2>&1") != 0); CHECK(history(*owner_db) == "1,2,3,4,5,6,7"); @@ -202,7 +203,8 @@ void upgrade_across_migration_and_barrier() { CHECK(shell("PGDATABASE=" + database + " EVO_PG_RUNTIME_ROLE=evo_runtime_test bash \"$EVO_SOURCE_ROOT/" "tools/migrate-postgres.sh\" > /dev/null") == 0); - CHECK(history(*owner_db) == "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21"); + CHECK(history(*owner_db) == + "1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22"); CHECK(scalar(*owner_db, "SELECT count(*) FROM information_schema.columns WHERE " "table_schema = 'evolution' AND table_name = 'region_actions' AND " "column_name IN ('input_bytes', 'result')") == "2"); diff --git a/tools/ci/postgres-check.sh b/tools/ci/postgres-check.sh index 0cdd1aa..f427db3 100644 --- a/tools/ci/postgres-check.sh +++ b/tools/ci/postgres-check.sh @@ -69,7 +69,7 @@ PGDATABASE=upgrade_path psql -X -v ON_ERROR_STOP=1 -1 -v "migration_sha=$v1_sha" -f "$root/migrations/0001_region_bootstrap.sql" > /dev/null test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',') FROM evolution.schema_migrations")" = 1 PGDATABASE=upgrade_path bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21 +test "$(PGDATABASE=upgrade_path psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22 # v2 heads have no payload binding: v2->v3 fails closed and leaves history at v2. v2_sha="$(sha256sum "$root/migrations/0002_region_fencing.sql")" v2_sha="${v2_sha%% *}" @@ -90,7 +90,7 @@ PGDATABASE=v3_commits psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '01000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32), 'hex'))" > /dev/null PGDATABASE=v3_commits bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 21:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21 +test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) || ':' || string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 22:1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22 test "$(PGDATABASE=v3_commits psql -XAtc "SELECT count(*) FROM evolution.region_commits")" = 1 # Review P1: a v4 database holding a published (possibly terminal) checkpoint has # no durable decisions for it. The v5 upgrade must refuse before changing anything. @@ -144,7 +144,7 @@ PGDATABASE=v6_populated psql -X -v ON_ERROR_STOP=1 \ -c "INSERT INTO evolution.retained_builds VALUES (decode(repeat('b1', 32), 'hex'))" \ -c "SELECT evolution.provision_region_checkpoint(decode('4145564f52434d31' || '02000000' || '01000000' || '0100000000000000' || '0100000000000000' || '0900000000000000' || repeat('11', 16) || repeat('00', 8) || repeat('00', 16) || repeat('00', 32) || repeat('00', 16) || repeat('00', 16) || repeat('00', 8) || '50c3000000000000' || '020000000200000001000000010000000100000001000000010000000100000001000000' || repeat('b1', 32) || '00000000' || '00000000' || '01000000' || '6400000000000000' || repeat('a0', 32) || repeat('00', 80), 'hex'))" > /dev/null PGDATABASE=v6_populated bash "$root/tools/migrate-postgres.sh" -test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21 +test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(version::text, ',' ORDER BY version) FROM evolution.schema_migrations")" = 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT count(*) FROM evolution.region_commits c, LATERAL evolution.parse_checkpoint_manifest(c.manifest) m WHERE m.commit_sequence = 0 AND m.predecessor_commit IS NULL")" = 1 test "$(PGDATABASE=v6_populated psql -XAtc "SELECT string_agg(conname, ',') FROM pg_constraint WHERE conrelid = 'evolution.region_heads'::regclass AND contype = 'c' AND pg_get_constraintdef(oid) LIKE '%end_tick%'")" = region_heads_published_tick # Inherited privileges bypass fencing, so every migrate run audits effective rights. @@ -190,9 +190,9 @@ expect_audit_failure 'GRANT SELECT ON evolution.account_controls TO evo_runtime_ expect_audit_failure 'REVOKE EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) FROM evo_runtime_test' 'GRANT EXECUTE ON FUNCTION evolution.claim_account_control(evolution.u64, evolution.u64, uuid, evolution.u64, uuid, evolution.u64, evolution.u64, integer) TO evo_runtime_test' bash "$root/tools/migrate-postgres.sh" # Restored configuration passes the audit again. # Unknown future versions and checksum drift must not be repaired silently. -psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=22 WHERE version=21' +psql -X -v ON_ERROR_STOP=1 -c 'UPDATE evolution.schema_migrations SET version=23 WHERE version=22' if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi -psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=21 WHERE version=22" +psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET version=22 WHERE version=23" psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=14" if bash "$root/tools/migrate-postgres.sh"; then exit 1; fi psql -X -v ON_ERROR_STOP=1 -c "UPDATE evolution.schema_migrations SET sha256=repeat('0',64) WHERE version=13" diff --git a/tools/ci/upgrade-check.sh b/tools/ci/upgrade-check.sh index be94c33..74c0980 100644 --- a/tools/ci/upgrade-check.sh +++ b/tools/ci/upgrade-check.sh @@ -61,7 +61,7 @@ SQL [ "$code" != 0 ] && grep -q 'division by zero' "$work/migration.log" || fail "the rehearsal migration did not fail" [ "$(q -c "SELECT count(*) FROM information_schema.columns WHERE table_name = 'region_commits' AND column_name = 'rehearsal_column'")" = 0 ] || fail "the failed migration left a column" -[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 21 ] || fail "schema version moved" +[ "$(q -c 'SELECT max(version) FROM evolution.schema_migrations')" = 22 ] || fail "schema version moved" wait_ack "$work/b1.log" $((before + 40)) || fail "b1 did not resume after the failed migration" kill -0 "$b1_pid" 2> /dev/null || fail "b1 exited during the failed migration" bash "$root/tools/migrate-postgres.sh" > /dev/null || fail "migrate after the failure" diff --git a/tools/migrate-postgres.sh b/tools/migrate-postgres.sh index d3a00f9..dce3de9 100644 --- a/tools/migrate-postgres.sh +++ b/tools/migrate-postgres.sh @@ -54,6 +54,8 @@ digest_20="$(sha256sum "$root/migrations/0020_cellular_publication.sql")" digest_20="${digest_20%% *}" digest_21="$(sha256sum "$root/migrations/0021_cellular_trace_and_recovery.sql")" digest_21="${digest_21%% *}" +digest_22="$(sha256sum "$root/migrations/0022_cellular_birth_trace_bound.sql")" +digest_22="${digest_22%% *}" psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \ --set="migration_sha_2=$digest_2" --set="migration_sha_3=$digest_3" \ --set="migration_sha_4=$digest_4" --set="migration_sha_5=$digest_5" \ @@ -68,5 +70,6 @@ psql -X --no-password --set=ON_ERROR_STOP=1 --set="migration_sha=$digest" \ --set="migration_sha_19=$digest_19" \ --set="migration_sha_20=$digest_20" \ --set="migration_sha_21=$digest_21" \ + --set="migration_sha_22=$digest_22" \ --set="runtime_role=$runtime_role" \ --file="$root/migrations/apply.sql"