Files
aevolution/tools/ci/deploy-policy-check.py
dingoandClaude Opus 5.5 46c106409e
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (client) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 4s
Foundation / checks (network) (push) Successful in 4s
Foundation / checks (postgres) (push) Successful in 4s
Foundation / checks (tsan) (push) Successful in 5s
Foundation / checks (deploy) (push) Successful in 4m33s
Foundation / postgres-tools (push) Successful in 2m14s
Foundation / server-image (push) Successful in 5m21s
Foundation / image (push) Successful in 6m18s
feat(admin): session windows in SQL, opened from the admin portal (schema 62)
The dev session window moves from Git to SQL (ADR-0087 amendment): the admin
portal opens, extends (30 min to 4 h from now) and closes it, and the cellular
region re-reads it every 5 s with --session-window-source db. No push, sync or
restart per window; a closed window keeps the region running instead of
crash-looping.

- Schema 62: append-only session_windows (4 h cap, no rewrites),
  load_session_window_v1 for the runtime group, admin_set_session_window_v1
  and admin_session_windows_v1 for a dedicated admin group role, granted and
  audited each migration run when EVO_PG_ADMIN_ROLE is set (aevolution_admin in
  DEV). Schema 61's views are revoked from the runtime group.
- RegionSessions admission switch: while closed every Hello is refused before
  signature work and closing closes open sessions (new sessions were still
  admitted after expiry before).
- Admin API: GET/POST /api/regions/<n>/session-window, window state in the
  regions list; the portal's session-window card with history.
- Git: REGION_SESSION_WINDOW and the pre-push refresh rule are removed; the
  policy check requires the database source and refuses a committed window.

Needs the new image and the aevolution_admin role (with the portal's login in
aevolution-admin-pg) before a sync.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 15:13:02 -04:00

768 lines
47 KiB
Python

#!/usr/bin/env python3
"""M8.2 offline deployment-policy check (ADR-0069). Needs Python 3 and PyYAML only.
Parses every manifest under deploy/ (including .yaml.example files) and fails if an
environment-isolation or World0-protection invariant is broken:
- no Secret objects in Git;
- every ServiceAccount disables token automounting;
- each platform overlay: labelled restricted namespace, quota, limit range,
default-deny and DNS-only policies, three service accounts; World0 alone is
labelled protected;
- the World0 admission policy denies what ADR-0069 lists, fails closed and is
bound with Deny;
- every AEvolution AppProject targets exactly its own namespace with no cluster
kinds; World0 blocks automated sync; no Application auto-syncs or carries the
cascade-delete finalizer, and each stays in its own environment;
- the live Argo path (deploy/kubernetes/dev) holds no Namespace, Secret, quota or
RBAC objects;
- the cleanup tool refuses World0 before invoking kubectl;
- World0 launch stays blocked (ADR-0075): no workload targets World0 and its
Application keeps an unresolved path; any image a World0 overlay pins must be
the digest already pinned in stage (promotion, never a fresh build);
- scheduling (ADR-0076): every pod that mounts the node-local objects volume
selects the store node; aevolution.io/compute is only ever preferred, never
required; upgrade-rehearsal flags never appear in a deployed workload;
- DEV placement (ADR-0170): every required node-affinity alternative excludes
both control-plane roles; direct nodeName bypasses are forbidden;
- the Pods the backup and barrier scripts generate (rendered with --print-manifest)
keep the database-tool security contract: non-root 10001 with fsGroup 10001 so
the 0440 Secret files are readable, no token, read-only root, no escalation;
- the dev session route (ADR-0087): loopback listener everywhere except the
evolution-dev route component; that component is the only LoadBalancer and the
only Service on the session port, narrowly shaped, and it cannot be enabled while
markers are unresolved or before server-enforced session windows exist;
- rendered output (ADR-0088): every overlay is built with the pinned kustomize and
the effective objects are checked (namespaces, listener/session/input environment,
arguments, ports, Services, quotas, policies, Ingresses, Secrets), so inline or
JSON patches and namespace transformations cannot bypass the source checks.
"""
import ipaddress
import re
# Enabling the dev session route needs ADR-0087 to be amended with the implemented
# session-window argument contract and runtime expiry tests (expiry during a session,
# restart after expiry, delayed application, extend/close races, clock movement,
# synthetic inputs stopping while commits drain). The runtime contract and focused
# expiry tests are now implemented in the M10 QUIC
# bridge. The route remains constrained to the dev overlay. Since the ADR-0087 amendment of
# 2026-10-27 the window lives in SQL (evolution.session_windows, set from the admin portal)
# and the region re-reads it; Git carries no window.
ROUTE_ENABLING_PERMITTED = True
# Addresses the route may use: RFC 1918 plus internal ranges the maintainer has
# explicitly approved in ADR-0087 (none yet; the cluster LAN 172.10.0.0/16 is not
# RFC 1918 and needs that approval before use).
APPROVED_INTERNAL = ['172.10.1.0/24']
INTERNAL = [ipaddress.ip_network(n) for n in ('10.0.0.0/8', '172.16.0.0/12', '192.168.0.0/16')] + \
[ipaddress.ip_network(n) for n in APPROVED_INTERNAL]
def internal(network):
return any(network.version == r.version and network.subnet_of(r) for r in INTERNAL)
import os
import shutil
import subprocess
import sys
import tempfile
import yaml
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..'))
failures = []
def check(condition, message):
if not condition:
failures.append(message)
def documents(path):
with open(path, encoding='utf-8') as f:
return [d for d in yaml.safe_load_all(f) if isinstance(d, dict)]
manifests = {}
for base, _, files in os.walk(os.path.join(ROOT, 'deploy')):
for name in files:
if name.endswith(('.yaml', '.yml', '.yaml.example')):
path = os.path.join(base, name)
manifests[os.path.relpath(path, ROOT).replace(os.sep, '/')] = documents(path)
objects = [(p, d) for p, ds in manifests.items() for d in ds
if 'kind' in d and d['kind'] not in ('Kustomization', 'Component')]
# Files used as kustomize patches hold partial objects (checked only for what they set).
patch_files = set()
for p, ds in manifests.items():
if p.endswith('kustomization.yaml') and ds:
for entry in ds[0].get('patches', []):
target = entry.get('path') if isinstance(entry, dict) else entry
if target:
patch_files.add(os.path.normpath(os.path.join(os.path.dirname(p), target)).replace(os.sep, '/'))
# Secrets and service-account tokens.
for path, doc in objects:
check(doc['kind'] != 'Secret', f'{path}: Secret objects never belong in Git')
if doc['kind'] == 'ServiceAccount':
check(doc.get('automountServiceAccountToken') is False,
f'{path}: ServiceAccount {doc["metadata"]["name"]} must not automount a token')
# Platform overlays.
platform = 'deploy/kubernetes/platform'
base = {d['kind'] + '/' + d['metadata']['name']: d
for p, d in objects if p.startswith(platform + '/base/')}
for key in ('ResourceQuota/aevolution-quota', 'LimitRange/aevolution-defaults',
'NetworkPolicy/aevolution-default-deny', 'NetworkPolicy/aevolution-allow-dns',
'ServiceAccount/aevolution-migrator', 'ServiceAccount/aevolution-region',
'ServiceAccount/aevolution-backup'):
check(key in base, f'{platform}/base: missing {key}')
deny = base.get('NetworkPolicy/aevolution-default-deny', {}).get('spec', {})
check(deny.get('podSelector') == {} and sorted(deny.get('policyTypes', [])) == ['Egress', 'Ingress']
and not deny.get('ingress') and not deny.get('egress'),
'default-deny must select every pod and allow nothing')
dns = base.get('NetworkPolicy/aevolution-allow-dns', {}).get('spec', {})
dns_ports = [p.get('port') for rule in dns.get('egress', []) for p in rule.get('ports', [])]
check(dns.get('policyTypes') == ['Egress'] and set(dns_ports) == {53} and len(dns.get('egress', [])) == 1,
'allow-dns must allow only egress to port 53')
quota = base.get('ResourceQuota/aevolution-quota', {}).get('spec', {}).get('hard', {})
check(quota.get('services.loadbalancers') == '0' and quota.get('services.nodeports') == '0',
'quota must forbid LoadBalancer and NodePort services')
# ADR-0087: only the dev overlay may raise LoadBalancers, to exactly one, and only
# together with the session route; node ports stay forbidden everywhere.
ROUTE = 'deploy/kubernetes/region/dev-session-route/'
route_users = sorted(p for p, ds in manifests.items() if p.endswith('kustomization.yaml') and ds
and any('dev-session-route' in str(e) for e in ds[0].get('resources', []) + ds[0].get('components', [])))
route_enabled = route_users == ['deploy/kubernetes/region/dev/kustomization.yaml']
check(route_users in ([], ['deploy/kubernetes/region/dev/kustomization.yaml']),
f'only region/dev may include the dev session route: {route_users}')
for path, doc in objects:
if doc['kind'] == 'ResourceQuota' and path != f'{platform}/base/quota.yaml':
hard = doc.get('spec', {}).get('hard', {})
check(hard.get('services.nodeports', '0') == '0', f'{path}: node ports stay forbidden')
allowed = ('0', '1') if path.startswith(f'{platform}/dev/') and route_enabled else ('0',)
check(hard.get('services.loadbalancers', '0') in allowed,
f'{path}: LoadBalancers only in evolution-dev, at most one, with the session route')
for env in ('dev', 'stage', 'world0'):
k = f'{platform}/{env}/kustomization.yaml'
ns_path = f'{platform}/{env}/namespace.yaml'
check(k in manifests and ns_path in manifests, f'{platform}/{env}: overlay missing')
if k not in manifests or ns_path not in manifests:
continue
kustomization = manifests[k][0]
check(kustomization.get('namespace') == f'evolution-{env}' and '../base' in kustomization.get('resources', []),
f'{k}: must set namespace evolution-{env} and include ../base')
labels = manifests[ns_path][0]['metadata'].get('labels', {})
check(manifests[ns_path][0]['metadata']['name'] == f'evolution-{env}', f'{ns_path}: wrong name')
check(labels.get('aevolution.io/environment') == env, f'{ns_path}: environment label')
check(labels.get('pod-security.kubernetes.io/enforce') == 'restricted', f'{ns_path}: restricted pod security')
check((labels.get('aevolution.io/protected') == 'true') == (env == 'world0'),
f'{ns_path}: only world0 is protected')
# World0 admission policy.
policy = [d for p, d in objects if d['kind'] == 'ValidatingAdmissionPolicy'
and d['metadata']['name'] == 'aevolution-world0-protection']
binding = [d for p, d in objects if d['kind'] == 'ValidatingAdmissionPolicyBinding'
and d['metadata']['name'] == 'aevolution-world0-protection']
check(len(policy) == 1 and len(binding) == 1, 'world0 admission policy and binding must exist once')
if policy and binding:
spec = policy[0]['spec']
check(spec.get('failurePolicy') == 'Fail', 'world0 policy must fail closed')
check(binding[0]['spec'].get('validationActions') == ['Deny'] and 'paramRef' not in binding[0]['spec']
and 'matchResources' not in binding[0]['spec'], 'world0 binding must Deny everywhere')
covered = {(r, op) for rule in spec['matchConstraints']['resourceRules']
for r in rule['resources'] for op in rule['operations']}
for resource in ('namespaces', 'persistentvolumes', 'persistentvolumeclaims', 'secrets',
'serviceaccounts', 'applications', 'appprojects'):
check((resource, 'DELETE') in covered, f'world0 policy must cover DELETE {resource}')
for resource in ('namespaces', 'persistentvolumes'):
check((resource, 'UPDATE') in covered, f'world0 policy must cover UPDATE {resource}')
text = yaml.safe_dump(spec)
for needle in ("aevolution:world0-break-glass", "evolution-world0", "aevolution-world0",
"aevolution-platform",
"aevolution.io/protected", "Retain",
"has(object.spec.claimRef.namespace)",
"object.spec.claimRef.name == oldObject.spec.claimRef.name"):
check(needle in text, f'world0 policy must reference {needle}')
# Argo CD projects and applications.
for path, doc in objects:
name = doc['metadata'].get('name', '')
if doc['kind'] == 'AppProject' and name == 'aevolution-platform':
spec = doc['spec']
namespaces = sorted(d.get('namespace') for d in spec.get('destinations', []))
check(namespaces == ['aevolution-monitoring', 'evolution-dev', 'evolution-stage', 'evolution-world0'],
f'{path}: the platform project targets exactly the AEvolution namespaces')
cluster = sorted((w.get('group'), w.get('kind')) for w in spec.get('clusterResourceWhitelist', []))
check(cluster == [('', 'Namespace'),
('admissionregistration.k8s.io', 'ValidatingAdmissionPolicy'),
('admissionregistration.k8s.io', 'ValidatingAdmissionPolicyBinding'),
('storage.k8s.io', 'StorageClass')],
f'{path}: the platform project may own only its four cluster kinds')
kinds = {w.get('kind') for w in spec.get('namespaceResourceWhitelist', [])}
check(not kinds & {'Secret', 'Role', 'RoleBinding', '*'}, f'{path}: platform whitelists a forbidden kind')
elif doc['kind'] == 'AppProject' and name == 'aevolution-monitoring':
spec = doc['spec']
check([d.get('namespace') for d in spec.get('destinations', [])] == ['aevolution-monitoring']
and spec.get('clusterResourceWhitelist') == [],
f'{path}: monitoring targets only its namespace, with no cluster kinds')
elif doc['kind'] == 'AppProject' and name.startswith('aevolution-'):
env = name[len('aevolution-'):]
spec = doc['spec']
namespaces = [d.get('namespace') for d in spec.get('destinations', [])]
check(namespaces == [f'evolution-{env}'], f'{path}: {name} must target only evolution-{env}')
check(spec.get('clusterResourceWhitelist') == [], f'{path}: {name} must allow no cluster kinds')
kinds = {w.get('kind') for w in spec.get('namespaceResourceWhitelist', [])}
check(not kinds & {'Secret', 'Namespace', 'ResourceQuota', 'LimitRange', 'Role', 'RoleBinding', '*'},
f'{path}: {name} whitelists a forbidden kind')
if env == 'world0':
windows = spec.get('syncWindows', [])
check(any(w.get('kind') == 'deny' and w.get('manualSync') is True and w.get('duration') == '24h'
and w.get('schedule') == '* * * * *' for w in windows),
f'{path}: world0 must deny automated sync at all times')
check('ServiceAccount' not in kinds, f'{path}: world0 service accounts are platform-managed')
if doc['kind'] == 'Application':
spec = doc['spec']
project = spec.get('project', '')
destination = spec.get('destination', {}).get('namespace', '')
automated = (spec.get('syncPolicy') or {}).get('automated')
if name == 'aevolution-platform':
# Self-heal only: never prune, so no Git change can delete durable objects.
check(automated is not None and automated.get('prune') is False,
f'{path}: the platform app self-heals and never prunes')
check(spec.get('source', {}).get('path') == 'deploy/kubernetes/platform/cluster',
f'{path}: the platform app syncs deploy/kubernetes/platform/cluster')
else:
check(automated is None, f'{path}: {name} must not auto-sync')
check('resources-finalizer.argocd.argoproj.io' not in doc['metadata'].get('finalizers', []),
f'{path}: {name} must not cascade-delete')
if project.startswith('aevolution-') and project not in ('aevolution-platform', 'aevolution-monitoring'):
check(destination == 'evolution-' + project[len('aevolution-'):],
f'{path}: {name} leaves its project environment')
check(destination != 'evolution-world0' or name in ('aevolution-world0', 'aevolution-platform'),
f'{path}: only aevolution-world0 may target evolution-world0')
# M8.3 storage (ADR-0070): durable volumes retain their data; database passwords are
# only ever mounted as files, never placed in the environment.
classes = {d['metadata']['name']: d for p, d in objects if d['kind'] == 'StorageClass'}
retain = classes.get('aevolution-local-retain')
check(retain is not None and retain.get('reclaimPolicy') == 'Retain'
and retain.get('volumeBindingMode') == 'WaitForFirstConsumer'
and 'storageclass.kubernetes.io/is-default-class' not in retain['metadata'].get('annotations', {}),
'aevolution-local-retain must be a non-default Retain class that binds on first consumer')
for path, doc in objects:
if doc['kind'] == 'PersistentVolumeClaim' and path.startswith(platform + '/'):
check(doc['spec'].get('storageClassName') == 'aevolution-local-retain',
f'{path}: platform volumes must use aevolution-local-retain')
if doc['kind'] in ('Job', 'Deployment', 'StatefulSet', 'Pod'):
pod = doc['spec'].get('template', {}).get('spec', doc['spec'])
for container in pod.get('containers', []) + pod.get('initContainers', []):
check(not any('secretRef' in source for source in container.get('envFrom', [])),
f'{path}: never import a whole Secret into the environment')
for variable in container.get('env', []):
ref = variable.get('valueFrom', {}).get('secretKeyRef', {})
check(ref.get('key') not in ('PGPASSWORD', 'POSTGRES_PASSWORD', 'PGUSER', 'POSTGRES_USER'),
f'{path}: {variable.get("name")} must not come from {ref.get("key")}')
if pod.get('volumes') and 'store' in doc['metadata'].get('name', ''):
check(doc['metadata'].get('namespace') != 'evolution-world0',
f'{path}: store Jobs never target World0')
# M8.4 region host (ADR-0071): one owner at a time, readiness means authority.
for path, doc in objects:
# Overlay patches (region-values.yaml) are partial objects; the base is complete.
if (doc['kind'] == 'Deployment' and doc['metadata'].get('name') == 'evo-region'
and path not in patch_files):
spec = doc['spec']
pod = spec['template']['spec']
container = pod['containers'][0]
check(spec.get('replicas') == 1 and spec.get('strategy', {}).get('type') == 'Recreate',
f'{path}: the region host runs one replica with the Recreate strategy')
check(container.get('readinessProbe', {}).get('httpGet', {}).get('path') == '/readyz'
and container.get('livenessProbe', {}).get('httpGet', {}).get('path') == '/livez',
f'{path}: readiness must be /readyz and liveness /livez')
limits = container.get('resources', {}).get('limits', {})
check('cpu' not in limits and 'memory' in limits,
f'{path}: memory limit required, no CPU limit (throttling delays renewals)')
script = ' '.join(container.get('args', []))
drain = [int(x) for x in __import__('re').findall(r'--drain-ms (\d+)', script)]
check(len(drain) == 1 and pod.get('terminationGracePeriodSeconds', 30) * 1000 > drain[0],
f'{path}: the termination grace period must exceed --drain-ms')
# M8.6 (ADR-0073): synthetic load is a drill-time override, never committed.
for path, doc in objects:
if path.startswith('deploy/kubernetes/region/') and doc['kind'] == 'Deployment':
for container in doc['spec']['template']['spec'].get('containers', []):
for variable in container.get('env', []):
if variable.get('name') == 'REGION_INPUTS':
check(variable.get('value') == '', f'{path}: synthetic inputs are never enabled in Git')
# M9.7 (ADR-0083): dev sessions are a drill-time override as well.
if variable.get('name') == 'REGION_SESSION':
expected = 'dev' if path.startswith(ROUTE) else ''
check(variable.get('value') == expected,
f'{path}: dev sessions are enabled only by the reviewed route component')
# M9.7 (ADR-0083) and M9.8f (ADR-0087): the session listener is loopback (kubectl
# port-forward) everywhere except the evolution-dev route component; its Secret is
# optional and 0440; only that component's Service exposes the session port.
for path, doc in objects:
if path.startswith('deploy/kubernetes/region/') and doc['kind'] == 'Deployment':
pod = doc['spec']['template']['spec']
for container in pod.get('containers', []):
for variable in container.get('env', []):
if variable.get('name') == 'REGION_SESSION_LISTEN':
expected = '0.0.0.0:7400' if path.startswith(ROUTE) else '127.0.0.1:7400'
check(variable.get('value') == expected,
f'{path}: REGION_SESSION_LISTEN must be {expected} here')
if path in patch_files:
continue
script = ' '.join(pod['containers'][0].get('args', []))
check('--m10-listen "$REGION_SESSION_LISTEN"' in script and '0.0.0.0' not in script.replace('--ops-listen 0.0.0.0:9100', ''),
f'{path}: the session listener address comes only from REGION_SESSION_LISTEN')
for volume in pod.get('volumes', []):
if volume.get('secret', {}).get('secretName') == 'evolution-dev-session':
check(volume['secret'].get('optional') is True and volume['secret'].get('defaultMode') == 0o440,
f'{path}: the dev session Secret is optional and 0440')
# The M10 session is QUIC (ADR-0090, ADR-0087 amendment 2026-10-05): UDP 7400 only.
ROUTE_PORT = [{'name': 'session', 'port': 7400, 'targetPort': 'session', 'protocol': 'UDP'}]
REGION_SELECTOR = {'app.kubernetes.io/name': 'evo-region'}
def validate_route_service(where, doc):
"""The route Service's full reviewed shape; returns its source ranges."""
spec = doc['spec']
ranges = spec.get('loadBalancerSourceRanges', [])
check(spec.get('type') == 'LoadBalancer' and spec.get('externalTrafficPolicy') == 'Local'
and spec.get('allocateLoadBalancerNodePorts') is False,
f'{where}: the route is a LoadBalancer with Local traffic policy and no node ports')
check(spec.get('ports') == ROUTE_PORT, f'{where}: the route exposes exactly UDP 7400 (QUIC)')
check(spec.get('selector') == REGION_SELECTOR, f'{where}: the route selects only the region pod')
annotations = doc['metadata'].get('annotations', {})
check(set(annotations) == {'metallb.io/loadBalancerIPs'},
f'{where}: only a pinned MetalLB address annotation (no PROXY protocol or sharing)')
vip = str(annotations.get('metallb.io/loadBalancerIPs', ''))
check(len(ranges) >= 1, f'{where}: loadBalancerSourceRanges must name the permitted CIDRs')
if not vip.startswith('REPLACE'):
try:
address = ipaddress.ip_address(vip)
check(internal(ipaddress.ip_network(f'{address}/{address.max_prefixlen}')),
f'{where}: the VIP must be an internal address (RFC 1918 or approved in ADR-0087)')
except ValueError:
check(False, f'{where}: the VIP must be one IP address')
for cidr in ranges:
if str(cidr).startswith('REPLACE'):
continue
try:
network = ipaddress.ip_network(cidr)
check(internal(network) and network.prefixlen >= (16 if network.version == 4 else 48),
f'{where}: {cidr} must be a private range no wider than /16 (IPv6 /48)')
except ValueError:
check(False, f'{where}: {cidr} is not a CIDR')
return ranges
def validate_route_policy(where, doc, ranges):
"""The route policy admits UDP 7400 to the region pod from exactly the Service ranges."""
spec = doc['spec']
rules = spec.get('ingress', [])
check(spec.get('podSelector') == {'matchLabels': REGION_SELECTOR}
and spec.get('policyTypes') == ['Ingress'] and len(rules) == 1
and rules[0].get('ports') == [{'protocol': 'UDP', 'port': 7400}],
f'{where}: the session policy admits only UDP 7400 to the region pod')
peers = rules[0].get('from', []) if rules else []
check(peers and all(set(peer) == {'ipBlock'} and set(peer['ipBlock']) == {'cidr'} for peer in peers)
and sorted(peer['ipBlock']['cidr'] for peer in peers) == sorted(ranges),
f'{where}: the session policy admits exactly the Service source ranges')
route_ranges = []
for path, doc in objects:
if doc['kind'] == 'Service':
spec = doc['spec']
session_port = any(port.get('port') == 7400 or port.get('targetPort') in (7400, 'session')
for port in spec.get('ports', []))
is_route = path == ROUTE + 'service.yaml' and doc['metadata'].get('name') == 'evo-region-session'
if not is_route:
check(not session_port, f'{path}: only the dev session route Service may expose the session port')
check(spec.get('type', 'ClusterIP') == 'ClusterIP',
f'{path}: {spec.get("type")} Services are forbidden (ADR-0069)')
continue
route_ranges = validate_route_service(path, doc)
if doc['kind'] == 'Ingress':
text = yaml.safe_dump(doc)
check('7400' not in text and 'session' not in text,
f'{path}: the dev session is QUIC over UDP, never an Ingress (ADR-0087)')
policies = [(p, d) for p, d in objects if p.startswith(ROUTE) and d['kind'] == 'NetworkPolicy']
check(len(policies) == 1, f'{ROUTE}: exactly one session ingress policy')
for path, doc in policies:
validate_route_policy(path, doc, route_ranges)
for path, doc in objects:
if path.startswith(ROUTE):
check(doc.get('metadata', {}).get('namespace') in (None, 'evolution-dev'),
f'{path}: the session route belongs to evolution-dev only')
# Enabling the route fails closed until its values are real and the server enforces
# operator-opened windows itself (expiry survives restarts; ADR-0087).
if route_enabled:
for name in os.listdir(os.path.join(ROOT, ROUTE)):
with open(os.path.join(ROOT, ROUTE, name), encoding='utf-8') as handle:
check('REPLACE' not in handle.read(), f'{ROUTE}{name}: unresolved REPLACE marker with the route enabled')
check(ROUTE_ENABLING_PERMITTED,
'the dev session route cannot be enabled until ADR-0087 records the implemented, tested '
'server-enforced session windows')
dev_quota = [d.get('spec', {}).get('hard', {}).get('services.loadbalancers') for p, d in objects
if d['kind'] == 'ResourceQuota' and p.startswith(f'{platform}/dev/')]
check(dev_quota == ['1'], 'the dev quota must allow exactly one LoadBalancer for the route')
for env in ('dev', 'stage'):
overlay = f'deploy/kubernetes/region/{env}/'
images = manifests.get(overlay + 'kustomization.yaml', [{}])[0].get('images', [])
server = [i.get('digest', '') for i in images if i.get('name') == 'aevolution-server']
builds = [v.get('value') for d in manifests.get(overlay + 'region-values.yaml', [])
for c in d['spec']['template']['spec']['containers'] for v in c.get('env', [])
if v.get('name') == 'REGION_BUILD']
check(len(server) == 1 and len(builds) == 2 and all('sha256:' + b == server[0] for b in builds),
f'{overlay}: the build identity must equal the server image digest')
retained = [v.get('value', '') for d in manifests.get(overlay + 'region-values.yaml', [])
for c in d['spec']['template']['spec']['containers'] for v in c.get('env', [])
if v.get('name') == 'REGION_RETAINED']
for value in retained:
entries = value.split()
check(all(__import__('re').fullmatch(r'[0-9a-f]{64}', e) for e in entries)
and not set(entries) & set(builds),
f'{overlay}: REGION_RETAINED must list other builds as 64 hex digits')
# M8.5 monitoring (ADR-0072): third-party images are pinned by digest, and Prometheus
# needs no Kubernetes API access.
for path, doc in objects:
if path.startswith('deploy/kubernetes/monitoring/') and doc['kind'] == 'Deployment':
pod = doc['spec']['template']['spec']
for container in pod['containers']:
check('@sha256:' in container.get('image', ''), f'{path}: {container["name"]} must be pinned by digest')
check(pod.get('automountServiceAccountToken') is False, f'{path}: monitoring pods mount no token')
if path.startswith('deploy/') and doc['kind'] in ('ClusterRole', 'ClusterRoleBinding', 'Role', 'RoleBinding'):
check(False, f'{path}: AEvolution grants no Kubernetes RBAC')
# M8.8 World0 launch gate (ADR-0075). This host has no independent durable failure
# domain, so an acknowledged death could be undone by host loss: World0 must not run
# or acknowledge. Lifting this rule is itself the launch decision and needs its own
# accepted ADR with host-loss RPO-0 evidence.
workload_kinds = {'Deployment', 'StatefulSet', 'DaemonSet', 'Job', 'CronJob', 'Pod', 'ReplicaSet'}
for path, doc in objects:
in_world0 = doc['metadata'].get('namespace') == 'evolution-world0' or '/world0/' in path
check(not (in_world0 and doc['kind'] in workload_kinds),
f'{path}: World0 launch is blocked (ADR-0075); no {doc["kind"]} may target World0')
if doc['kind'] == 'Application' and doc['metadata'].get('name') == 'aevolution-world0':
check('REPLACE' in doc['spec'].get('source', {}).get('path', ''),
f'{path}: the World0 Application must keep an unresolved path while launch is blocked')
for component in ('region', 'stores'):
check(not os.path.exists(os.path.join(ROOT, 'deploy', 'kubernetes', component, 'world0')),
f'deploy/kubernetes/{component}/world0: World0 launch is blocked (ADR-0075)')
# Promotion: World0 may only ever pin a digest that stage already runs.
for directory, _, files in os.walk(os.path.join(ROOT, 'deploy', 'kubernetes')):
rel = os.path.relpath(directory, ROOT).replace(os.sep, '/')
if rel.endswith('/world0') and 'kustomization.yaml' in files:
stage = rel[:-len('world0')] + 'stage/kustomization.yaml'
staged = {(i.get('name'), i.get('digest')) for i in manifests.get(stage, [{}])[0].get('images', [])}
for image in manifests[rel + '/kustomization.yaml'][0].get('images', []):
check((image.get('name'), image.get('digest')) in staged,
f'{rel}: {image.get("name")} must be promoted from {stage}, not pinned independently')
# M8.9 scheduling (ADR-0076). Node-local volumes pin their pods: added CPU does not
# make the objects volume portable. Compute is a preference so one node still works.
for path, doc in objects:
if doc['kind'] not in workload_kinds or path in patch_files:
continue
spec = doc.get('spec', {})
pod = spec.get('jobTemplate', {}).get('spec', {}).get('template', {}).get('spec') \
or spec.get('template', {}).get('spec') or spec
claims = [v.get('persistentVolumeClaim', {}).get('claimName') for v in pod.get('volumes', [])]
if 'aevolution-objects' in claims:
check(pod.get('nodeSelector', {}).get('aevolution.io/stores') == 'true',
f'{path}: a pod mounting aevolution-objects must select aevolution.io/stores')
required = (pod.get('affinity', {}).get('nodeAffinity', {})
.get('requiredDuringSchedulingIgnoredDuringExecution', {}))
check('aevolution.io/compute' not in yaml.safe_dump(required)
and 'aevolution.io/compute' not in pod.get('nodeSelector', {}),
f'{path}: aevolution.io/compute may only be preferred, never required')
for container in pod.get('containers', []):
text = ' '.join(container.get('args', []) + container.get('command', []))
check('--accelerate' not in text and '--synthetic-until' not in text,
f'{path}: upgrade-rehearsal flags never run in a deployed workload')
# The platform bundle: every rendered object is protected from prune and deletion.
bundle = manifests.get(platform + '/cluster/kustomization.yaml', [{}])[0]
check(bundle.get('commonAnnotations', {}).get('argocd.argoproj.io/sync-options') == 'Prune=false,Delete=false',
'the platform bundle must mark every object Prune=false,Delete=false')
check(sorted(bundle.get('resources', [])) == ['../admission', '../dev', '../monitoring', '../stage', '../storage',
'../world0'],
'the platform bundle must include admission, storage, the environments and monitoring')
# Paths a real (non-example) Application syncs must be complete: no REPLACE markers
# anywhere they render, following kustomization resources.
def kustomize_files(directory, seen):
directory = os.path.normpath(directory)
if directory in seen:
return []
seen.add(directory)
found = []
k = os.path.join(directory, 'kustomization.yaml')
if not os.path.isfile(k):
return found
found.append(k)
spec = documents(k)[0]
for entry in spec.get('resources', []) + spec.get('components', []):
target = os.path.normpath(os.path.join(directory, entry))
if os.path.isdir(target):
found += kustomize_files(target, seen)
else:
found.append(target)
return found
for path, doc in objects:
# Examples count too once their path is real (the live dev app is only an example
# here); an example whose path is itself still a marker is not yet deployable.
source_path = doc.get('spec', {}).get('source', {}).get('path', '')
if doc['kind'] == 'Application' and 'REPLACE' not in source_path:
source = os.path.join(ROOT, source_path)
for f in kustomize_files(source, set()):
with open(f, encoding='utf-8') as handle:
check('REPLACE' not in handle.read(),
f'{os.path.relpath(f, ROOT)}: unresolved REPLACE marker in a synced path')
# The live Argo path.
for path, doc in objects:
if path.startswith('deploy/kubernetes/dev/'):
check(doc['kind'] not in {'Namespace', 'Secret', 'ResourceQuota', 'LimitRange', 'Role',
'RoleBinding', 'ClusterRole', 'ClusterRoleBinding'},
f'{path}: {doc["kind"]} must not live in the Argo workload path')
# The cleanup, drill and backup tools refuse World0 before kubectl runs (no kubectl on PATH here).
with tempfile.TemporaryDirectory() as empty:
env = dict(os.environ, PATH=empty + os.pathsep + os.path.dirname(sys.executable))
for tool in ('tools/ops/cleanup-environment.sh', 'tools/ops/region-drill.sh', 'tools/ops/fault-drill.sh',
'tools/backup/cluster-backup.sh', 'tools/ops/scheduling-check.sh',
'tools/ops/upgrade-barrier.sh'):
for bad in ('world0', 'evolution-world0', 'World0', 'prod', ''):
result = subprocess.run([shutil.which('bash') or '/bin/bash', os.path.join(ROOT, tool), bad],
capture_output=True, text=True, env=env)
check(result.returncode == 2 and ('refused' in result.stderr or 'usage' in result.stderr)
and 'kubectl' not in result.stderr,
f'{tool} must refuse {bad!r} before kubectl (rc={result.returncode})')
# Script-generated Pods (review M8 P2): rendered offline and checked like manifests.
retained_hex = next((v.get('value', '').split()[0] for d in manifests.get(
'deploy/kubernetes/region/dev/region-values.yaml', [])
for c in d['spec']['template']['spec']['containers'] for v in c.get('env', [])
if v.get('name') == 'REGION_RETAINED' and v.get('value')), None)
generators = [('tools/backup/cluster-backup.sh', ['dev', '--print-manifest'], 'backup')]
if retained_hex:
generators.append(('tools/ops/upgrade-barrier.sh',
['dev', '--retire', retained_hex, '--print-manifest'], 'migrate'))
else:
check(False, 'dev REGION_RETAINED is empty: the barrier Pod cannot be rendered')
egress = next((d for p, d in objects if d['kind'] == 'NetworkPolicy'
and d['metadata']['name'] == 'aevolution-database-egress'), {})
admitted = [v for e in egress.get('spec', {}).get('podSelector', {}).get('matchExpressions', [])
for v in e.get('values', [])]
for tool, args, component in generators:
rendered = subprocess.run([shutil.which('bash') or '/bin/bash', os.path.join(ROOT, tool)] + args,
capture_output=True, text=True)
check(rendered.returncode == 0, f'{tool} --print-manifest failed: {rendered.stderr.strip()}')
if rendered.returncode != 0:
continue
pod = yaml.safe_load(rendered.stdout)
spec = pod['spec']
context = spec.get('securityContext', {})
check(context.get('runAsNonRoot') is True and context.get('runAsUser') == 10001
and context.get('runAsGroup') == 10001 and context.get('fsGroup') == 10001,
f'{tool}: the generated Pod must run as 10001 with fsGroup 10001 (Secret access)')
check(spec.get('automountServiceAccountToken') is False, f'{tool}: the generated Pod mounts no token')
for container in spec['containers']:
sc = container.get('securityContext', {})
check(sc.get('readOnlyRootFilesystem') is True and sc.get('allowPrivilegeEscalation') is False,
f'{tool}: read-only root and no privilege escalation')
for volume in spec.get('volumes', []):
if 'secret' in volume:
check(volume['secret'].get('defaultMode') == 0o440,
f'{tool}: Secret files stay 0440 (never world-readable)')
labels = pod['metadata'].get('labels', {})
check(labels.get('app.kubernetes.io/component') == component and component in admitted,
f'{tool}: component {component} must be admitted by the database egress policy')
check(pod['metadata'].get('namespace') == 'evolution-dev', f'{tool}: renders into its environment')
# ADR-0088: the effective configuration is what kustomize renders. Every overlay is
# built with the pinned binary and its output checked, independent of how patches,
# components or namespace transformations produced it.
def kustomize_binary():
with open(os.path.join(ROOT, 'tools', 'ci', 'prepare-kustomize.sh'), encoding='utf-8') as f:
version = re.search(r'^version=(v[0-9.]+)$', f.read(), re.M).group(1)
candidates = [os.environ.get('KUSTOMIZE', ''),
os.path.join(ROOT, '.cache', 'tools', f'kustomize-{version}', 'kustomize'),
shutil.which('kustomize') or '']
for candidate in candidates:
if candidate and os.path.isfile(candidate):
reported = subprocess.run([candidate, 'version'], capture_output=True, text=True).stdout.strip()
if reported == version:
return candidate
return None
SUPPORTED_KEYS = {'apiVersion', 'kind', 'namespace', 'resources', 'components', 'patches', 'images',
'labels', 'commonAnnotations', 'commonLabels', 'configMapGenerator', 'generatorOptions'}
for path, ds in manifests.items():
if path.endswith('kustomization.yaml') and ds:
unsupported = set(ds[0]) - SUPPORTED_KEYS
check(not unsupported, f'{path}: unsupported kustomize features {sorted(unsupported)} (ADR-0088)')
EXPECTED_NAMESPACE = {'deploy/kubernetes/dev': 'evolution-dev', 'deploy/kubernetes/stage': 'evolution-stage',
'deploy/kubernetes/monitoring': 'aevolution-monitoring'}
for env in ('dev', 'stage', 'world0'):
for component in ('platform', 'region', 'stores'):
EXPECTED_NAMESPACE[f'deploy/kubernetes/{component}/{env}'] = f'evolution-{env}'
EXPECTED_NAMESPACE['deploy/kubernetes/platform/monitoring'] = 'aevolution-monitoring'
CLUSTER_KINDS = {'Namespace', 'StorageClass', 'ValidatingAdmissionPolicy', 'ValidatingAdmissionPolicyBinding',
'ClusterRole', 'ClusterRoleBinding', 'PersistentVolume'}
binary = kustomize_binary()
check(binary is not None, 'the pinned kustomize is required: run tools/ci/prepare-kustomize.sh (ADR-0088)')
renders = {}
if binary:
for path, ds in sorted(manifests.items()):
if not path.endswith('kustomization.yaml') or not ds or ds[0].get('kind') == 'Component':
continue
overlay = os.path.dirname(path)
result = subprocess.run([binary, 'build', os.path.join(ROOT, overlay)], capture_output=True, text=True)
check(result.returncode == 0, f'{overlay}: kustomize build failed: {result.stderr.strip()[:300]}')
if result.returncode == 0:
renders[overlay] = [d for d in yaml.safe_load_all(result.stdout) if isinstance(d, dict)]
REGION_POD_LABELS = [d['spec']['template']['metadata'].get('labels', {}) for docs in renders.values()
for d in docs if d['kind'] == 'Deployment' and d['metadata'].get('name') == 'evo-region']
REGION_POD_LABELS = REGION_POD_LABELS or [REGION_SELECTOR]
def selects_region(selector):
"""True when a pod selector can select a region pod (empty and expressions count)."""
selector = selector or {}
if not selector or selector.get('matchExpressions'):
return True
wanted = selector.get('matchLabels', {})
return any(all(labels.get(k) == v for k, v in wanted.items()) for labels in REGION_POD_LABELS)
def admits_session(rule):
"""True when an ingress rule admits 7400 on the region pod in any protocol (UDP for the
QUIC session, and TCP or SCTP, which nothing may expose there either): no or empty ports,
an entry without a port, the 'session' name, 7400 itself or a range spanning it."""
ports = rule.get('ports')
if not ports:
return True
for port in ports:
number = port.get('port')
if number is None or number in (7400, 'session'):
return True
if isinstance(number, int) and isinstance(port.get('endPort'), int) and number <= 7400 <= port['endPort']:
return True
return False
for env in ('dev', 'stage', 'world0'):
for overlay in (f'deploy/kubernetes/platform/{env}', 'deploy/kubernetes/platform/cluster'):
if overlay in renders:
check(any(d['kind'] == 'NetworkPolicy' and d['metadata'].get('name') == 'aevolution-default-deny'
and d['metadata'].get('namespace') == f'evolution-{env}' for d in renders[overlay]),
f'{overlay} (rendered): evolution-{env} must keep its default-deny policy')
for overlay, docs in renders.items():
expected = EXPECTED_NAMESPACE.get(overlay)
route_services = [d for d in docs if d['kind'] == 'Service' and d['metadata']['name'] == 'evo-region-session']
route_active = bool(route_services)
for doc in docs:
kind, name = doc['kind'], doc['metadata'].get('name', '')
namespace = doc['metadata'].get('namespace')
where = f'{overlay} (rendered) {kind}/{name}'
if expected and kind not in CLUSTER_KINDS:
check(namespace == expected, f'{where}: renders into {namespace}, expected {expected}')
check(kind != 'Secret', f'{where}: rendered Secrets never come from Git')
if namespace == 'evolution-dev' and kind in workload_kinds:
spec = doc.get('spec', {})
pod = (spec.get('jobTemplate', {}).get('spec', {}).get('template', {}).get('spec')
or spec.get('template', {}).get('spec') or spec)
terms = (pod.get('affinity', {}).get('nodeAffinity', {})
.get('requiredDuringSchedulingIgnoredDuringExecution', {})
.get('nodeSelectorTerms', []))
control_roles = {'node-role.kubernetes.io/control-plane', 'node-role.kubernetes.io/master'}
# Selector terms are OR-ed: every possible term must exclude both roles.
excludes_control = bool(terms) and all(
control_roles <= {e.get('key') for e in term.get('matchExpressions', [])
if e.get('operator') == 'DoesNotExist' and not e.get('values')}
for term in terms)
check(excludes_control and not pod.get('nodeName'),
f'{where}: DEV workloads must exclude control-plane and master nodes in every required term '
'and must not bypass scheduling with nodeName')
if kind == 'Deployment' and name == 'evo-region':
pod = doc['spec']['template']['spec']
for container in pod.get('containers', []) + pod.get('initContainers', []):
env = {v.get('name'): v.get('value') for v in container.get('env', [])}
text = ' '.join(container.get('command', []) + container.get('args', []))
if container.get('name') == 'evo-server':
expected_session = 'dev' if route_active else ''
check(env.get('REGION_SESSION') == expected_session and env.get('REGION_INPUTS') == '',
f'{where}: sessions must be dev only with the route, and synthetic inputs stay off in Git')
listen = '0.0.0.0:7400' if route_active else '127.0.0.1:7400'
check(env.get('REGION_SESSION_LISTEN') == listen,
f'{where}: REGION_SESSION_LISTEN must be {listen} (route {"on" if route_active else "off"})')
check(text.count('--m10-listen') == 1 and '--m10-listen "$REGION_SESSION_LISTEN"' in text
and '--session-window-source db' in text and '--session-window ' not in text,
f'{where}: one M10 listener whose session window is read from SQL (ADR-0087)')
# The window lives in SQL; a Git-committed one would never be read.
check('REGION_SESSION_WINDOW' not in env,
f'{where}: the session window lives in SQL, not in REGION_SESSION_WINDOW')
else:
check('--m10-listen' not in text and '--synthetic-inputs' not in text,
f'{where}: only evo-server may listen for sessions')
ports = sorted((p.get('name'), p.get('containerPort'), p.get('protocol', 'TCP'))
for p in container.get('ports', []))
allowed = ([('ops', 9100, 'TCP'), ('session', 7400, 'UDP')] if route_active
else [('ops', 9100, 'TCP')])
if container.get('name') == 'evo-server':
check(ports == sorted(allowed), f'{where}: container ports must be {allowed}')
if kind == 'Service':
spec = doc['spec']
session = any(p.get('port') == 7400 or p.get('targetPort') in (7400, 'session')
for p in spec.get('ports', []))
if name == 'evo-region-session':
check(namespace == 'evolution-dev', f'{where}: the route belongs to evolution-dev only')
validate_route_service(where, doc)
else:
check(spec.get('type', 'ClusterIP') == 'ClusterIP' and not session,
f'{where}: only the route may be a LoadBalancer or expose the session port')
if kind == 'Ingress':
check('7400' not in yaml.safe_dump(doc) and 'session' not in yaml.safe_dump(doc),
f'{where}: the dev session is never an Ingress')
if kind == 'NetworkPolicy':
spec = doc.get('spec', {})
if name == 'aevolution-default-deny':
check(spec.get('podSelector') == {} and sorted(spec.get('policyTypes', [])) == ['Egress', 'Ingress']
and not spec.get('ingress') and not spec.get('egress'),
f'{where}: default-deny must select every pod and allow nothing')
ranges = route_services[0]['spec'].get('loadBalancerSourceRanges', []) if route_services else []
if name == 'evo-region-session-ingress':
check(route_active, f'{where}: the session policy renders only with the route')
validate_route_policy(where, doc, ranges)
elif selects_region(spec.get('podSelector')):
for rule in spec.get('ingress', []):
check(not admits_session(rule),
f'{where}: an ingress rule admits the region session port '
'(no or empty ports, the session name, 7400 or a range spanning it)')
if kind == 'ResourceQuota':
hard = doc['spec'].get('hard', {})
check(hard.get('services.nodeports') == '0', f'{where}: node ports stay forbidden')
limit = ('0', '1') if namespace == 'evolution-dev' and route_enabled else ('0',)
check(hard.get('services.loadbalancers') in limit,
f'{where}: LoadBalancers only in evolution-dev with the enabled route')
if route_active:
check(overlay in ('deploy/kubernetes/region/dev', 'deploy/kubernetes/dev'),
f'{overlay}: renders the dev session route outside evolution-dev')
check(ROUTE_ENABLING_PERMITTED,
f'{overlay}: renders the dev session route, which cannot be enabled until ADR-0087 records '
'the implemented, tested server-enforced session windows')
check('REPLACE' not in yaml.safe_dump_all(docs), f'{overlay}: unresolved REPLACE marker in the rendered route')
for message in failures:
print('FAIL', message)
print(f'deploy policy: {len(objects)} objects in {len(manifests)} files, '
f'{sum(len(d) for d in renders.values())} rendered objects in {len(renders)} overlays, {len(failures)} failures')
sys.exit(1 if failures else 0)