Files
aevolution/docs/adr/0069-environment-isolation.md
T
dingoandClaude Opus 5.5 9f622cf6b3 Add M9.8f dev session route design, disabled with fail-closed policy
ADR-0087 records the maintainer's route (void.aeternalisgames.com:7400 via a
dedicated internal MetalLB VIP to the evolution-dev region pod, TLS end to end,
CIDR-limited) and operator-opened windows enforced by evo-server. The route is a
kustomize component included nowhere; the policy check encodes its shape and
refuses enabling until VIP/CIDR, the dev quota and --session-window exist, with a
20-case self-test in the CI deploy stage. The base listener stays loopback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-02 17:34:12 -04:00

7.6 KiB

ADR-0069: Environment isolation and World0 protection

Status

ADR-0087 allows exactly one LoadBalancer Service, in evolution-dev only, for the dev session route once enabled; node ports stay forbidden and stage/World0 are unchanged. Accepted for M8.2, 2026-09-29. It builds on the M8.1 inventory (infrastructure/INVENTORY.md), ADR-0020 (delivery), and OPERATIONS.md (World0 protection). Nothing here deploys a world; World0 launch stays gated by M8.8.

Context

The inventory found five things that matter here:

  • Argo CD's default AppProject allows every cluster resource in every namespace. Three applications use it, two with automated prune and self-heal.
  • The only StorageClass reclaims with Delete.
  • Collection ran as system:admin.
  • There are no quotas, limit ranges or priority classes.
  • Only the AEvolution application is manual-sync.

The roadmap's named risk is that one permissive application or credential could destroy every world. Argo project boundaries cannot address that alone, because a default-project application, a cascading parent application or a cluster-admin is not constrained by an AEvolution AppProject.

Decision

Three environments, each in its own namespace: evolution-dev, evolution-stage and evolution-world0. Each has its own AppProject, service accounts, Secrets, database, database logins and object PVC. No identity, Secret or database is shared between them.

Platform layer (deploy/kubernetes/platform/<env>), applied by a cluster administrator and never by an Argo CD Application, so no sync or prune can remove it. It contains:

  • The Namespace, labelled with its environment and with Pod Security restricted enforced. Only World0 carries aevolution.io/protected=true.
  • A ResourceQuota:
    • dev and stage: 2 CPU and 2 GiB requests, 4 GiB of memory limits, 4 PVCs and 40 GiB;
    • World0: 4 CPU and 4 GiB requests, 8 GiB of limits, 8 PVCs and 200 GiB;
    • no LoadBalancer or NodePort services anywhere.
  • A LimitRange (defaults 50m/64Mi with a 256Mi limit, at most 2Gi per container) sized from the measured host profile.
  • A default-deny NetworkPolicy for every pod, plus DNS egress only. Workloads add narrow rules in M8.4.
  • Service accounts aevolution-migrator, aevolution-region and aevolution-backup, with no token and no RBAC; nothing needs the Kubernetes API. They are platform-owned, so workload projects cannot create them.

Secrets never enter Git. Operators create them per environment, by reference name (ENVIRONMENTS.md). CI holds no cluster or data credentials (ADR-0020).

World0 admission protection. A native ValidatingAdmissionPolicy (deploy/kubernetes/platform/admission) is GA on the cluster's Kubernetes 1.32. It needs no webhook or dependency, and it fails closed. It applies to everyone, including system:masters and Argo's controller. Unless the requester is in the group aevolution:world0-break-glass, it denies:

  • deleting the World0 namespace, or removing its protected label;
  • deleting its PVCs, Secrets or service accounts;
  • deleting a PV bound to it, or moving such a PV away from Retain;
  • deleting the aevolution-world0 Application or AppProject.

A cascading parent app, a prune, a default-project application or a stray kubectl delete therefore fails at admission. Kubernetes does not apply policies to policy objects, so removing the policy is itself the break-glass action, reserved to cluster-admins.

Delivery.

  • The stage and World0 AppProjects (examples) each target one namespace. They allow no cluster kinds, and whitelist only namespaced workload kinds.
  • World0 carries a permanent deny sync window with manualSync: true, so no automated sync can ever run.
  • Applications never auto-sync and never carry the cascade finalizer. World0's must not appear in a pruning parent.
  • Durable PVCs will carry Prune=false,Delete=false.

Cleanup. tools/ops/cleanup-environment.sh accepts only dev or stage. It refuses anything else before invoking kubectl, re-checks the namespace labels, never touches platform objects or Secrets, and dry-runs unless --execute is given.

Verification.

  • tools/ci/deploy-policy-check.py runs offline in CI (Ubuntu python3-yaml) and asserts every invariant above over all manifests.
  • tools/ops/verify-isolation.sh runs live. It makes only can-i queries, reads and server dry-runs, and covers negative RBAC, cross-environment denial, World0 admission denials, project scopes and the cleanup refusal.

Consequences

  • Residual risk: creation. A default-project application can still create objects in AEvolution namespaces; admission protects only World0's data and identity from deletion. The maintainer accepted this risk (2026-09-29) and is leaving default unrestricted (docs/BACKLOG.md).
  • Residual risk: advisory locks. The runtime login's advisory-lock capability (ADR-0068) remains.
  • Local-path storage. Its PVs are Delete-reclaim. A World0 PV must be patched to Retain when bound, and the policy then keeps it there. Volumes remain node-local until M8.3.
  • Quotas and defaults. Quotas are starting values from measured profiles. They are raised by editing the platform overlay, never by the workload.
  • Existing dev namespace. Applying the dev overlay to the existing namespace adds labels, the quota and the default-deny policy. Its proof Job already complies.

Alternatives rejected

  • An admission webhook or policy engine (Kyverno, Gatekeeper). Either is a new dependency with its own availability risk; the native policy suffices.
  • Relying on AppProjects alone. They do not constrain other projects, parents or cluster-admins.
  • Namespaces created by Argo CreateNamespace. A prune or cascade could then remove them.

Amendment: platform delivered by Argo CD (2026-09-29)

The maintainer prefers that every deployment and configuration goes through Argo CD. The platform layer is therefore an Argo Application, aevolution-platform (deploy/argocd/platform), no longer a hand-applied layer. The original concern, that a sync, prune or cascade could delete namespaces and durable objects, is met differently:

  • Never prune. The Application self-heals but never prunes.
  • Protected objects. Every object in the bundle carries argocd.argoproj.io/sync-options: Prune=false,Delete=false, so neither a Git removal nor deleting the Application deletes it.
  • No cascade. There is no cascade finalizer.
  • Protected by admission. The World0 admission policy also denies deleting the aevolution-platform Application and AppProject.
  • Scoped project. Its project may target only the three AEvolution namespaces, and only four cluster kinds: Namespace, StorageClass, ValidatingAdmissionPolicy and its binding.

Self-heal adds a protection the hand-applied layer lacked: drift, such as a deleted quota or label, is restored. The admission policy itself is still not subject to admission, so a policy change is a reviewed Git change.

What stays outside Argo:

  • Secrets, until a secret manager is chosen (docs/BACKLOG.md);
  • the store node label;
  • the one-time bootstrap of the platform project and Application.

Amendment: World0 volume claim references (M8 review, 2026-09-30)

The policy recognizes a World0 PersistentVolume by its claim reference. Clearing or reassigning that reference first would have removed the protection. An UPDATE that removes the reference, or changes its namespace or claim name, is now denied without the break-glass group. The binder may still fill in uid and resourceVersion. verify-isolation.sh tests this with a server-side dry run when a World0 volume exists.