Files
aevolution/docs/adr/0069-environment-isolation.md
T
2026-09-29 13:03:45 -04:00

5.5 KiB

ADR-0069: Environment isolation and World0 protection

Status

Accepted for M8.2, 2026-09-29. It builds on the M8.1 inventory (infrastructure/INVENTORY.md), ADR-0020 (delivery), and OPERATIONS.md (World0 protection). Nothing here deploys a world; World0 launch stays gated by M8.8.

Context

The inventory found five things that matter here:

  • Argo CD's default AppProject allows every cluster resource in every namespace. Three applications use it, two with automated prune and self-heal.
  • The only StorageClass reclaims with Delete.
  • Collection ran as system:admin.
  • There are no quotas, limit ranges or priority classes.
  • Only the AEvolution application is manual-sync.

The roadmap's named risk is that one permissive application or credential could destroy every world. Argo project boundaries cannot address that alone, because a default-project application, a cascading parent application or a cluster-admin is not constrained by an AEvolution AppProject.

Decision

Three environments, each in its own namespace: evolution-dev, evolution-stage and evolution-world0. Each has its own AppProject, service accounts, Secrets, database, database logins and object PVC. No identity, Secret or database is shared between them.

Platform layer (deploy/kubernetes/platform/<env>), applied by a cluster administrator and never by an Argo CD Application, so no sync or prune can remove it. It contains:

  • The Namespace, labelled with its environment and with Pod Security restricted enforced. Only World0 carries aevolution.io/protected=true.
  • A ResourceQuota:
    • dev and stage: 2 CPU and 2 GiB requests, 4 GiB of memory limits, 4 PVCs and 40 GiB;
    • World0: 4 CPU and 4 GiB requests, 8 GiB of limits, 8 PVCs and 200 GiB;
    • no LoadBalancer or NodePort services anywhere.
  • A LimitRange (defaults 50m/64Mi with a 256Mi limit, at most 2Gi per container) sized from the measured host profile.
  • A default-deny NetworkPolicy for every pod, plus DNS egress only. Workloads add narrow rules in M8.4.
  • Service accounts aevolution-migrator, aevolution-region and aevolution-backup, with no token and no RBAC; nothing needs the Kubernetes API. They are platform-owned, so workload projects cannot create them.

Secrets never enter Git. Operators create them per environment, by reference name (ENVIRONMENTS.md). CI holds no cluster or data credentials (ADR-0020).

World0 admission protection. A native ValidatingAdmissionPolicy (deploy/kubernetes/platform/admission) is GA on the cluster's Kubernetes 1.32. It needs no webhook or dependency, and it fails closed. It applies to everyone, including system:masters and Argo's controller. Unless the requester is in the group aevolution:world0-break-glass, it denies:

  • deleting the World0 namespace, or removing its protected label;
  • deleting its PVCs, Secrets or service accounts;
  • deleting a PV bound to it, or moving such a PV away from Retain;
  • deleting the aevolution-world0 Application or AppProject.

A cascading parent app, a prune, a default-project application or a stray kubectl delete therefore fails at admission. Kubernetes does not apply policies to policy objects, so removing the policy is itself the break-glass action, reserved to cluster-admins.

Delivery.

  • The stage and World0 AppProjects (examples) each target one namespace. They allow no cluster kinds, and whitelist only namespaced workload kinds.
  • World0 carries a permanent deny sync window with manualSync: true, so no automated sync can ever run.
  • Applications never auto-sync and never carry the cascade finalizer. World0's must not appear in a pruning parent.
  • Durable PVCs will carry Prune=false,Delete=false.

Cleanup. tools/ops/cleanup-environment.sh accepts only dev or stage. It refuses anything else before invoking kubectl, re-checks the namespace labels, never touches platform objects or Secrets, and dry-runs unless --execute is given.

Verification.

  • tools/ci/deploy-policy-check.py runs offline in CI (Ubuntu python3-yaml) and asserts every invariant above over all manifests.
  • tools/ops/verify-isolation.sh runs live. It makes only can-i queries, reads and server dry-runs, and covers negative RBAC, cross-environment denial, World0 admission denials, project scopes and the cleanup refusal.

Consequences

  • Residual risk: creation. A default-project application can still create objects in AEvolution namespaces; admission protects only World0's data and identity from deletion. The maintainer accepted this risk (2026-09-29) and is leaving default unrestricted (docs/BACKLOG.md).
  • Residual risk: advisory locks. The runtime login's advisory-lock capability (ADR-0068) remains.
  • Local-path storage. Its PVs are Delete-reclaim. A World0 PV must be patched to Retain when bound, and the policy then keeps it there. Volumes remain node-local until M8.3.
  • Quotas and defaults. Quotas are starting values from measured profiles. They are raised by editing the platform overlay, never by the workload.
  • Existing dev namespace. Applying the dev overlay to the existing namespace adds labels, the quota and the default-deny policy. Its proof Job already complies.

Alternatives rejected

  • An admission webhook or policy engine (Kyverno, Gatekeeper). Either is a new dependency with its own availability risk; the native policy suffices.
  • Relying on AppProjects alone. They do not constrain other projects, parents or cluster-admins.
  • Namespaces created by Argo CreateNamespace. A prune or cascade could then remove them.