ADR-0087 records the maintainer's route (void.aeternalisgames.com:7400 via a dedicated internal MetalLB VIP to the evolution-dev region pod, TLS end to end, CIDR-limited) and operator-opened windows enforced by evo-server. The route is a kustomize component included nowhere; the policy check encodes its shape and refuses enabling until VIP/CIDR, the dev quota and --session-window exist, with a 20-case self-test in the CI deploy stage. The base listener stays loopback. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
143 lines
7.6 KiB
Markdown
143 lines
7.6 KiB
Markdown
# ADR-0069: Environment isolation and World0 protection
|
|
|
|
## Status
|
|
ADR-0087 allows exactly one LoadBalancer Service, in evolution-dev only, for the dev
|
|
session route once enabled; node ports stay forbidden and stage/World0 are unchanged.
|
|
Accepted for M8.2, 2026-09-29. It builds on the M8.1 inventory
|
|
(infrastructure/INVENTORY.md), ADR-0020 (delivery), and OPERATIONS.md (World0
|
|
protection). Nothing here deploys a world; World0 launch stays gated by M8.8.
|
|
|
|
## Context
|
|
The inventory found five things that matter here:
|
|
- Argo CD's `default` AppProject allows every cluster resource in every namespace.
|
|
Three applications use it, two with automated prune and self-heal.
|
|
- The only StorageClass reclaims with Delete.
|
|
- Collection ran as `system:admin`.
|
|
- There are no quotas, limit ranges or priority classes.
|
|
- Only the AEvolution application is manual-sync.
|
|
|
|
The roadmap's named risk is that one permissive application or credential could
|
|
destroy every world. Argo project boundaries cannot address that alone, because a
|
|
`default`-project application, a cascading parent application or a cluster-admin
|
|
is not constrained by an AEvolution AppProject.
|
|
|
|
## Decision
|
|
**Three environments,** each in its own namespace: `evolution-dev`,
|
|
`evolution-stage` and `evolution-world0`. Each has its own AppProject, service
|
|
accounts, Secrets, database, database logins and object PVC. No identity, Secret or
|
|
database is shared between them.
|
|
|
|
**Platform layer** (`deploy/kubernetes/platform/<env>`), applied by a cluster
|
|
administrator and never by an Argo CD Application, so no sync or prune can remove
|
|
it. It contains:
|
|
- The Namespace, labelled with its environment and with Pod Security `restricted`
|
|
enforced. Only World0 carries `aevolution.io/protected=true`.
|
|
- A ResourceQuota:
|
|
- dev and stage: 2 CPU and 2 GiB requests, 4 GiB of memory limits, 4 PVCs and
|
|
40 GiB;
|
|
- World0: 4 CPU and 4 GiB requests, 8 GiB of limits, 8 PVCs and 200 GiB;
|
|
- no LoadBalancer or NodePort services anywhere.
|
|
- A LimitRange (defaults 50m/64Mi with a 256Mi limit, at most 2Gi per container)
|
|
sized from the measured host profile.
|
|
- A default-deny NetworkPolicy for every pod, plus DNS egress only. Workloads add
|
|
narrow rules in M8.4.
|
|
- Service accounts `aevolution-migrator`, `aevolution-region` and
|
|
`aevolution-backup`, with no token and no RBAC; nothing needs the Kubernetes API.
|
|
They are platform-owned, so workload projects cannot create them.
|
|
|
|
**Secrets never enter Git.** Operators create them per environment, by reference
|
|
name (ENVIRONMENTS.md). CI holds no cluster or data credentials (ADR-0020).
|
|
|
|
**World0 admission protection.** A native ValidatingAdmissionPolicy
|
|
(`deploy/kubernetes/platform/admission`) is GA on the cluster's Kubernetes 1.32. It
|
|
needs no webhook or dependency, and it fails closed. It applies to everyone,
|
|
including system:masters and Argo's controller. Unless the requester is in the
|
|
group `aevolution:world0-break-glass`, it denies:
|
|
- deleting the World0 namespace, or removing its protected label;
|
|
- deleting its PVCs, Secrets or service accounts;
|
|
- deleting a PV bound to it, or moving such a PV away from `Retain`;
|
|
- deleting the `aevolution-world0` Application or AppProject.
|
|
|
|
A cascading parent app, a prune, a `default`-project application or a stray
|
|
`kubectl delete` therefore fails at admission. Kubernetes does not apply policies
|
|
to policy objects, so removing the policy is itself the break-glass action,
|
|
reserved to cluster-admins.
|
|
|
|
**Delivery.**
|
|
- The stage and World0 AppProjects (examples) each target one namespace. They
|
|
allow no cluster kinds, and whitelist only namespaced workload kinds.
|
|
- World0 carries a permanent deny sync window with `manualSync: true`, so no
|
|
automated sync can ever run.
|
|
- Applications never auto-sync and never carry the cascade finalizer. World0's must
|
|
not appear in a pruning parent.
|
|
- Durable PVCs will carry `Prune=false,Delete=false`.
|
|
|
|
**Cleanup.** `tools/ops/cleanup-environment.sh` accepts only `dev` or `stage`. It
|
|
refuses anything else before invoking kubectl, re-checks the namespace labels,
|
|
never touches platform objects or Secrets, and dry-runs unless `--execute` is given.
|
|
|
|
**Verification.**
|
|
- `tools/ci/deploy-policy-check.py` runs offline in CI (Ubuntu `python3-yaml`) and
|
|
asserts every invariant above over all manifests.
|
|
- `tools/ops/verify-isolation.sh` runs live. It makes only `can-i` queries, reads
|
|
and server dry-runs, and covers negative RBAC, cross-environment denial, World0
|
|
admission denials, project scopes and the cleanup refusal.
|
|
|
|
## Consequences
|
|
- **Residual risk: creation.** A `default`-project application can still *create*
|
|
objects in AEvolution namespaces; admission protects only World0's data and
|
|
identity from deletion. The maintainer accepted this risk (2026-09-29) and is
|
|
leaving `default` unrestricted (docs/BACKLOG.md).
|
|
- **Residual risk: advisory locks.** The runtime login's advisory-lock capability
|
|
(ADR-0068) remains.
|
|
- **Local-path storage.** Its PVs are Delete-reclaim. A World0 PV must be patched
|
|
to Retain when bound, and the policy then keeps it there. Volumes remain
|
|
node-local until M8.3.
|
|
- **Quotas and defaults.** Quotas are starting values from measured profiles. They
|
|
are raised by editing the platform overlay, never by the workload.
|
|
- **Existing dev namespace.** Applying the dev overlay to the existing namespace
|
|
adds labels, the quota and the default-deny policy. Its proof Job already
|
|
complies.
|
|
|
|
## Alternatives rejected
|
|
- **An admission webhook or policy engine (Kyverno, Gatekeeper).** Either is a new
|
|
dependency with its own availability risk; the native policy suffices.
|
|
- **Relying on AppProjects alone.** They do not constrain other projects, parents
|
|
or cluster-admins.
|
|
- **Namespaces created by Argo `CreateNamespace`.** A prune or cascade could then
|
|
remove them.
|
|
|
|
## Amendment: platform delivered by Argo CD (2026-09-29)
|
|
The maintainer prefers that every deployment and configuration goes through Argo
|
|
CD. The platform layer is therefore an Argo Application, `aevolution-platform`
|
|
(`deploy/argocd/platform`), no longer a hand-applied layer. The original concern,
|
|
that a sync, prune or cascade could delete namespaces and durable objects, is met
|
|
differently:
|
|
- **Never prune.** The Application self-heals but never prunes.
|
|
- **Protected objects.** Every object in the bundle carries
|
|
`argocd.argoproj.io/sync-options: Prune=false,Delete=false`, so neither a Git
|
|
removal nor deleting the Application deletes it.
|
|
- **No cascade.** There is no cascade finalizer.
|
|
- **Protected by admission.** The World0 admission policy also denies deleting the
|
|
`aevolution-platform` Application and AppProject.
|
|
- **Scoped project.** Its project may target only the three AEvolution namespaces,
|
|
and only four cluster kinds: Namespace, StorageClass, ValidatingAdmissionPolicy
|
|
and its binding.
|
|
|
|
Self-heal adds a protection the hand-applied layer lacked: drift, such as a
|
|
deleted quota or label, is restored. The admission policy itself is still not
|
|
subject to admission, so a policy change is a reviewed Git change.
|
|
|
|
What stays outside Argo:
|
|
- Secrets, until a secret manager is chosen (docs/BACKLOG.md);
|
|
- the store node label;
|
|
- the one-time bootstrap of the platform project and Application.
|
|
|
|
## Amendment: World0 volume claim references (M8 review, 2026-09-30)
|
|
The policy recognizes a World0 PersistentVolume by its claim reference. Clearing
|
|
or reassigning that reference first would have removed the protection. An UPDATE
|
|
that removes the reference, or changes its namespace or claim name, is now denied
|
|
without the break-glass group. The binder may still fill in uid and
|
|
resourceVersion. `verify-isolation.sh` tests this with a server-side dry run when
|
|
a World0 volume exists.
|