Start M8.1 inventory: stated facts, workload profile and collectors

Record the maintainer-stated host facts (R730xd ~128 GB, 32 GB workers,
automatic Harbor publish and Argo CD deploy, existing aevolution-db
Secret and its conflicts with the pinned version and role split), the
measured evo-server profile (peak RSS ~9.7 MiB, ~0.3% of a core) and a
local storage baseline. Add a read-only cluster collector that never
fetches Secret data and a by-hand storage latency probe outside the
Argo path. Cluster facts remain uninspected pending read-only access.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
dingo
2026-09-29 11:57:08 -04:00
co-authored by Claude Opus 5.5
parent 168c90d5b2
commit 680e7e5725
8 changed files with 376 additions and 2 deletions
+6 -1
View File
@@ -72,7 +72,12 @@ of the live application's configuration:
| Path | deploy/kubernetes/dev |
| Destination namespace | evolution-dev |
| Project | aevolution-dev, or an equivalently restricted project |
| Sync | Manual full sync; no automated prune/self-heal |
| Sync | Manual full sync; no automated prune/self-heal (recommended) |
The maintainer reports (2026-09-29) that Argo CD currently deploys each Git change
automatically. Treat every file under a watched path as live on push, and confirm
the actual sync and prune policy with `tools/inventory/collect-cluster.sh`
(docs/infrastructure/INVENTORY.md).
Bootstrap examples in `deploy/argocd/*.yaml.example` are outside the workload
path. Replace the Argo namespace and registered cluster API URL, or use its UI.
@@ -0,0 +1,110 @@
# M8.1 storage latency probe. NOT part of any Argo path: apply by hand, once per
# candidate StorageClass, after replacing <STORAGE_CLASS> (left unresolved on
# purpose so this file cannot be applied unchanged). It measures the durability
# pattern the region host depends on, on a fresh PVC of that class:
# - 4 KiB synchronous writes (O_DSYNC), the PostgreSQL WAL-like cost;
# - write + fsync + rename + directory fsync of a 40 KiB object (the object
# store's publication pattern, ADR-0061);
# - 256 MiB sequential write with fdatasync (throughput and capacity sanity).
# Results are printed to the Job log. Afterwards delete the Job and the PVC
# (kubectl delete job,pvc,networkpolicy -l app.kubernetes.io/name=evo-disk-probe
# -n evolution-dev) and record the numbers in docs/infrastructure/INVENTORY.md.
# The image is the already-pinned evo-sim runtime (Ubuntu 24.04 coreutils); no
# new image is introduced.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: evo-disk-probe
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
accessModes: [ReadWriteOnce]
storageClassName: <STORAGE_CLASS>
resources:
requests:
storage: 2Gi
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: evo-disk-probe
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: evo-disk-probe
policyTypes: [Ingress, Egress] # No traffic in or out.
---
apiVersion: batch/v1
kind: Job
metadata:
name: evo-disk-probe
namespace: evolution-dev
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
backoffLimit: 0
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: evo-disk-probe
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: probe
# Same repository and digest as deploy/kubernetes/dev (verified run 1399).
image: registry.aeternalisgames.com/aeternalis/aevolution@sha256:0c6c6e1ac28de2ce0648fd0992ca694963e303ef5c9414fe5ac668937136800d
command: ["/bin/bash", "-c"]
args:
- |
set -eu
cd /data
ns() { date +%s%N; }
echo "filesystem: $(df -hT /data | tail -1)"
t0=$(ns)
dd if=/dev/zero of=dsync.bin bs=4k count=1000 oflag=dsync status=none
t1=$(ns)
echo "dsync_4k_writes=1000 mean_us=$(( (t1 - t0) / 1000 / 1000 ))"
head -c 40960 /dev/urandom > payload.src
mkdir -p objects
worst=0; t0=$(ns)
for i in $(seq 1 200); do
s=$(ns)
dd if=payload.src of=objects/.tmp status=none conv=fsync
mv objects/.tmp objects/o$i
sync objects
e=$(( ($(ns) - s) / 1000 )); [ "$e" -gt "$worst" ] && worst=$e
done
t1=$(ns)
echo "object_publish_40k=200 mean_us=$(( (t1 - t0) / 200 / 1000 )) max_us=$worst"
t0=$(ns)
dd if=/dev/zero of=seq.bin bs=1M count=256 conv=fdatasync status=none
t1=$(ns)
echo "sequential_256m_ms=$(( (t1 - t0) / 1000000 ))"
rm -rf dsync.bin payload.src objects seq.bin
resources:
requests: {cpu: 100m, memory: 64Mi}
limits: {cpu: 500m, memory: 128Mi}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: [ALL]
volumeMounts:
- {name: data, mountPath: /data}
volumes:
- name: data
persistentVolumeClaim:
claimName: evo-disk-probe
+16
View File
@@ -959,3 +959,19 @@ Review fixes (2026-09-29): schema v9 serializes build retirement with acquisitio
leases and provisioning under one lock order. It also serializes clone-world
classification per world. The operations listener's accept work per loop pass is
now bounded. Regression tests and five mutations cover each fix.
## M8.1 inventory (in progress, 2026-09-29)
docs/infrastructure/INVENTORY.md records:
- the maintainer's facts: an R730xd with about 128 GB, 32 GB worker nodes,
automatic publish to Harbor and automatic Argo CD deployment, and the existing
`aevolution-db` Secret. The Secret's single login conflicts with the required
18.6 pin and with the separate migrator/runtime roles;
- the measured workload profile: `evo-server` peaks at about 9.7 MiB and 0.3 % of
one core;
- a local storage baseline, and a proposed budget model.
Still needed:
- run the read-only `tools/inventory/collect-cluster.sh` with cluster access;
- hypervisor, VM and disk details from the maintainer;
- the storage probe on each candidate StorageClass, after approval.
+20
View File
@@ -2580,3 +2580,23 @@ Evidence:
- an accept budget of 1000.
- Qualification was unchanged: SIGKILL to ready took 19.55–19.60 s.
- Windows was not rerun. All changes are Linux PostgreSQL, server or doc files.
## M8.1a inventory preparation - 2026-09-29
- No cluster access was available: there is no kubeconfig on the workstation.
Nothing in the cluster was read or changed, and the `aevolution-db` Secret was
not accessed. Only the key names the maintainer supplied are recorded.
- `tools/inventory/collect-cluster.sh` passed `bash -n`. It has not been run
against a cluster. It only reads, and it lists Secrets through the server-side
table, so no data is fetched.
- `deploy/kubernetes/inventory/disk-probe.yaml.example` parses as a PVC,
NetworkPolicy and Job. Its script was run locally in WSL2 on ext4 as a
baseline:
- 4 KiB O_DSYNC writes: mean 1169 µs;
- 40 KiB object publication: mean 4646 µs, max 7983 µs;
- 256 MiB fdatasync: 365 ms.
It has not been applied in any cluster.
- The qualification now records the server's resource profile from /proc. Over
three owners in Release: peak RSS 9,940 KiB, and 60 ms CPU over 16.3 s of
serving. The full PostgreSQL Release suite passed with it. The analysis build
compiled clean, and the format check passed.
+113
View File
@@ -0,0 +1,113 @@
# Host and cluster inventory (M8.1)
Status: **partial, 2026-09-29.** This file separates what has been measured, what
the maintainer has stated, and what has not been inspected yet. Nothing in the
"Not inspected" table may be assumed. Budgets below are proposals until the
cluster facts and the storage probe replace their inputs.
## Maintainer-stated facts
| Fact | Source | Implication |
|---|---|---|
| One Dell R730xd, about 128 GB RAM | Founding brief | The whole capacity envelope, shared by the OS/hypervisor, every node, stores, observability and recovery headroom. Never simulation-only memory. |
| Each Kubernetes worker node has 32 GB RAM | Maintainer, 2026-09-29 | Workers are presumably VMs on the one host. The worker count, control-plane placement and whether memory is overcommitted are unknown. Nodes share one physical failure domain. |
| Every Git push builds and publishes to Harbor | Maintainer, 2026-09-29; Gitea runs 1399/1402 | Publication is automatic. It is not promotion: nothing reaches a cluster except through Git. |
| Argo CD pulls Git and images and deploys to Kubernetes on each change | Maintainer, 2026-09-29 | Deployment is **automatic**. `deploy/README.md` still recommends a manual full sync with no automated prune, so actual auto-sync and prune settings must be read from the Application (see `argocd-apps.txt`). Anything added to a watched path is live on push. That is why every M7/M8 example stays `.yaml.example` outside `deploy/kubernetes/dev`. The dev Job image is pinned by digest in Git, so a push alone does not change the running image. |
| Secret `aevolution-db` exists in `evolution-dev` | Maintainer review note, 2026-09-29 | See below. Its values were not read. |
**The existing database secret.** It has these keys: `PGDATABASE`, `PGHOST`,
`PGHOSTADDR`, `PGPASSWORD`, `PGPORT`, `PGSSLMODE`, `PGSSLROOTCERT`, `PGUSER` and
`POSTGRES_*`. That points to a single database login provisioned outside this
repository. Before M8.3 can use it, confirm:
- **Server version.** The adapter and migrations require exactly 18.6 (ADR-0059).
Any other version is refused, not tolerated.
- **Role split.** ADR-0060 requires a migrator/owner login plus a separate
NOLOGIN runtime group with its own member login. The migration audit refuses a
privileged or owner-identity runtime role. One login for both would fail that
audit by design.
- **Password delivery.** `PGPASSWORD` in the environment is not how the adapter
is used; it reads a mounted password file (`--pg-passfile`, POSTGRESQL.md).
Mount the Secret as files.
- **TLS.** `PGSSLMODE` and `PGSSLROOTCERT` suggest TLS with a private CA. Use
`verify-full` with the CA mounted.
- **Backup and failure domain.** Where does the database live? It is part of the
M8.3/M8.7 evidence.
## Not inspected (needs read-only cluster access)
This workstation has no kubeconfig. Run
`tools/inventory/collect-cluster.sh <dir>` with a read-only (get/list) identity.
It never fetches Secret or ConfigMap data. Then record the results here.
| Item | Collected by | Needed for |
|---|---|---|
| Node count and roles, CPU, allocatable memory, kernel, runtime, taints | nodes.txt, node-resources.txt, nodes-describe.txt | Budgets, affinity plan (8.9) |
| Hypervisor, VM layout, host disks and RAID, overcommit | Maintainer (outside Kubernetes) | Failure domains, the 128 GB envelope |
| CNI and NetworkPolicy enforcement | daemonsets.txt, kube-system-pods.txt | The deny-by-default flows the region example relies on |
| CSI drivers, StorageClasses, reclaim and binding modes, node-affine PVs | storageclasses.*, pv.txt, csidrivers.txt | 8.3. A local PV binds a region to one node (risk named in the roadmap) |
| Volume snapshots | volumesnapshotclasses.txt | Backup design (8.7) |
| Ingress, load balancers, DNS, certificate issuers | ingress*.txt, loadbalancers.txt, coredns.txt, cert-issuers.txt | Metrics and probe exposure, later gateway |
| Quotas, limit ranges, priority classes, PDBs | quotas.txt, priorityclasses.txt, pdbs.txt | 8.2 isolation |
| Argo CD applications, auto-sync/prune, projects | argocd-apps.txt, argocd-projects.yaml | 8.2 World0 prune protection |
| Current usage | top-nodes.txt, top-pods.txt (needs metrics-server) | Available capacity |
| Storage latency per candidate class | `deploy/kubernetes/inventory/disk-probe.yaml.example`, applied by hand | Commit latency and lease margins |
## Measured workload profile (local fixture)
The measurements come from `durability_qualification`
(engineering/DURABILITY_QUALIFICATION.md): one lineage region under the
production lease, paced at 20 ticks/s, on WSL2 with a local PostgreSQL 18.6.
| Measure (Release, three successive owners) | Value |
|---|---|
| `evo-server` peak resident memory (VmHWM) | 9,940 KiB (about 9.7 MiB) |
| CPU while serving, including start and recovery | 60 ms over 16.3 s of wall time, about 0.3 % of one core |
| Committed payload | 37,756 bytes |
| Head verification p50 / max | 1.6 / 12 ms |
| Acknowledgement latency p50 / p99 (1 s flush) | 549 / 1048 ms |
| SIGKILL to successor ready | 19.6 s (lease-bound) |
Local storage baseline: the same script as the disk probe, run in WSL2 on ext4
over the workstation's NVMe (VHD):
| Pattern | Result |
|---|---|
| 4 KiB O_DSYNC writes | mean 1169 µs |
| 40 KiB object publication (write, fsync, rename, directory fsync) | mean 4646 µs, max 7983 µs |
| 256 MiB sequential with fdatasync | 365 ms |
Cluster storage should be compared against this baseline. A class whose
synchronous-write p99 approaches the 2 s renewal deadline cannot host regions
under the current lease policy.
A region host is small at this profile. Its requests are set by headroom, not by
the simulation itself. **Proposed** starting values, to be confirmed by in-cluster
measurement in M8.4: `requests: {cpu: 50m, memory: 64Mi}` and
`limits: {memory: 256Mi}` with no CPU limit (throttling would delay renewals).
The limit allows 8–16 MiB of process memory, the 4.3 MiB payload cap and
restore and verification buffers several times over. Larger profiles
(populations, fields) must be re-measured before these values are reused.
## Proposed budget model (to be replaced by measured inputs)
Reserve per 32 GB worker before placing any region:
- **Operating system and Kubernetes.** Kubelet, system and eviction reserves, and
the CNI/CSI agents: take the difference between capacity and allocatable from
`node-resources.txt`, plus the measured daemon usage.
- **Stores.** PostgreSQL memory (shared_buffers, work_mem × connections,
maintenance) and the object store, on the node or nodes that host them, sized by
M8.3 rather than guessed.
- **Observability.** Metrics and log agents, and at most 8 KiB of exposition per
region scrape (ADR-0066).
- **Recovery headroom.** A successor host can start while its fenced predecessor
still holds memory. Budget two region footprints per region. On the database
and object store, keep headroom for audit and clone work (read-only, bounded by
payload × chain length).
Region hosts are not the memory driver at the current profile size (see the
measured peak above). The first capacity limits are expected to be:
- storage latency and fsync behaviour;
- database connections (one per host plus the migrator);
- the lease margins: the 2 s renewal deadline and 5 s commit deadline
(ADR-0060) must stay well above p99 storage plus database latency.
Do not set per-node region counts until the probe results exist.
+3
View File
@@ -128,6 +128,9 @@ class Child {
bool wait_line(std::string_view prefix, std::chrono::milliseconds timeout) {
return wait_for([prefix](const std::string& l) { return starts(l, prefix); }, timeout);
}
[[nodiscard]] pid_t pid() const noexcept {
return pid_;
}
void signal(int sig) {
expect(::kill(pid_, sig) == 0, "signal");
}
+43 -1
View File
@@ -5,7 +5,10 @@
#include "evo/persistence/sha256.hpp"
#include "fixtures/server_process.hpp"
#include <algorithm>
#include <fstream>
#include <iterator>
#include <source_location>
#include <sstream>
#include <thread>
void run_region_migration_tests();
@@ -256,6 +259,28 @@ void upgrade_across_migration_and_barrier() {
CHECK(load_region_checkpoint(*owner_db, r.key, 5s)->head == final_head->head);
}
// Resource profile of a live process from /proc (M8.1 budgets): peak resident
// set (VmHWM, KiB) and consumed CPU time (user + system, ms).
struct ProcessProfile {
std::uint64_t peak_rss_kib{}, cpu_ms{};
};
ProcessProfile profile_of(pid_t pid) {
ProcessProfile out;
std::ifstream status("/proc/" + std::to_string(pid) + "/status");
for (std::string line; std::getline(status, line);)
if (line.starts_with("VmHWM:"))
out.peak_rss_kib = std::stoull(line.substr(6));
std::ifstream stat("/proc/" + std::to_string(pid) + "/stat");
std::string text((std::istreambuf_iterator<char>(stat)), {});
// Fields after the parenthesized command: state is field 3; utime/stime 14/15.
std::istringstream fields(text.substr(text.rfind(')') + 2));
std::vector<std::string> f{std::istream_iterator<std::string>(fields), {}};
CHECK(f.size() > 13 && out.peak_rss_kib > 0);
const auto ticks = std::stoull(f[11]) + std::stoull(f[12]);
out.cpu_ms = ticks * 1000 / static_cast<std::uint64_t>(::sysconf(_SC_CLK_TCK));
return out;
}
std::uint64_t percentile(std::vector<std::uint64_t> v, unsigned p) {
CHECK(!v.empty());
std::ranges::sort(v);
@@ -302,7 +327,8 @@ void durability_qualification() {
const auto args =
with(with(server_args(r), "--lease-profile", "default"), "--flush-ms", "1000");
std::vector<std::uint64_t> latency, kill_to_ready;
std::uint64_t death_latency = 0;
std::uint64_t death_latency = 0, peak_rss_kib = 0;
std::vector<std::uint64_t> cpu_ms, busy_ms; // Per owner: CPU used, wall time serving.
std::uint64_t next = 1;
std::uint64_t epoch = 1;
for (int round = 0; round < 3; ++round) {
@@ -326,6 +352,10 @@ void durability_qualification() {
server.send(input_lines(last + 1, last + 10, true));
server.pump(200ms);
const auto acked = max_acked(server);
const auto used = profile_of(server.pid());
peak_rss_kib = std::max(peak_rss_kib, used.peak_rss_kib);
cpu_ms.push_back(used.cpu_ms);
busy_ms.push_back(ms_between(sent.front(), Clock::now()));
server.signal(SIGKILL);
const auto killed = Clock::now();
CHECK(server.wait_exit(20s) == 128 + SIGKILL);
@@ -358,6 +388,18 @@ void durability_qualification() {
kill_to_ready.size(), static_cast<unsigned long long>(kill_to_ready[0]),
static_cast<unsigned long long>(kill_to_ready[1]));
std::uint64_t cpu_total = 0, busy_total = 0;
for (std::size_t i = 0; i < cpu_ms.size(); ++i) {
cpu_total += cpu_ms[i];
busy_total += busy_ms[i];
}
std::printf(
"qualification server_profile owners=%zu peak_rss_kib=%llu cpu_ms=%llu "
"wall_ms=%llu cpu_permille=%llu\n",
cpu_ms.size(), static_cast<unsigned long long>(peak_rss_kib),
static_cast<unsigned long long>(cpu_total), static_cast<unsigned long long>(busy_total),
static_cast<unsigned long long>(busy_total == 0 ? 0 : cpu_total * 1000 / busy_total));
// Verification cost on the final head (no lease needed to read).
auto reader = connect_to(env("PGDATABASE"), env("EVO_PG_RUNTIME_USER"));
auto objects = store(r);
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env bash
# M8.1 read-only cluster inventory. Run with a kubeconfig whose identity can only
# get/list (cluster-reader); it never creates, patches or deletes anything, and it
# never fetches Secret or ConfigMap data: Secrets are listed only through the
# server-side table (namespace, name, type, number of keys).
#
# tools/inventory/collect-cluster.sh <output-dir> [context]
#
# Review the output before sharing it: node names, addresses, hostnames and image
# references are infrastructure details, not credentials, but they are not public.
set -uo pipefail
out="${1:?output directory}"
context="${2:-}"
mkdir -p "$out"
chmod 700 "$out"
k=(kubectl --request-timeout=20s)
[ -n "$context" ] && k+=(--context "$context")
run() { # run <file> <command...>; failures are recorded, not fatal (missing CRDs).
local file="$out/$1"; shift
{ echo "\$ ${*}"; "$@" 2>&1; echo "exit=$?"; } > "$file"
}
run version.txt "${k[@]}" version
run whoami.txt "${k[@]}" auth whoami
run can-i.txt "${k[@]}" auth can-i --list
run api-resources.txt "${k[@]}" api-resources -o wide
# Nodes: capacity, allocatable, pressure, labels, taints, kernel/runtime, usage.
run nodes.txt "${k[@]}" get nodes -o wide --show-labels
run node-resources.txt "${k[@]}" get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memory,EPHEMERAL:.status.capacity.ephemeral-storage,PODS:.status.capacity.pods,ALLOC_CPU:.status.allocatable.cpu,ALLOC_MEM:.status.allocatable.memory,ALLOC_EPHEMERAL:.status.allocatable.ephemeral-storage,TAINTS:.spec.taints[*].key'
run nodes-describe.txt "${k[@]}" describe nodes
run top-nodes.txt "${k[@]}" top nodes
run top-pods.txt "${k[@]}" top pods -A --sort-by=memory
# Storage: classes, reclaim/binding modes, volumes and their node affinity, CSI.
run storageclasses.txt "${k[@]}" get storageclass -o custom-columns='NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy,BINDING:.volumeBindingMode,EXPAND:.allowVolumeExpansion,DEFAULT:.metadata.annotations.storageclass\.kubernetes\.io/is-default-class'
run storageclasses.yaml "${k[@]}" get storageclass -o yaml
run pv.txt "${k[@]}" get pv -o custom-columns='NAME:.metadata.name,CAPACITY:.spec.capacity.storage,RECLAIM:.spec.persistentVolumeReclaimPolicy,CLASS:.spec.storageClassName,STATUS:.status.phase,CLAIM:.spec.claimRef.namespace,CLAIMNAME:.spec.claimRef.name,NODE_AFFINITY:.spec.nodeAffinity.required.nodeSelectorTerms[*].matchExpressions[*].values'
run pvc.txt "${k[@]}" get pvc -A -o wide
run csidrivers.txt "${k[@]}" get csidrivers -o wide
run csinodes.txt "${k[@]}" get csinodes -o wide
run volumesnapshotclasses.txt "${k[@]}" get volumesnapshotclasses
# Network: CNI and other node agents, policies, ingress, load balancers, DNS.
run daemonsets.txt "${k[@]}" get daemonsets -A -o wide
run kube-system-pods.txt "${k[@]}" get pods -n kube-system -o wide
run networkpolicies.txt "${k[@]}" get networkpolicies -A
run ingressclasses.txt "${k[@]}" get ingressclass -o wide
run ingresses.txt "${k[@]}" get ingress -A -o wide
run loadbalancers.txt "${k[@]}" get services -A --field-selector spec.type=LoadBalancer -o wide
run coredns.txt "${k[@]}" get deploy,svc -n kube-system -l k8s-app=kube-dns -o wide
run cert-issuers.txt "${k[@]}" get clusterissuers,issuers -A
run certificates.txt "${k[@]}" get certificates -A
# Workloads, quotas and scheduling policy.
run namespaces.txt "${k[@]}" get namespaces --show-labels
run workloads.txt "${k[@]}" get deploy,statefulset,daemonset,job,cronjob -A -o wide
run quotas.txt "${k[@]}" get resourcequota,limitrange -A
run priorityclasses.txt "${k[@]}" get priorityclasses
run pdbs.txt "${k[@]}" get pdb -A
# Delivery: Argo CD applications, their sources and sync policies (auto-sync/prune).
run argocd-apps.txt "${k[@]}" get applications.argoproj.io -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,PROJECT:.spec.project,REPO:.spec.source.repoURL,PATH:.spec.source.path,REVISION:.spec.source.targetRevision,DEST_NS:.spec.destination.namespace,AUTOMATED:.spec.syncPolicy.automated,SYNC:.status.sync.status,HEALTH:.status.health.status'
run argocd-projects.yaml "${k[@]}" get appprojects.argoproj.io -A -o yaml
# Target namespace: what already runs there. Secrets are listed with the default
# server-side table (namespace, name, type, key count): values are never fetched.
run evolution-dev.txt "${k[@]}" get all,pvc,networkpolicy,serviceaccount -n evolution-dev -o wide
run secrets.txt "${k[@]}" get secrets -A
echo "inventory written to $out (review before sharing)"