diff --git a/MEMORY.md b/MEMORY.md index eb57f3c..fd043a2 100644 --- a/MEMORY.md +++ b/MEMORY.md @@ -24,3 +24,4 @@ Long-term curated memory for Shard. - 2026-04-03: SOR auth reliability advanced at `cfb10ae` with Keycloak audience-validation and packet-drain race fixes, setting up end-to-end login/reconnect validation as the next execution gate. - 2026-04-04: SOR admin auth hardening progressed to `e4ddd80`, persisting DataProtection keys in Redis to prevent cross-pod cookie decryption failures in multi-pod deployments. - 2026-04-04: Platform IAM posture tightened with Argo CD default-deny RBAC (explicit `/argo-admin` + `/argo-readonly` grants only) and removal of local Argo user logins. +- 2026-04-08: Critical infra lesson captured — new VMs (including Ceph/PostgreSQL roles) must use the storage pool, not host local primary disks (90–100GB). Prior local-disk placement filled hypervisor disks and caused VM pauses. diff --git a/memory/2026-04-08.md b/memory/2026-04-08.md index 2a213d0..deb03d6 100644 --- a/memory/2026-04-08.md +++ b/memory/2026-04-08.md @@ -7,6 +7,7 @@ - 2026-04-08 03:27 UTC Heartbeat SOR check: unchanged dirty working tree (OVERALL.md, deploy/k8s/ingress.yaml); latest commit still e4ddd80. No new blocker signal. - 03:57 UTC heartbeat: SOR still has unstaged changes in `OVERALL.md` and `deploy/k8s/ingress.yaml`; HEAD remains `e4ddd80` (Redis-backed DataProtection keys). No new blocker observed; next action is to finalize scope and commit or revert pending edits. - 04:27 UTC heartbeat: SOR status unchanged (`OVERALL.md`, `deploy/k8s/ingress.yaml` modified), HEAD `e4ddd80`. No new blocker signal from repo check. +- 16:xx UTC note from Jason: root cause learned for paused VMs — Ceph/PostgreSQL VMs had been created on hypervisor local disks instead of storage pool, filling host primary disks and pausing VMs. Rule going forward: always place new VM disks on storage pool; never use host 90–100GB primary disks. - 11:31 UTC heartbeat: SOR head advanced to `f59f0af`; local changes still `OVERALL.md` + `deploy/k8s/ingress.yaml`. No blocker signal. - 08:57 UTC heartbeat: SOR unchanged; working tree still dirty in `OVERALL.md` and `deploy/k8s/ingress.yaml`, HEAD `e4ddd80`. No new blocker signal. - 08:27 UTC heartbeat: SOR state unchanged (`OVERALL.md`, `deploy/k8s/ingress.yaml` modified), HEAD `e4ddd80`. No blocker signal. diff --git a/memory/heartbeat-state.json b/memory/heartbeat-state.json index ef2b03c..cf0414d 100644 --- a/memory/heartbeat-state.json +++ b/memory/heartbeat-state.json @@ -1,5 +1,5 @@ { - "lastCheckAt": "2026-04-08T11:31:00Z", + "lastCheckAt": "2026-04-08T22:57:00Z", "lastHead": "f59f0af", "lastDirtySet": [ "OVERALL.md",