A window ending 5.2 h after start (b2f224d) put evo-region into a crashloop (error m10-window-too-long); the server only sees it at startup. The deploy policy now requires REGION_SESSION_WINDOW to name its UTC activation and end within 4 h of it, with two selftest cases (one is theb2f224dvalue). DEPLOYMENT.md documents the follow-on quarantine: syncs recorded as Deployment-only run no hooks, so a promoted build is never retained; the provision hook is re-run from the rendered manifest. VALIDATION records the incident and the recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.7 KiB
AEvolution image deployment guide
This guide covers the supported path for promoting an immutable server image to
evolution-dev. It is deliberately conservative: image publication, GitOps
promotion, region-build retention, and Argo reconciliation are separate gates.
Rules
- Promote immutable digests, never mutable tags.
- Do not edit live Deployments or run ad-hoc SQL against the region database.
- Do not reset the database, PVC, object store, region history, or world data.
- Stage and dev have independent image pins. A dev promotion must not change stage.
- A server image must be retained in the database before the region can acquire it.
- Use the cluster clock for
REGION_SESSION_WINDOW; the expiry must be in the future and within the server's maximum allowed window.
1. Start from current main
git fetch origin main
git checkout main
git reset --hard origin/main
git status --short
The working tree must be clean before promotion.
2. Build and publish the server image
Use the repository Foundation/build workflow for main. Do not build a local image
and substitute a tag in GitOps.
After the workflow succeeds, record the exact published digest:
registry.aeternalisgames.com/aeternalis/aevolution-server@sha256:<64 lowercase hex>
Do not continue if the publication step was skipped or no immutable digest was reported.
3. Promote the digest to dev
The supported Gitea action is Promote server image to dev. Supply the complete
sha256:... digest as image_digest.
The action updates both of these files, and they must match:
deploy/kubernetes/region/dev/kustomization.yaml
deploy/kubernetes/region/dev/region-values.yaml
The action prepares the pinned kustomize binary before running policy validation. If running the equivalent process locally, use:
python3 -m pip install --user --break-system-packages PyYAML
bash tools/ci/prepare-kustomize.sh
python3 tools/ci/deploy-policy-check.py
git diff --check
Then commit and push only the intended dev promotion changes:
git add deploy/kubernetes/region/dev/kustomization.yaml \
deploy/kubernetes/region/dev/region-values.yaml
git commit -m "deploy(dev): promote selected server image"
git push origin HEAD:main
The policy check must report zero failures. Do not bypass it with
kubectl apply, --force, or a mutable image tag.
4. Refresh the session window when the dev route is enabled
Generate the window using the cluster-synchronized clock:
ssh shard@172.10.1.160 '
id="dev-$(date -u +%Y%m%dT%H%M%S)-$RANDOM"
expiry="$(date -u -d "+20 minutes" +%s)"
printf "%s:%s\\n" "$id" "$expiry"
'
Update the value in:
deploy/kubernetes/region/dev-session-route/listener.yaml
Run the policy check, commit, push, and sync again. The identifier is opaque; the
contract is identifier:unix-expiry.
5. Run a full Argo sync
A resource-targeted sync is not sufficient for an image promotion. It can update the Deployment while skipping the sync-wave provision hook.
Use a full sync of aevolution-dev, not a sync limited to Deployment/evo-region.
Verify the Application revision and operation before proceeding.
The sync order is:
| Wave | Resource | Purpose |
|---|---|---|
| -2 | database egress/policy | Permit the migration/provision path |
| -1 | database migration Job | Apply schema migrations and privilege audit |
| 0 | aevolution-region-provision hook |
Retain the selected server build |
| 1 | evo-region Deployment |
Start the region using that retained build |
6. Verify the provision hook
The hook must use the same server digest as the Deployment. A successful existing
region normally reports already-provisioned, but the command still runs
--retain-build before that check and therefore records a newly promoted build.
Check:
kubectl -n evolution-dev get job aevolution-region-provision -o wide
kubectl -n evolution-dev logs job/aevolution-region-provision
If the new build is not retained, the region will start and immediately show:
phase quarantined epoch=0 committed=0 simulated=0
exit 4
That is a build-retention failure, not a reason to reset the region. Re-run the supported sync hook using the current rendered manifest.
A sync that Argo records with resources: [Deployment/evo-region] runs no hooks. On
2026-10-08 syncs requested with the aevolution-dev-agent token were recorded that way
even when no resource filter was requested, so a promotion never retained its build.
Check the operation with:
kubectl -n argocd get application aevolution-dev -o jsonpath='{.status.operationState.operation.sync}'
If it was limited to the Deployment, re-run the provision hook from the rendered manifest:
- Render
deploy/kubernetes/devwith the pinned kustomize and takeJob/aevolution-region-provision. Its image andREGION_BUILDmust equal the Deployment's. - Delete the old Job, create the rendered one and wait for
Complete. The log showsalready-provisioned; the build was retained first. kubectl -n evolution-dev rollout restart deployment/evo-region, then verify below.
REGION_SESSION_WINDOW must name its UTC activation (dev-YYYYMMDDTHHMMSS-...) and end
at most 4 hours after it. evo-server refuses a longer window at startup
(error m10-window-too-long), and the deploy policy check refuses it before push.
7. Verify the region
All of these must be true:
kubectl -n evolution-dev get pod -l app.kubernetes.io/name=evo-region
kubectl -n evolution-dev logs deployment/evo-region --tail=50
kubectl -n argocd get application aevolution-dev
Expected runtime indicators include:
phase running
ready 1
Also verify Ready=True, zero new restarts, and Argo Synced / Healthy. Running
alone is not sufficient.
8. Probe and acceptance
For an authenticated M10 probe, retain the matching operator bundle until all tests finish:
dev-ca.pemissuer.key
The normal dev overlay has REGION_INPUTS empty, so a probe can authenticate and
receive an intention receipt without producing a simulated outcome. Full durable
outcome testing requires the repository-supported fault-drill/synthetic-input path;
do not enable it with an ad-hoc live environment change.
Recovery checklist
If the region enters CrashLoopBackOff:
- Capture the current image digest,
REGION_BUILD, Argo revision, previous logs, and Job image. - If logs show
quarantined epoch=0with zero store failures, compare the build against the retained-build Job/database path. - If the provision Job is stale, perform a full Argo sync so the sync-wave 0 hook runs.
- Never delete the database, PVC, object store, or committed region history.
- Do not force-sync only the Deployment while the provision hook is pending.