Files
dingoandClaude Opus 5.5 47b84bce74
Foundation / checks (client) (push) Successful in 4s
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 4s
Foundation / checks (deploy) (push) Successful in 3m31s
Foundation / postgres-tools (push) Successful in 5s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 5s
Refuse session windows beyond the server's 4-hour maximum in deploy policy
A window ending 5.2 h after start (b2f224d) put evo-region into a
crashloop (error m10-window-too-long); the server only sees it at
startup. The deploy policy now requires REGION_SESSION_WINDOW to name its
UTC activation and end within 4 h of it, with two selftest cases (one is
the b2f224d value). DEPLOYMENT.md documents the follow-on quarantine:
syncs recorded as Deployment-only run no hooks, so a promoted build is
never retained; the provision hook is re-run from the rendered manifest.
VALIDATION records the incident and the recovery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-08 07:41:06 -04:00

6.7 KiB

AEvolution image deployment guide

This guide covers the supported path for promoting an immutable server image to evolution-dev. It is deliberately conservative: image publication, GitOps promotion, region-build retention, and Argo reconciliation are separate gates.

Rules

  • Promote immutable digests, never mutable tags.
  • Do not edit live Deployments or run ad-hoc SQL against the region database.
  • Do not reset the database, PVC, object store, region history, or world data.
  • Stage and dev have independent image pins. A dev promotion must not change stage.
  • A server image must be retained in the database before the region can acquire it.
  • Use the cluster clock for REGION_SESSION_WINDOW; the expiry must be in the future and within the server's maximum allowed window.

1. Start from current main

git fetch origin main
git checkout main
git reset --hard origin/main
git status --short

The working tree must be clean before promotion.

2. Build and publish the server image

Use the repository Foundation/build workflow for main. Do not build a local image and substitute a tag in GitOps.

After the workflow succeeds, record the exact published digest:

registry.aeternalisgames.com/aeternalis/aevolution-server@sha256:<64 lowercase hex>

Do not continue if the publication step was skipped or no immutable digest was reported.

3. Promote the digest to dev

The supported Gitea action is Promote server image to dev. Supply the complete sha256:... digest as image_digest.

The action updates both of these files, and they must match:

deploy/kubernetes/region/dev/kustomization.yaml
deploy/kubernetes/region/dev/region-values.yaml

The action prepares the pinned kustomize binary before running policy validation. If running the equivalent process locally, use:

python3 -m pip install --user --break-system-packages PyYAML
bash tools/ci/prepare-kustomize.sh
python3 tools/ci/deploy-policy-check.py
git diff --check

Then commit and push only the intended dev promotion changes:

git add deploy/kubernetes/region/dev/kustomization.yaml \
        deploy/kubernetes/region/dev/region-values.yaml
git commit -m "deploy(dev): promote selected server image"
git push origin HEAD:main

The policy check must report zero failures. Do not bypass it with kubectl apply, --force, or a mutable image tag.

4. Refresh the session window when the dev route is enabled

Generate the window using the cluster-synchronized clock:

ssh shard@172.10.1.160 '
  id="dev-$(date -u +%Y%m%dT%H%M%S)-$RANDOM"
  expiry="$(date -u -d "+20 minutes" +%s)"
  printf "%s:%s\\n" "$id" "$expiry"
'

Update the value in:

deploy/kubernetes/region/dev-session-route/listener.yaml

Run the policy check, commit, push, and sync again. The identifier is opaque; the contract is identifier:unix-expiry.

5. Run a full Argo sync

A resource-targeted sync is not sufficient for an image promotion. It can update the Deployment while skipping the sync-wave provision hook.

Use a full sync of aevolution-dev, not a sync limited to Deployment/evo-region. Verify the Application revision and operation before proceeding.

The sync order is:

Wave Resource Purpose
-2 database egress/policy Permit the migration/provision path
-1 database migration Job Apply schema migrations and privilege audit
0 aevolution-region-provision hook Retain the selected server build
1 evo-region Deployment Start the region using that retained build

6. Verify the provision hook

The hook must use the same server digest as the Deployment. A successful existing region normally reports already-provisioned, but the command still runs --retain-build before that check and therefore records a newly promoted build.

Check:

kubectl -n evolution-dev get job aevolution-region-provision -o wide
kubectl -n evolution-dev logs job/aevolution-region-provision

If the new build is not retained, the region will start and immediately show:

phase quarantined epoch=0 committed=0 simulated=0
exit 4

That is a build-retention failure, not a reason to reset the region. Re-run the supported sync hook using the current rendered manifest.

A sync that Argo records with resources: [Deployment/evo-region] runs no hooks. On 2026-10-08 syncs requested with the aevolution-dev-agent token were recorded that way even when no resource filter was requested, so a promotion never retained its build. Check the operation with:

kubectl -n argocd get application aevolution-dev -o jsonpath='{.status.operationState.operation.sync}'

If it was limited to the Deployment, re-run the provision hook from the rendered manifest:

  1. Render deploy/kubernetes/dev with the pinned kustomize and take Job/aevolution-region-provision. Its image and REGION_BUILD must equal the Deployment's.
  2. Delete the old Job, create the rendered one and wait for Complete. The log shows already-provisioned; the build was retained first.
  3. kubectl -n evolution-dev rollout restart deployment/evo-region, then verify below.

REGION_SESSION_WINDOW must name its UTC activation (dev-YYYYMMDDTHHMMSS-...) and end at most 4 hours after it. evo-server refuses a longer window at startup (error m10-window-too-long), and the deploy policy check refuses it before push.

7. Verify the region

All of these must be true:

kubectl -n evolution-dev get pod -l app.kubernetes.io/name=evo-region
kubectl -n evolution-dev logs deployment/evo-region --tail=50
kubectl -n argocd get application aevolution-dev

Expected runtime indicators include:

phase running
ready 1

Also verify Ready=True, zero new restarts, and Argo Synced / Healthy. Running alone is not sufficient.

8. Probe and acceptance

For an authenticated M10 probe, retain the matching operator bundle until all tests finish:

  • dev-ca.pem
  • issuer.key

The normal dev overlay has REGION_INPUTS empty, so a probe can authenticate and receive an intention receipt without producing a simulated outcome. Full durable outcome testing requires the repository-supported fault-drill/synthetic-input path; do not enable it with an ad-hoc live environment change.

Recovery checklist

If the region enters CrashLoopBackOff:

  1. Capture the current image digest, REGION_BUILD, Argo revision, previous logs, and Job image.
  2. If logs show quarantined epoch=0 with zero store failures, compare the build against the retained-build Job/database path.
  3. If the provision Job is stale, perform a full Argo sync so the sync-wave 0 hook runs.
  4. Never delete the database, PVC, object store, or committed region history.
  5. Do not force-sync only the Deployment while the provision hook is pending.