Foundation / checks (client) (push) Successful in 4s
Foundation / checks (checks) (push) Successful in 5s
Foundation / checks (ecs) (push) Successful in 5s
Foundation / checks (network) (push) Successful in 5s
Foundation / checks (postgres) (push) Successful in 5s
Foundation / checks (tsan) (push) Successful in 4s
Foundation / checks (deploy) (push) Successful in 3m31s
Foundation / postgres-tools (push) Successful in 5s
Foundation / server-image (push) Successful in 5s
Foundation / image (push) Successful in 5s
A window ending 5.2 h after start (b2f224d) put evo-region into a crashloop (error m10-window-too-long); the server only sees it at startup. The deploy policy now requires REGION_SESSION_WINDOW to name its UTC activation and end within 4 h of it, with two selftest cases (one is theb2f224dvalue). DEPLOYMENT.md documents the follow-on quarantine: syncs recorded as Deployment-only run no hooks, so a promoted build is never retained; the provision hook is re-run from the rendered manifest. VALIDATION records the incident and the recovery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
202 lines
6.7 KiB
Markdown
202 lines
6.7 KiB
Markdown
# AEvolution image deployment guide
|
|
|
|
This guide covers the supported path for promoting an immutable server image to
|
|
`evolution-dev`. It is deliberately conservative: image publication, GitOps
|
|
promotion, region-build retention, and Argo reconciliation are separate gates.
|
|
|
|
## Rules
|
|
|
|
- Promote immutable digests, never mutable tags.
|
|
- Do not edit live Deployments or run ad-hoc SQL against the region database.
|
|
- Do not reset the database, PVC, object store, region history, or world data.
|
|
- Stage and dev have independent image pins. A dev promotion must not change stage.
|
|
- A server image must be retained in the database before the region can acquire it.
|
|
- Use the cluster clock for `REGION_SESSION_WINDOW`; the expiry must be in the future
|
|
and within the server's maximum allowed window.
|
|
|
|
## 1. Start from current main
|
|
|
|
```bash
|
|
git fetch origin main
|
|
git checkout main
|
|
git reset --hard origin/main
|
|
git status --short
|
|
```
|
|
|
|
The working tree must be clean before promotion.
|
|
|
|
## 2. Build and publish the server image
|
|
|
|
Use the repository Foundation/build workflow for `main`. Do not build a local image
|
|
and substitute a tag in GitOps.
|
|
|
|
After the workflow succeeds, record the exact published digest:
|
|
|
|
```text
|
|
registry.aeternalisgames.com/aeternalis/aevolution-server@sha256:<64 lowercase hex>
|
|
```
|
|
|
|
Do not continue if the publication step was skipped or no immutable digest was
|
|
reported.
|
|
|
|
## 3. Promote the digest to dev
|
|
|
|
The supported Gitea action is **Promote server image to dev**. Supply the complete
|
|
`sha256:...` digest as `image_digest`.
|
|
|
|
The action updates both of these files, and they must match:
|
|
|
|
```text
|
|
deploy/kubernetes/region/dev/kustomization.yaml
|
|
deploy/kubernetes/region/dev/region-values.yaml
|
|
```
|
|
|
|
The action prepares the pinned kustomize binary before running policy validation.
|
|
If running the equivalent process locally, use:
|
|
|
|
```bash
|
|
python3 -m pip install --user --break-system-packages PyYAML
|
|
bash tools/ci/prepare-kustomize.sh
|
|
python3 tools/ci/deploy-policy-check.py
|
|
git diff --check
|
|
```
|
|
|
|
Then commit and push only the intended dev promotion changes:
|
|
|
|
```bash
|
|
git add deploy/kubernetes/region/dev/kustomization.yaml \
|
|
deploy/kubernetes/region/dev/region-values.yaml
|
|
git commit -m "deploy(dev): promote selected server image"
|
|
git push origin HEAD:main
|
|
```
|
|
|
|
The policy check must report zero failures. Do not bypass it with
|
|
`kubectl apply`, `--force`, or a mutable image tag.
|
|
|
|
## 4. Refresh the session window when the dev route is enabled
|
|
|
|
Generate the window using the cluster-synchronized clock:
|
|
|
|
```bash
|
|
ssh shard@172.10.1.160 '
|
|
id="dev-$(date -u +%Y%m%dT%H%M%S)-$RANDOM"
|
|
expiry="$(date -u -d "+20 minutes" +%s)"
|
|
printf "%s:%s\\n" "$id" "$expiry"
|
|
'
|
|
```
|
|
|
|
Update the value in:
|
|
|
|
```text
|
|
deploy/kubernetes/region/dev-session-route/listener.yaml
|
|
```
|
|
|
|
Run the policy check, commit, push, and sync again. The identifier is opaque; the
|
|
contract is `identifier:unix-expiry`.
|
|
|
|
## 5. Run a full Argo sync
|
|
|
|
A resource-targeted sync is not sufficient for an image promotion. It can update the
|
|
Deployment while skipping the sync-wave provision hook.
|
|
|
|
Use a full sync of `aevolution-dev`, not a sync limited to `Deployment/evo-region`.
|
|
Verify the Application revision and operation before proceeding.
|
|
|
|
The sync order is:
|
|
|
|
| Wave | Resource | Purpose |
|
|
|---:|---|---|
|
|
| -2 | database egress/policy | Permit the migration/provision path |
|
|
| -1 | database migration Job | Apply schema migrations and privilege audit |
|
|
| 0 | `aevolution-region-provision` hook | Retain the selected server build |
|
|
| 1 | `evo-region` Deployment | Start the region using that retained build |
|
|
|
|
## 6. Verify the provision hook
|
|
|
|
The hook must use the same server digest as the Deployment. A successful existing
|
|
region normally reports `already-provisioned`, but the command still runs
|
|
`--retain-build` before that check and therefore records a newly promoted build.
|
|
|
|
Check:
|
|
|
|
```bash
|
|
kubectl -n evolution-dev get job aevolution-region-provision -o wide
|
|
kubectl -n evolution-dev logs job/aevolution-region-provision
|
|
```
|
|
|
|
If the new build is not retained, the region will start and immediately show:
|
|
|
|
```text
|
|
phase quarantined epoch=0 committed=0 simulated=0
|
|
exit 4
|
|
```
|
|
|
|
That is a build-retention failure, not a reason to reset the region. Re-run the
|
|
supported sync hook using the current rendered manifest.
|
|
|
|
A sync that Argo records with `resources: [Deployment/evo-region]` runs no hooks. On
|
|
2026-10-08 syncs requested with the `aevolution-dev-agent` token were recorded that way
|
|
even when no resource filter was requested, so a promotion never retained its build.
|
|
Check the operation with:
|
|
|
|
```bash
|
|
kubectl -n argocd get application aevolution-dev -o jsonpath='{.status.operationState.operation.sync}'
|
|
```
|
|
|
|
If it was limited to the Deployment, re-run the provision hook from the rendered manifest:
|
|
1. Render `deploy/kubernetes/dev` with the pinned kustomize and take
|
|
`Job/aevolution-region-provision`. Its image and `REGION_BUILD` must equal the
|
|
Deployment's.
|
|
2. Delete the old Job, create the rendered one and wait for `Complete`. The log shows
|
|
`already-provisioned`; the build was retained first.
|
|
3. `kubectl -n evolution-dev rollout restart deployment/evo-region`, then verify below.
|
|
|
|
`REGION_SESSION_WINDOW` must name its UTC activation (`dev-YYYYMMDDTHHMMSS-...`) and end
|
|
at most 4 hours after it. evo-server refuses a longer window at startup
|
|
(`error m10-window-too-long`), and the deploy policy check refuses it before push.
|
|
|
|
## 7. Verify the region
|
|
|
|
All of these must be true:
|
|
|
|
```bash
|
|
kubectl -n evolution-dev get pod -l app.kubernetes.io/name=evo-region
|
|
kubectl -n evolution-dev logs deployment/evo-region --tail=50
|
|
kubectl -n argocd get application aevolution-dev
|
|
```
|
|
|
|
Expected runtime indicators include:
|
|
|
|
```text
|
|
phase running
|
|
ready 1
|
|
```
|
|
|
|
Also verify `Ready=True`, zero new restarts, and Argo `Synced / Healthy`. `Running`
|
|
alone is not sufficient.
|
|
|
|
## 8. Probe and acceptance
|
|
|
|
For an authenticated M10 probe, retain the matching operator bundle until all tests
|
|
finish:
|
|
|
|
- `dev-ca.pem`
|
|
- `issuer.key`
|
|
|
|
The normal dev overlay has `REGION_INPUTS` empty, so a probe can authenticate and
|
|
receive an intention receipt without producing a simulated outcome. Full durable
|
|
outcome testing requires the repository-supported fault-drill/synthetic-input path;
|
|
do not enable it with an ad-hoc live environment change.
|
|
|
|
## Recovery checklist
|
|
|
|
If the region enters CrashLoopBackOff:
|
|
|
|
1. Capture the current image digest, `REGION_BUILD`, Argo revision, previous logs, and
|
|
Job image.
|
|
2. If logs show `quarantined epoch=0` with zero store failures, compare the build
|
|
against the retained-build Job/database path.
|
|
3. If the provision Job is stale, perform a full Argo sync so the sync-wave 0 hook runs.
|
|
4. Never delete the database, PVC, object store, or committed region history.
|
|
5. Do not force-sync only the Deployment while the provision hook is pending.
|