Files
aevolution/docs/DEPLOYMENT.md
T
aevolution-gitops-bot d781492957
Foundation / checks (checks) (push) Successful in 35s
Foundation / checks (ecs) (push) Successful in 17s
Foundation / checks (network) (push) Successful in 18s
Foundation / checks (postgres) (push) Successful in 24s
Foundation / checks (deploy) (push) Successful in 3m15s
Foundation / checks (tsan) (push) Successful in 5m51s
Foundation / checks (client) (push) Failing after 27m3s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
fix deployment promotion workflow and guide
2026-10-06 13:39:49 -04:00

181 lines
5.6 KiB
Markdown

# AEvolution image deployment guide
This guide covers the supported path for promoting an immutable server image to
`evolution-dev`. It is deliberately conservative: image publication, GitOps
promotion, region-build retention, and Argo reconciliation are separate gates.
## Rules
- Promote immutable digests, never mutable tags.
- Do not edit live Deployments or run ad-hoc SQL against the region database.
- Do not reset the database, PVC, object store, region history, or world data.
- Stage and dev have independent image pins. A dev promotion must not change stage.
- A server image must be retained in the database before the region can acquire it.
- Use the cluster clock for `REGION_SESSION_WINDOW`; the expiry must be in the future
and within the server's maximum allowed window.
## 1. Start from current main
```bash
git fetch origin main
git checkout main
git reset --hard origin/main
git status --short
```
The working tree must be clean before promotion.
## 2. Build and publish the server image
Use the repository Foundation/build workflow for `main`. Do not build a local image
and substitute a tag in GitOps.
After the workflow succeeds, record the exact published digest:
```text
registry.aeternalisgames.com/aeternalis/aevolution-server@sha256:<64 lowercase hex>
```
Do not continue if the publication step was skipped or no immutable digest was
reported.
## 3. Promote the digest to dev
The supported Gitea action is **Promote server image to dev**. Supply the complete
`sha256:...` digest as `image_digest`.
The action updates both of these files, and they must match:
```text
deploy/kubernetes/region/dev/kustomization.yaml
deploy/kubernetes/region/dev/region-values.yaml
```
The action prepares the pinned kustomize binary before running policy validation.
If running the equivalent process locally, use:
```bash
python3 -m pip install --user --break-system-packages PyYAML
bash tools/ci/prepare-kustomize.sh
python3 tools/ci/deploy-policy-check.py
git diff --check
```
Then commit and push only the intended dev promotion changes:
```bash
git add deploy/kubernetes/region/dev/kustomization.yaml \
deploy/kubernetes/region/dev/region-values.yaml
git commit -m "deploy(dev): promote selected server image"
git push origin HEAD:main
```
The policy check must report zero failures. Do not bypass it with
`kubectl apply`, `--force`, or a mutable image tag.
## 4. Refresh the session window when the dev route is enabled
Generate the window using the cluster-synchronized clock:
```bash
ssh shard@172.10.1.160 '
id="dev-$(date -u +%Y%m%dT%H%M%S)-$RANDOM"
expiry="$(date -u -d "+20 minutes" +%s)"
printf "%s:%s\\n" "$id" "$expiry"
'
```
Update the value in:
```text
deploy/kubernetes/region/dev-session-route/listener.yaml
```
Run the policy check, commit, push, and sync again. The identifier is opaque; the
contract is `identifier:unix-expiry`.
## 5. Run a full Argo sync
A resource-targeted sync is not sufficient for an image promotion. It can update the
Deployment while skipping the sync-wave provision hook.
Use a full sync of `aevolution-dev`, not a sync limited to `Deployment/evo-region`.
Verify the Application revision and operation before proceeding.
The sync order is:
| Wave | Resource | Purpose |
|---:|---|---|
| -2 | database egress/policy | Permit the migration/provision path |
| -1 | database migration Job | Apply schema migrations and privilege audit |
| 0 | `aevolution-region-provision` hook | Retain the selected server build |
| 1 | `evo-region` Deployment | Start the region using that retained build |
## 6. Verify the provision hook
The hook must use the same server digest as the Deployment. A successful existing
region normally reports `already-provisioned`, but the command still runs
`--retain-build` before that check and therefore records a newly promoted build.
Check:
```bash
kubectl -n evolution-dev get job aevolution-region-provision -o wide
kubectl -n evolution-dev logs job/aevolution-region-provision
```
If the new build is not retained, the region will start and immediately show:
```text
phase quarantined epoch=0 committed=0 simulated=0
exit 4
```
That is a build-retention failure, not a reason to reset the region. Re-run the
supported sync hook using the current rendered manifest.
## 7. Verify the region
All of these must be true:
```bash
kubectl -n evolution-dev get pod -l app.kubernetes.io/name=evo-region
kubectl -n evolution-dev logs deployment/evo-region --tail=50
kubectl -n argocd get application aevolution-dev
```
Expected runtime indicators include:
```text
phase running
ready 1
```
Also verify `Ready=True`, zero new restarts, and Argo `Synced / Healthy`. `Running`
alone is not sufficient.
## 8. Probe and acceptance
For an authenticated M10 probe, retain the matching operator bundle until all tests
finish:
- `dev-ca.pem`
- `issuer.key`
The normal dev overlay has `REGION_INPUTS` empty, so a probe can authenticate and
receive an intention receipt without producing a simulated outcome. Full durable
outcome testing requires the repository-supported fault-drill/synthetic-input path;
do not enable it with an ad-hoc live environment change.
## Recovery checklist
If the region enters CrashLoopBackOff:
1. Capture the current image digest, `REGION_BUILD`, Argo revision, previous logs, and
Job image.
2. If logs show `quarantined epoch=0` with zero store failures, compare the build
against the retained-build Job/database path.
3. If the provision Job is stale, perform a full Argo sync so the sync-wave 0 hook runs.
4. Never delete the database, PVC, object store, or committed region history.
5. Do not force-sync only the Deployment while the provision hook is pending.