fix deployment promotion workflow and guide
Foundation / checks (checks) (push) Successful in 35s
Foundation / checks (ecs) (push) Successful in 17s
Foundation / checks (network) (push) Successful in 18s
Foundation / checks (postgres) (push) Successful in 24s
Foundation / checks (deploy) (push) Successful in 3m15s
Foundation / checks (tsan) (push) Successful in 5m51s
Foundation / checks (client) (push) Failing after 27m3s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
Foundation / checks (checks) (push) Successful in 35s
Foundation / checks (ecs) (push) Successful in 17s
Foundation / checks (network) (push) Successful in 18s
Foundation / checks (postgres) (push) Successful in 24s
Foundation / checks (deploy) (push) Successful in 3m15s
Foundation / checks (tsan) (push) Successful in 5m51s
Foundation / checks (client) (push) Failing after 27m3s
Foundation / postgres-tools (push) Skipped
Foundation / server-image (push) Skipped
Foundation / image (push) Skipped
This commit is contained in:
@@ -32,6 +32,11 @@ jobs:
|
||||
python3 -m pip install --user --break-system-packages \
|
||||
--disable-pip-version-check PyYAML
|
||||
|
||||
- name: Prepare pinned kustomize
|
||||
run: |
|
||||
set -eu
|
||||
bash tools/ci/prepare-kustomize.sh
|
||||
|
||||
- name: Update dev image and build identity
|
||||
env:
|
||||
IMAGE_DIGEST: ${{ inputs.image_digest }}
|
||||
|
||||
@@ -12,7 +12,7 @@ spec:
|
||||
env:
|
||||
- {name: REGION_SESSION, value: "dev"}
|
||||
- {name: REGION_SESSION_LISTEN, value: "0.0.0.0:7400"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261006T170005-18538:1791307205"}
|
||||
- {name: REGION_SESSION_WINDOW, value: "dev-20261006T173934:1791309574"}
|
||||
ports:
|
||||
# QUIC (ADR-0090): UDP only; nothing listens on TCP 7400.
|
||||
- {name: session, containerPort: 7400, protocol: UDP}
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
# AEvolution image deployment guide
|
||||
|
||||
This guide covers the supported path for promoting an immutable server image to
|
||||
`evolution-dev`. It is deliberately conservative: image publication, GitOps
|
||||
promotion, region-build retention, and Argo reconciliation are separate gates.
|
||||
|
||||
## Rules
|
||||
|
||||
- Promote immutable digests, never mutable tags.
|
||||
- Do not edit live Deployments or run ad-hoc SQL against the region database.
|
||||
- Do not reset the database, PVC, object store, region history, or world data.
|
||||
- Stage and dev have independent image pins. A dev promotion must not change stage.
|
||||
- A server image must be retained in the database before the region can acquire it.
|
||||
- Use the cluster clock for `REGION_SESSION_WINDOW`; the expiry must be in the future
|
||||
and within the server's maximum allowed window.
|
||||
|
||||
## 1. Start from current main
|
||||
|
||||
```bash
|
||||
git fetch origin main
|
||||
git checkout main
|
||||
git reset --hard origin/main
|
||||
git status --short
|
||||
```
|
||||
|
||||
The working tree must be clean before promotion.
|
||||
|
||||
## 2. Build and publish the server image
|
||||
|
||||
Use the repository Foundation/build workflow for `main`. Do not build a local image
|
||||
and substitute a tag in GitOps.
|
||||
|
||||
After the workflow succeeds, record the exact published digest:
|
||||
|
||||
```text
|
||||
registry.aeternalisgames.com/aeternalis/aevolution-server@sha256:<64 lowercase hex>
|
||||
```
|
||||
|
||||
Do not continue if the publication step was skipped or no immutable digest was
|
||||
reported.
|
||||
|
||||
## 3. Promote the digest to dev
|
||||
|
||||
The supported Gitea action is **Promote server image to dev**. Supply the complete
|
||||
`sha256:...` digest as `image_digest`.
|
||||
|
||||
The action updates both of these files, and they must match:
|
||||
|
||||
```text
|
||||
deploy/kubernetes/region/dev/kustomization.yaml
|
||||
deploy/kubernetes/region/dev/region-values.yaml
|
||||
```
|
||||
|
||||
The action prepares the pinned kustomize binary before running policy validation.
|
||||
If running the equivalent process locally, use:
|
||||
|
||||
```bash
|
||||
python3 -m pip install --user --break-system-packages PyYAML
|
||||
bash tools/ci/prepare-kustomize.sh
|
||||
python3 tools/ci/deploy-policy-check.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Then commit and push only the intended dev promotion changes:
|
||||
|
||||
```bash
|
||||
git add deploy/kubernetes/region/dev/kustomization.yaml \
|
||||
deploy/kubernetes/region/dev/region-values.yaml
|
||||
git commit -m "deploy(dev): promote selected server image"
|
||||
git push origin HEAD:main
|
||||
```
|
||||
|
||||
The policy check must report zero failures. Do not bypass it with
|
||||
`kubectl apply`, `--force`, or a mutable image tag.
|
||||
|
||||
## 4. Refresh the session window when the dev route is enabled
|
||||
|
||||
Generate the window using the cluster-synchronized clock:
|
||||
|
||||
```bash
|
||||
ssh shard@172.10.1.160 '
|
||||
id="dev-$(date -u +%Y%m%dT%H%M%S)-$RANDOM"
|
||||
expiry="$(date -u -d "+20 minutes" +%s)"
|
||||
printf "%s:%s\\n" "$id" "$expiry"
|
||||
'
|
||||
```
|
||||
|
||||
Update the value in:
|
||||
|
||||
```text
|
||||
deploy/kubernetes/region/dev-session-route/listener.yaml
|
||||
```
|
||||
|
||||
Run the policy check, commit, push, and sync again. The identifier is opaque; the
|
||||
contract is `identifier:unix-expiry`.
|
||||
|
||||
## 5. Run a full Argo sync
|
||||
|
||||
A resource-targeted sync is not sufficient for an image promotion. It can update the
|
||||
Deployment while skipping the sync-wave provision hook.
|
||||
|
||||
Use a full sync of `aevolution-dev`, not a sync limited to `Deployment/evo-region`.
|
||||
Verify the Application revision and operation before proceeding.
|
||||
|
||||
The sync order is:
|
||||
|
||||
| Wave | Resource | Purpose |
|
||||
|---:|---|---|
|
||||
| -2 | database egress/policy | Permit the migration/provision path |
|
||||
| -1 | database migration Job | Apply schema migrations and privilege audit |
|
||||
| 0 | `aevolution-region-provision` hook | Retain the selected server build |
|
||||
| 1 | `evo-region` Deployment | Start the region using that retained build |
|
||||
|
||||
## 6. Verify the provision hook
|
||||
|
||||
The hook must use the same server digest as the Deployment. A successful existing
|
||||
region normally reports `already-provisioned`, but the command still runs
|
||||
`--retain-build` before that check and therefore records a newly promoted build.
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
kubectl -n evolution-dev get job aevolution-region-provision -o wide
|
||||
kubectl -n evolution-dev logs job/aevolution-region-provision
|
||||
```
|
||||
|
||||
If the new build is not retained, the region will start and immediately show:
|
||||
|
||||
```text
|
||||
phase quarantined epoch=0 committed=0 simulated=0
|
||||
exit 4
|
||||
```
|
||||
|
||||
That is a build-retention failure, not a reason to reset the region. Re-run the
|
||||
supported sync hook using the current rendered manifest.
|
||||
|
||||
## 7. Verify the region
|
||||
|
||||
All of these must be true:
|
||||
|
||||
```bash
|
||||
kubectl -n evolution-dev get pod -l app.kubernetes.io/name=evo-region
|
||||
kubectl -n evolution-dev logs deployment/evo-region --tail=50
|
||||
kubectl -n argocd get application aevolution-dev
|
||||
```
|
||||
|
||||
Expected runtime indicators include:
|
||||
|
||||
```text
|
||||
phase running
|
||||
ready 1
|
||||
```
|
||||
|
||||
Also verify `Ready=True`, zero new restarts, and Argo `Synced / Healthy`. `Running`
|
||||
alone is not sufficient.
|
||||
|
||||
## 8. Probe and acceptance
|
||||
|
||||
For an authenticated M10 probe, retain the matching operator bundle until all tests
|
||||
finish:
|
||||
|
||||
- `dev-ca.pem`
|
||||
- `issuer.key`
|
||||
|
||||
The normal dev overlay has `REGION_INPUTS` empty, so a probe can authenticate and
|
||||
receive an intention receipt without producing a simulated outcome. Full durable
|
||||
outcome testing requires the repository-supported fault-drill/synthetic-input path;
|
||||
do not enable it with an ad-hoc live environment change.
|
||||
|
||||
## Recovery checklist
|
||||
|
||||
If the region enters CrashLoopBackOff:
|
||||
|
||||
1. Capture the current image digest, `REGION_BUILD`, Argo revision, previous logs, and
|
||||
Job image.
|
||||
2. If logs show `quarantined epoch=0` with zero store failures, compare the build
|
||||
against the retained-build Job/database path.
|
||||
3. If the provision Job is stale, perform a full Argo sync so the sync-wave 0 hook runs.
|
||||
4. Never delete the database, PVC, object store, or committed region history.
|
||||
5. Do not force-sync only the Deployment while the provision hook is pending.
|
||||
Reference in New Issue
Block a user