Same failure signature as the boss test fixed last round ("Test
timeout of 45000ms exceeded", the whole test's hard deadline, not a
specific assertion), this time landing on this test instead -- not
caused by the Helm chart fix or the boss test.slow() change, both of
which touch entirely disjoint code (a Kubernetes manifest template
never exercised by npm run test:e2e, and a different test block).
Consistent with a CI runner that's intermittently slow enough that
whichever test draws the short straw on a given run can blow its
budget before its own poll timeout gets to report cleanly -- this is
the most elaborate test here (WebGL pixel readback, model diagnostics,
a full reload/reconnect cycle), so the most likely next candidate.
Verified: full local e2e suite (7/7) still passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Latest CI failure was a different signature than anything the retry
fix addresses: "Test timeout of 45000ms exceeded" -- the whole test's
hard deadline, not the specific 20s poll assertion (which has its own
budget and, per the retry fix, gives up cleanly well within it). That
means cumulative time for enter() + awakening the boss + closing
settings is eating far enough into the config's 45s per-test timeout
on a slow run that the poll never gets the chance to report on its own
terms before the whole test gets killed.
test.slow() triples this one test's timeout (45s -> 135s) rather than
inflating the config's global timeout for every test. Confirmed the
retry-logic fix from two rounds ago (which passed on CI last round) is
untouched here -- this is purely a timeout-budget change layered on
top of it.
Verified: full local e2e suite (7/7) still passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Re-applying the independently-diagnosed Helm bug fix that got reverted
along with everything else a few rounds back (flagged at the time that
CI would hit this again the moment it wasn't reapplied -- it just did).
helm lint deploy/chart (the default 5-world values.yaml) fails:
templates/worlds.yaml: unable to parse YAML: error converting YAML
to JSON: yaml: line 24: did not find expected key
Root cause (confirmed earlier via `helm template --debug` on the raw,
pre-parse output): the per-world loop's `{{- $host := ... -}}` line has
a leading whitespace-trim that eats the one newline separating one
world's rendered manifests from the next world's "---" separator, so
consecutive iterations run together as "...port: { name: http } }---
apiVersion: apps/v1" with no newline -- invalid YAML. values-srs.yaml
only has one world, so the loop never repeats and never trips this;
the default values.yaml has five.
Fix: drop the leading trim (`{{-` -> `{{`) on just that line, so the
one separating newline survives.
Verified with the same helm v4.2.4 CI uses: `helm lint`/`helm template`
clean against both deploy/chart's default values (5 worlds) and
-f deploy/values-srs.yaml (1 world); rendered manifests for all 5
worlds look correct.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Testing this in isolation from the anti-throttle Chromium flags and
the DB-lock change (reverted separately, ruled out on its own): the
single fire-and-forget gather/attack action has no recovery if the one
evaluated frame doesn't line up with the authoritative server position
-- confirmed by live-instrumenting the unpatched code locally and
watching it hang. This reinstates the fix from the earlier round, with
both bugs found via live diagnostic testing already fixed going in:
- Retries at a conservative cadence (~0.3s, well under the intent rate
limit) instead of firing once and giving up forever.
- Gives up on a target only once an actual progress signal (target HP
for attack, held-item count for gather/loot) has stopped moving for
a wall-clock budget -- not "hasn't succeeded since the click" (which
wrongly aborts mid-fight during a normal cooldown lull) and not a
frame/attempt count (which balloons into tens of real seconds on a
tab with sparse rendered frames).
- The dedicated boss-attack button now also resets the progress clock;
it used to bypass action() entirely and read an uninitialized clock,
immediately abandoning the boss for the nearest mob on click.
Deliberately NOT re-adding the Chromium anti-throttle flags this round
-- keeping this an isolated variable after the last round's session
crash coincided with adding them.
Also bumped one unrelated, independently-diagnosed timeout: the
post-death new-character join in portal.spec.ts was polling on
Playwright's default 5s timeout for a fresh join's first snapshot,
with no connection to the retry logic above.
Verified: tsc/build clean, three full local e2e runs (7/7 each).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Testing one specific, targeted hypothesis rather than another blind
patch: the portal merge added an unconditional per-second DB query
(controls(), checking for pending admin commands) directly inside the
world tick loop's s.mu-locked section -- the same lock every websocket
intent, every HTTP handler, and every snapshot broadcast to every
connected player depends on. That runs forever, for every world,
whether or not the admin feature is ever used.
Verified the actual game logic isn't broken (a live-instrumented run
of the unpatched single-shot gather/attack code landed its first hit
immediately and correctly), so the e2e flakiness has to be about
something making the server slower under load rather than a logic
defect. This is the one change in the merge positioned to tax *every*
player's snapshot/intent latency, every second, permanently -- not
just an occasional per-request cost like the other additions.
controls() now runs on its own ticker, independent of the simulation
tick, and only takes s.mu at all if there's an actual pending command
to apply (the DB read itself needs no lock -- world.ID is immutable
after creation). The idle case, which is the overwhelming majority of
ticks, no longer touches s.mu for this at all.
Verified: go build/vet/test clean, and the full local e2e suite (7/7,
both spec files, including the admin-command test that exercises
controls() directly) passes across two runs.
If this doesn't change CI's result, it should be reverted and ruled
out -- the next candidate would be the new /worlds/<id> routing
rewrite or the added per-request ActiveCharacter DB lookups.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reverts the working tree for these six files to their state at 7c5cb78
(the control-portal merge, reconciled with the auto-deploy digest bump)
-- undoing c5b8361, 74ff146, b1a07b7, db3e5a8, 0bb3e4b, and 9b226f7.
Those six commits were reactive fixes chasing CI failures one at a
time; each one changed the failure mode without ever reaching a fully
green run, and the last one coincided with the browser session itself
crashing mid-test rather than a logic assertion failing. Reverting to
a clean, known point (right after the merge, before any of that
patching) rather than continuing to layer fixes on an unverified base.
Note: this reintroduces a real, independently-confirmed bug fixed by
db3e5a8 -- deploy/chart/templates/worlds.yaml's per-world loop is
missing a newline between iterations (a leading whitespace-trim on the
$host template variable eats it), which breaks `helm lint` /
`helm template` for any values file with more than one world. That
will fail again on the next push if it isn't re-applied.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found this by reproducing locally (installed a Go toolchain + helm
earlier in this session; this time added Chromium anti-throttling
flags and instrumented walkTo/pendingAction/keys onto window.__aeternalis
to watch it live) rather than guessing from CI logs again: the
"Target Harbinger" button sets pendingAction/walkTo directly, bypassing
action() entirely -- so the give-up budget from the last commit, which
only got initialized inside action(), read Date.now() against its
initial value of 0 on the very first tick after clicking. That's
always "expired" by billions of milliseconds, so it gave up on "boss"
immediately, blacklisted it, and reselected the nearest sentinel mob
instead -- explaining every "boss out of reach" / zero-progress boss
failure we've chased across the last several rounds. Confirmed via a
throwaway diagnostic test logging position/target every second: the
character walked calmly away from the boss it started standing on top
of, chasing first one sentinel then another.
Fixed the immediate bug (also set the budget's clock in the button's
handler), then found a second one the same repro immediately exposed:
attack against a live target is *mostly* cooldown rejections between
the hits that land, so "no progress since the initial click" was still
wrong -- it gave up mid-fight after ~10 successful hits during a lull.
Replaced the give-up condition with an actual progress signal (target
HP for attack, held-item count for gather/loot); it now only gives up
once that has genuinely stopped moving, confirmed via the same
diagnostic: 10 clean hits (1200 -> 1020 HP) holding position, matching
the server's own attack cooldown.
Also added the standard Chromium flags
(--disable-background-timer-throttling etc.) to the test browser
launch: the CI logs' 2-11s gaps between retries (instead of the
intended ~0.3s) are the classic signature of requestAnimationFrame
throttling down to ~1Hz for a backgrounded/occluded headless tab,
which independently made every frame-driven client action crawl. Left
walkTo/pendingAction/keys exposed on window.__aeternalis since they
were useful for this and will be again.
Verified: tsc/build clean, and three full local e2e runs (7/7 each)
all show the boss test landing clean periodic hits via "attack
cooldown" rejections between them, never wandering to a mob.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The unbounded retry from the last commit was a real regression: CI's
gather test picked "arrival:ore", which an earlier sequential test's
character had already depleted in this same long-lived shared world
(a real ~120s respawn wait, not a stray-frame timing race), and the
retry loop hammered that dead target for the entire 45s test timeout
instead of failing cleanly at the 20s assertion -- which then looks to
have left the world/character state in a bad spot for the next test
("character cannot act" on a brand-new character in the boss test
right after).
Now each target gets at most 8 retries (~2.4s) before the client gives
up on it, marks it briefly avoided, and re-runs the same target search
that action() already does -- so a gather walks to the next-nearest
tree/ore/herb instead of retrying a resource that won't respawn for two
minutes, and a stuck attack at least fails fast instead of consuming
the whole test budget. Boss still has no substitute target, so it
correctly falls through to "No target nearby" after giving up, rather
than retrying forever.
Verified: tsc/build clean, and two full local e2e runs (7/7 both
times) -- one incidentally still showed the depleted-arrival-resource
case, resolving in the normal 2-3 retries once the retry itself no
longer needed to fight a stale position, confirming the mechanism this
was meant to catch is bounded now rather than open-ended.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
helm lint deploy/chart (the default 5-world values.yaml) failed:
templates/worlds.yaml: unable to parse YAML: error converting YAML
to JSON: yaml: line 24: did not find expected key
Root cause, confirmed by diffing raw --debug template output against
the pre-merge version: the old per-world loop started with
`{{- range .Values.worlds }}` (no trailing trim) directly followed by
a bare "---", so every iteration began with a preserved leading
newline that separated it from the previous iteration's output (which
itself ends with no trailing newline, closed by a flush-left
`{{- end }}`). The portal merge inserted two new per-iteration
`{{- $host := ... -}}` / `{{- $primary := ... -}}` assignments right
after the range statement -- both leading-trimmed, so the first one
ate that same newline. With it gone, iteration N's last line
("...port: { name: http } }") ran directly into iteration N+1's
"---apiVersion: apps/v1" with zero separator, which is what actually
produced the YAML parse error (confirmed via `helm template --debug`:
the raw output literally showed "}---\napiVersion:" glued together;
values-srs.yaml's single-world value set never exercises the loop a
second time, so it never tripped this).
Fix: drop the leading trim (`{{-` -> `{{`) on just the $host
assignment, restoring the one preserved newline per iteration. Verified
via `helm lint`/`helm template` with both deploy/chart's default values
(5 worlds) and -f deploy/values-srs.yaml (1 world) -- both clean now,
and the rendered Deployment/Service/Ingress manifests for all 5 worlds
look correct (each gets its own /worlds/<id> path; the shared "/" path
still applies to every world when no single portal.host is set, and
only to the primary world once one is).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>