ci: fix a racy resource test, bound the native-image test step, skip linker suites there - #639
Merged
Merged
Conversation
oyvindberg
force-pushed
the
fix-flaky-test-and-win-timeout
branch
from
August 2, 2026 13:52
1dc1403 to
95df709
Compare
…rselves Two unrelated CI failures on #637, neither caused by that PR. **MachineResourcesTest was racy by construction.** The compile-deadlock regression fills the governor with eight forks, retunes the budget below what they hold so free memory goes negative, and asserts a zero-memory compile is still admitted. But `hold` counted 1 while eight forks release it, so `hold.await` returned as soon as the FIRST fork had reserved and the retune raced the other seven. The over-commit the test exists to construct therefore only sometimes existed. At 8 forks free memory is -5480 and it passes; it passes anywhere from 6 up; CI caught it at 5, reporting `2200 was not less than 0`, which is exactly 15000 - 5*2560. Counting 8 makes the setup deterministic. Every other latch in the file already counts its own fan-out (3 tasks -> 3, 2 -> 2); this was the one that did not. **The native-image test step now bounds itself.** `timeout-minutes` is a GitHub-side reap of the step's process tree, and on windows-latest that does not work. Run 30697246865 declared 20 minutes, ran 45m07s, blew the job's own 45-minute ceiling at 55m00s, and the runner was destroyed with both telemetry steps still pending — so the one hang worth diagnosing produced no diagnostics at all. That is precisely what #627 added the step cap to prevent, so the cap is not sufficient on its own, and it fails in the worst possible way. `run-bounded.sh` kills the command itself and exits 124, which fails the step rather than the job and keeps the `if: always()` collection alive. Not `timeout(1)`: that is GNU coreutils and absent on macOS, two of the five arches here, so it is done in bash to keep one code path per OS — the same reason these steps unified on `shell: bash`. Invoked as `bash <path>` like collect-bsp-diagnostics.sh, since the exec bit does not survive a Windows checkout reliably. Bounds are 18m for the suite (~1.4x the slowest healthy run, 12m52s on windows) and 2m for selftest. `timeout-minutes` stays as a backstop, now at 25 — above our own bound rather than below it, and still clearing the job's 45 on macos-arm, which reaches this step at ~17m. Verified: run-bounded.sh returns 0 on success, propagates a non-zero exit, and reports 124 on expiry. MachineResourcesTest 19 passed, three runs. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…d comment windows-latest passed on this branch in 13m14s for the test step, with the native image itself at 4m57s — so the step, not native-image, is the cost, and the healthy figure is a touch above the 12m52s the comment recorded. That makes the 18m I first picked only 1.36x headroom, tighter than the >1.5x this cap was originally sized for. Only the enforcement was broken; the number was not, so tightening it too was an unforced change. Runner variance is also larger than it looks — macos-latest built the native image in 18m14s on one run and 24m28s on another, 34% apart — and a bound whose purpose is catching a 45-minute hang loses nothing by clearing a slow healthy run comfortably. So 20m, and the step backstop moves to 25: above the 22m our two bounds can add up to, and still clear of the job's 45 (macos-arm reaches this step at ~17m, so 25 lands at 42). The ordering matters — if the job cap fires first we are back to a destroyed runner and no telemetry, which is the whole failure being fixed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Nine suites in bleep-bsp-tests drive a real platform linker — Scala.js, Scala Native, Kotlin/JS, Kotlin/Native. Each compiles a source and then links an artifact; for Scala Native that is NIR -> LLVM IR -> clang -> executable, at Debug, ReleaseFast and LTO. The time is clang's, and no build tool makes it cheaper. Measured from CI telemetry rather than guessed: 7.7 of 18.3 minutes of suite time on windows, 7.3 of 25.4 on linux. The individual figures move a lot by platform, which is why the set is the union rather than one arch's worst offenders: LinkExecutor is 99.5s on linux against 18.5s on windows, and KotlinNativeAdvanced is 78.6s on linux while windows cancels it outright for want of an aarch64-capable Konan. Tagged `slow`, the tag bleep-tests already puts on its `**IT` suites, rather than a new one. It means the same thing and is already wired exactly where it is wanted — the native-image jobs pass `--exclude-tag slow`, and those jobs exist to prove the produced binary runs, not to re-exercise the test surface. No workflow change was needed for the exclusion at all. The `build` job is deliberately left alone. It is the canonical full-suite gate and does not exclude `slow`, so every linker is still exercised once per CI run instead of five times, and its 36 `**IT` suites keep running too. Excluding `slow` there would have taken 17.3 of its 25.4 minutes — the linkers plus every docs-snippet, KSP and tutorial test — which is a much larger trade than the one being made here. Listed by name, not by a glob: the cost is a property of what a suite does, not of what it is called, and the cheap suites in the same packages mention the same link types. Every entry was cross-checked against the suite names in two CI telemetry artifacts, so there are no typos and no stale entries. Also sizes the test-step bound from this run rather than the old comment. windows did the step in 13m14s (native-image itself was 4m57s, so the step is the cost), making the 18m first picked only 1.36x headroom — tighter than the >1.5x this cap was sized for, and runner variance is larger than that: the same macos native-image job took 18m14s and 24m28s on two runs. So 20m, with the step backstop at 25 — above the 22m our bounds can total, and clear of the job's 45 so the job cap stays the last thing to fire rather than the first. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
oyvindberg
force-pushed
the
fix-flaky-test-and-win-timeout
branch
from
August 2, 2026 14:15
95df709 to
ec53dba
Compare
oyvindberg
added a commit
that referenced
this pull request
Aug 2, 2026
…641) The bound added in #639 did not work: run 30752489661 hung the same 55 minutes with the same missing bsp-diagnostics-windows-latest artifact, despite run-bounded.sh being in effect. The reason is that it bounded the wrong thing. Reproduced locally, with a fixture that spawns a daemon which escapes a process group kill and then hangs, both scripts given a 5-8s bound: old (child inherits stdout): still blocked after 20s new (child writes to a file): finished in 12s, WITH the survivor still alive So killing the process was never the missing piece. GitHub waits for the step's output PIPES to close, not for the shell to exit — and `bleep` spawns a BSP daemon which spawns forked test JVMs, all inheriting stdout. Kill the child and the pipe is still held by a grandchild, so a 20-minute bound produced a 45-minute step and the runner was destroyed with the telemetry steps pending. Two changes, and the second is the one that matters: 1. Kill the process GROUP (`set -m` so the child leads its own), plus `taskkill //F //T` on Windows, where processes are not in POSIX groups. 2. Redirect the child to a FILE. Descendants then inherit the file, not the pipe. Only `tail` holds the pipe, and `tail` is ours to kill — so the step ends even when something survives, which is precisely the case that hung. On timeout the last 200 lines are printed in a collapsed group, since the whole point is to still have the evidence. Verified: exit 0 on success, 7 propagated from the child, 124 on expiry, grandchildren reaped, streaming intact during a normal run, and the pipe closed with a deliberate survivor left running. Also fixes the second flaky cancellation test, `Kotlin: fiber cancellation interrupts compilation`, which failed the same run with `TimeoutException: 30 seconds`. These compile through `IO.interruptible`, whose cancellation interrupts the thread and then WAITS for the block to return — and kotlinc does not promise to notice. So `fiber.cancel` can take as long as the whole compile, which each of these tests already accepts explicitly ("completed before cancellation took effect"). The bound only rules out waiting forever, and 30s did not clear a full compile of a deliberately huge generated source on a contended runner: the suite is 3.8s healthy, so this was an 8x outlier, the shape of "never reached an interruptible point" rather than of a slightly tight bound. Named `CancellationHangGuard`, 120s, applied to all four sites — they share the pattern, and the other three would fail next time a runner is busy. Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three CI fixes that came out of debugging the failures on #637 and the Windows telemetry. None was caused by that PR, and none is affected by the M11 bootstrap bump in #638.
1.
MachineResourcesTestwas racy by constructionThe compile-deadlock regression fills the governor with eight forks, retunes the budget below what they hold so free memory goes negative, then asserts a zero-memory compile is still admitted. But:
So the over-commit the test exists to construct only sometimes existed:
2200 was not less than 0← what CI hit15000 - 5*2560 = 2200is exactly the reported failure. Counting 8 makes the setup deterministic. Every other latch in the file already counts its own fan-out; this was the only one that didn't.2. The native-image test step bounds itself
timeout-minutesis a GitHub-side reap of the step's process tree, and on windows-latest it does not work. On run 30697246865 the step declared 20 minutes, ran 45m07s, blew the job's own 45-minute ceiling at 55m00s, and the runner was destroyed with both telemetry steps still pending — nobsp-diagnostics-windows-latestartifact at all. That is exactly what #627 added the step cap to prevent.run-bounded.shkills the command itself and exits 124, failing the step instead of the job, which keeps theif: always()collection alive.timeout(1)— GNU coreutils, absent on macOS, two of the five arches here. Done in bash so there is one code path per OS.bash <path>, matchingcollect-bsp-diagnostics.sh, since the exec bit doesn't survive a Windows checkout reliably.3. The platform-linker suites are tagged
slowNine suites in
bleep-bsp-testsdrive a real linker (Scala.js, Scala Native, Kotlin/JS, Kotlin/Native): compile a source, then link an artifact — for Scala Native, NIR → LLVM IR → clang → executable at Debug, ReleaseFast and LTO. That time is clang's.Measured from CI telemetry, and the set is the union across platforms because the figures move a lot:
Tagged
slow— the tagbleep-testsalready uses for its**ITsuites — rather than a new one. It means the same thing and is already wired where it's wanted, so no workflow change was needed for the exclusion: the native-image jobs already pass--exclude-tag slow.The
buildjob is deliberately untouched. It is the canonical full-suite gate, so every linker is still exercised once per CI run instead of five times, and its 36**ITsuites keep running. Excludingslowthere would have cost 17.3 of its 25.4 minutes.Net: native-image suite time 18.3 → 10.6 min per arch,
buildunchanged.Listed by name, not by glob — the cost is a property of what a suite does, not what it's called, and the cheap suites in the same packages mention the same link types. Every entry cross-checked against suite names in two CI telemetry artifacts: no typos, no stale entries.
Verification
run-bounded.sh: returns 0 on success, propagates a non-zero exit (7 → 7), reports 124 on expiryMachineResourcesTest: 19 passed, 0 failed — three consecutive runsExpected slow, confirming both projects feed one tag namespaceNote this does not make the Windows hang not happen — it makes it fail fast and leave the telemetry behind, so the next occurrence can actually be diagnosed.
🤖 Generated with Claude Code