Skip to content

ci: fix a racy resource test, bound the native-image test step, skip linker suites there - #639

Merged
oyvindberg merged 3 commits into
masterfrom
fix-flaky-test-and-win-timeout
Aug 2, 2026
Merged

ci: fix a racy resource test, bound the native-image test step, skip linker suites there#639
oyvindberg merged 3 commits into
masterfrom
fix-flaky-test-and-win-timeout

Conversation

@oyvindberg

@oyvindberg oyvindberg commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Three CI fixes that came out of debugging the failures on #637 and the Windows telemetry. None was caused by that PR, and none is affected by the M11 bootstrap bump in #638.

1. MachineResourcesTest was racy by construction

The compile-deadlock regression fills the governor with eight forks, retunes the budget below what they hold so free memory goes negative, then asserts a zero-memory compile is still admitted. But:

hold <- CountDownLatch[IO](1)   // ← eight forks release it
forks <- (1 to 8).toList.parTraverse(i => m.reserve(...).use(_ => hold.release *> release.await).start)
_ <- hold.await                  // returns after the FIRST fork
_ <- m.retuneMemoryBudget(15000) // races the other seven

So the over-commit the test exists to construct only sometimes existed:

forks reserved used free result
5 12800 +2200 2200 was not less than 0 ← what CI hit
6 15360 -360
8 20480 -5480 ✅ (intended)

15000 - 5*2560 = 2200 is exactly the reported failure. Counting 8 makes the setup deterministic. Every other latch in the file already counts its own fan-out; this was the only one that didn't.

2. The native-image test step bounds itself

timeout-minutes is a GitHub-side reap of the step's process tree, and on windows-latest it does not work. On run 30697246865 the step declared 20 minutes, ran 45m07s, blew the job's own 45-minute ceiling at 55m00s, and the runner was destroyed with both telemetry steps still pending — no bsp-diagnostics-windows-latest artifact at all. That is exactly what #627 added the step cap to prevent.

run-bounded.sh kills the command itself and exits 124, failing the step instead of the job, which keeps the if: always() collection alive.

  • Not timeout(1) — GNU coreutils, absent on macOS, two of the five arches here. Done in bash so there is one code path per OS.
  • Invoked as bash <path>, matching collect-bsp-diagnostics.sh, since the exec bit doesn't survive a Windows checkout reliably.
  • Bound sized from a real run: windows did this step in 13m14s (native-image itself was only 4m57s, so the step is the cost). 18m would have been 1.36x headroom, tighter than the >1.5x this cap was sized for — and runner variance is larger than that, with the same macos native-image job taking 18m14s and 24m28s on two runs. So 20m, backstop 25 — above the 22m our bounds can total, and clear of the job's 45 so the job cap stays the last thing to fire.

3. The platform-linker suites are tagged slow

Nine suites in bleep-bsp-tests drive a real linker (Scala.js, Scala Native, Kotlin/JS, Kotlin/Native): compile a source, then link an artifact — for Scala Native, NIR → LLVM IR → clang → executable at Debug, ReleaseFast and LTO. That time is clang's.

Measured from CI telemetry, and the set is the union across platforms because the figures move a lot:

Suite Linux Windows
ScalaNativeAdvancedIntegrationTest 63.5s 139.1s
ScalaNativeAdvancedTestIntegrationTest 75.6s 132.2s
ScalaNativeAdvancedLinkIntegrationTest 69.4s 73.7s
LinkExecutorIntegrationTest 99.5s 18.5s
KotlinNativeAdvancedIntegrationTest 78.6s 1.7s (cancelled)
KotlinNativeIntegrationTest 24.8s 53.6s
ScalaJs / KotlinJs Advanced (×3) 27.6s 40.8s

Tagged slow — the tag bleep-tests already uses for its **IT suites — rather than a new one. It means the same thing and is already wired where it's wanted, so no workflow change was needed for the exclusion: the native-image jobs already pass --exclude-tag slow.

The build job is deliberately untouched. It is the canonical full-suite gate, so every linker is still exercised once per CI run instead of five times, and its 36 **IT suites keep running. Excluding slow there would have cost 17.3 of its 25.4 minutes.

Net: native-image suite time 18.3 → 10.6 min per arch, build unchanged.

Listed by name, not by glob — the cost is a property of what a suite does, not what it's called, and the cheap suites in the same packages mention the same link types. Every entry cross-checked against suite names in two CI telemetry artifacts: no typos, no stale entries.

Verification

  • run-bounded.sh: returns 0 on success, propagates a non-zero exit (7 → 7), reports 124 on expiry
  • MachineResourcesTest: 19 passed, 0 failed — three consecutive runs
  • tag registration: bleep reports Expected slow, confirming both projects feed one tag namespace

Note this does not make the Windows hang not happen — it makes it fail fast and leave the telemetry behind, so the next occurrence can actually be diagnosed.

🤖 Generated with Claude Code

@oyvindberg
oyvindberg force-pushed the fix-flaky-test-and-win-timeout branch from 1dc1403 to 95df709 Compare August 2, 2026 13:52
@oyvindberg oyvindberg changed the title ci: fix a racy resource test, and bound the native-image test step ourselves ci: fix a racy resource test, bound the native-image test step, skip linker suites there Aug 2, 2026
oyvindberg and others added 3 commits August 2, 2026 16:15
…rselves

Two unrelated CI failures on #637, neither caused by that PR.

**MachineResourcesTest was racy by construction.** The compile-deadlock
regression fills the governor with eight forks, retunes the budget below
what they hold so free memory goes negative, and asserts a zero-memory
compile is still admitted. But `hold` counted 1 while eight forks release
it, so `hold.await` returned as soon as the FIRST fork had reserved and
the retune raced the other seven.

The over-commit the test exists to construct therefore only sometimes
existed. At 8 forks free memory is -5480 and it passes; it passes anywhere
from 6 up; CI caught it at 5, reporting `2200 was not less than 0`, which
is exactly 15000 - 5*2560. Counting 8 makes the setup deterministic.

Every other latch in the file already counts its own fan-out (3 tasks -> 3,
2 -> 2); this was the one that did not.

**The native-image test step now bounds itself.** `timeout-minutes` is a
GitHub-side reap of the step's process tree, and on windows-latest that
does not work. Run 30697246865 declared 20 minutes, ran 45m07s, blew the
job's own 45-minute ceiling at 55m00s, and the runner was destroyed with
both telemetry steps still pending — so the one hang worth diagnosing
produced no diagnostics at all. That is precisely what #627 added the step
cap to prevent, so the cap is not sufficient on its own, and it fails in
the worst possible way.

`run-bounded.sh` kills the command itself and exits 124, which fails the
step rather than the job and keeps the `if: always()` collection alive.
Not `timeout(1)`: that is GNU coreutils and absent on macOS, two of the
five arches here, so it is done in bash to keep one code path per OS — the
same reason these steps unified on `shell: bash`. Invoked as `bash <path>`
like collect-bsp-diagnostics.sh, since the exec bit does not survive a
Windows checkout reliably.

Bounds are 18m for the suite (~1.4x the slowest healthy run, 12m52s on
windows) and 2m for selftest. `timeout-minutes` stays as a backstop, now
at 25 — above our own bound rather than below it, and still clearing the
job's 45 on macos-arm, which reaches this step at ~17m.

Verified: run-bounded.sh returns 0 on success, propagates a non-zero exit,
and reports 124 on expiry. MachineResourcesTest 19 passed, three runs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
…d comment

windows-latest passed on this branch in 13m14s for the test step, with the
native image itself at 4m57s — so the step, not native-image, is the cost, and
the healthy figure is a touch above the 12m52s the comment recorded.

That makes the 18m I first picked only 1.36x headroom, tighter than the >1.5x
this cap was originally sized for. Only the enforcement was broken; the number
was not, so tightening it too was an unforced change. Runner variance is also
larger than it looks — macos-latest built the native image in 18m14s on one run
and 24m28s on another, 34% apart — and a bound whose purpose is catching a
45-minute hang loses nothing by clearing a slow healthy run comfortably.

So 20m, and the step backstop moves to 25: above the 22m our two bounds can add
up to, and still clear of the job's 45 (macos-arm reaches this step at ~17m, so
25 lands at 42). The ordering matters — if the job cap fires first we are back
to a destroyed runner and no telemetry, which is the whole failure being fixed.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Nine suites in bleep-bsp-tests drive a real platform linker — Scala.js, Scala
Native, Kotlin/JS, Kotlin/Native. Each compiles a source and then links an
artifact; for Scala Native that is NIR -> LLVM IR -> clang -> executable, at
Debug, ReleaseFast and LTO. The time is clang's, and no build tool makes it
cheaper.

Measured from CI telemetry rather than guessed: 7.7 of 18.3 minutes of suite
time on windows, 7.3 of 25.4 on linux. The individual figures move a lot by
platform, which is why the set is the union rather than one arch's worst
offenders: LinkExecutor is 99.5s on linux against 18.5s on windows, and
KotlinNativeAdvanced is 78.6s on linux while windows cancels it outright for
want of an aarch64-capable Konan.

Tagged `slow`, the tag bleep-tests already puts on its `**IT` suites, rather
than a new one. It means the same thing and is already wired exactly where it
is wanted — the native-image jobs pass `--exclude-tag slow`, and those jobs
exist to prove the produced binary runs, not to re-exercise the test surface.
No workflow change was needed for the exclusion at all.

The `build` job is deliberately left alone. It is the canonical full-suite gate
and does not exclude `slow`, so every linker is still exercised once per CI run
instead of five times, and its 36 `**IT` suites keep running too. Excluding
`slow` there would have taken 17.3 of its 25.4 minutes — the linkers plus every
docs-snippet, KSP and tutorial test — which is a much larger trade than the one
being made here.

Listed by name, not by a glob: the cost is a property of what a suite does, not
of what it is called, and the cheap suites in the same packages mention the same
link types. Every entry was cross-checked against the suite names in two CI
telemetry artifacts, so there are no typos and no stale entries.

Also sizes the test-step bound from this run rather than the old comment.
windows did the step in 13m14s (native-image itself was 4m57s, so the step is
the cost), making the 18m first picked only 1.36x headroom — tighter than the
>1.5x this cap was sized for, and runner variance is larger than that: the same
macos native-image job took 18m14s and 24m28s on two runs. So 20m, with the step
backstop at 25 — above the 22m our bounds can total, and clear of the job's 45
so the job cap stays the last thing to fire rather than the first.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@oyvindberg
oyvindberg force-pushed the fix-flaky-test-and-win-timeout branch from 95df709 to ec53dba Compare August 2, 2026 14:15
@oyvindberg
oyvindberg merged commit fa273b5 into master Aug 2, 2026
9 checks passed
@oyvindberg
oyvindberg deleted the fix-flaky-test-and-win-timeout branch August 2, 2026 14:36
oyvindberg added a commit that referenced this pull request Aug 2, 2026
…641)

The bound added in #639 did not work: run 30752489661 hung the same 55 minutes
with the same missing bsp-diagnostics-windows-latest artifact, despite
run-bounded.sh being in effect. The reason is that it bounded the wrong thing.

Reproduced locally, with a fixture that spawns a daemon which escapes a process
group kill and then hangs, both scripts given a 5-8s bound:

  old (child inherits stdout):  still blocked after 20s
  new (child writes to a file): finished in 12s, WITH the survivor still alive

So killing the process was never the missing piece. GitHub waits for the step's
output PIPES to close, not for the shell to exit — and `bleep` spawns a BSP
daemon which spawns forked test JVMs, all inheriting stdout. Kill the child and
the pipe is still held by a grandchild, so a 20-minute bound produced a
45-minute step and the runner was destroyed with the telemetry steps pending.

Two changes, and the second is the one that matters:

  1. Kill the process GROUP (`set -m` so the child leads its own), plus
     `taskkill //F //T` on Windows, where processes are not in POSIX groups.
  2. Redirect the child to a FILE. Descendants then inherit the file, not the
     pipe. Only `tail` holds the pipe, and `tail` is ours to kill — so the step
     ends even when something survives, which is precisely the case that hung.

On timeout the last 200 lines are printed in a collapsed group, since the whole
point is to still have the evidence.

Verified: exit 0 on success, 7 propagated from the child, 124 on expiry,
grandchildren reaped, streaming intact during a normal run, and the pipe closed
with a deliberate survivor left running.

Also fixes the second flaky cancellation test, `Kotlin: fiber cancellation
interrupts compilation`, which failed the same run with
`TimeoutException: 30 seconds`. These compile through `IO.interruptible`, whose
cancellation interrupts the thread and then WAITS for the block to return —
and kotlinc does not promise to notice. So `fiber.cancel` can take as long as
the whole compile, which each of these tests already accepts explicitly
("completed before cancellation took effect"). The bound only rules out waiting
forever, and 30s did not clear a full compile of a deliberately huge generated
source on a contended runner: the suite is 3.8s healthy, so this was an 8x
outlier, the shape of "never reached an interruptible point" rather than of a
slightly tight bound. Named `CancellationHangGuard`, 120s, applied to all four
sites — they share the pattern, and the other three would fail next time a
runner is busy.

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant