Skip to content

[fixtures] De-flake the dev-registry tests on Windows by making tail teardown safe - #15017

Draft
petebacondarwin wants to merge 2 commits into
mainfrom
fix/dev-registry-teardown-order
Draft

[fixtures] De-flake the dev-registry tests on Windows by making tail teardown safe#15017
petebacondarwin wants to merge 2 commits into
mainfrom
fix/dev-registry-teardown-order

Conversation

@petebacondarwin

@petebacondarwin petebacondarwin commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The Tests (Windows, fixtures) required check has been failing roughly half the time on fixtures/dev-registry, almost always as Error: Test timed out in 50000ms in one of the vite dev <-> vite dev tests.

The failure is a workerd abort during test teardown. When a dev session is killed while another running session is forwarding tail events to it, the surviving session's workerd calls std::terminate — filed upstream as cloudflare/workerd#6913. Miniflare then starts a replacement runtime, the Vite plugin restarts the dev server to rebuild its module-runner sockets, and the ~30s of churn lands on whichever test happens to still be running. That's why the timeout kept moving between tests and never pointed at the code it was blaming.

This PR does not fix the abort — that belongs in workerd. It removes the shape that triggers it from the fixture.

The fixture made the crash easy to hit

tail_consumers formed two cycles:

  • exported-handler <-> worker-entrypoint-with-assets
  • worker-entrypoint <-> exported-handler-with-assets

So most tests carried a live tail edge whether or not they were testing tail handlers, and because the edges were cyclic there was no shutdown order that could keep every producer shorter-lived than its consumer. Whichever session died first left the other one forwarding to a dead peer.

Changes

Tail relationships are now a one-directional chainworker-entrypoint -> exported-handler-with-assets -> exported-handler — and the incidental edges on exported-handler, worker-entrypoint-with-assets and external-durable-object are gone. Reversing the middle edge was enough to break both cycles, so no new config files were needed. Tail edges now exist only in the tests that assert on them.

The three bidirectional tail tests are split into one test per direction, so each can start its tail consumer first and its producer second. Vitest tears sessions down in LIFO order, so the producer is always killed while its consumer is still alive. The ordering is expressed through a documented startTailPair() helper rather than a comment, so a future test can't quietly get it backwards. All existing tail assertions are preserved, including the [worker-entrypoint] log prefixes that only appear for multi-worker Wrangler sessions.

Two supporting fixes, both of which let a session watch a peer disappear mid-teardown:

  • The devRegistryPath fixture deleted the registry directory in its teardown. Vitest runs fixture teardown before onTestFinished callbacks (verified against @vitest/[email protected]: afterEach -> fixture teardown -> onTestFinished LIFO), so the registry was being pulled out from under every session while they were all still running. Registering the removal as an onTestFinished callback during fixture setup makes it the first registered and therefore the last to run.
  • runWranglerDev's stop() resolved once the kill signal had been delivered — on Windows, once taskkill exited — not once the process was gone. It now waits for the actual exit, bounded at 10s, so sequential teardown really is sequential.

Re-enables the tests on Windows

#15018 landed while this was in review and skipped the entire suite on Windows as a temporary measure "until the underlying issue is fixed". This PR fixes it, so the second commit reverts that skip. Windows is the only platform the flake ever appeared on, so leaving it skipped there would mean the suite no longer guards the thing it exists to guard.

Verification

  • Audited all 30 tests for the dangerous shape (a running producer whose assets-enabled tail consumer is killed first). It now occurs in exactly three tests, all of which start the producer last. The previously-failing test carries no live tail edge at all.
  • 7 local runs of the full fixture suite. Suite cost goes from ~58s to ~70s for the three extra dev-session pairs.
  • Type-check output is byte-identical to main (169 pre-existing lib.dom / @cloudflare/workers-types conflicts). Worth noting separately: this fixture's script is check:types while the turbo task is check:type, so CI never runs it.

Unrelated flake found along the way, not fixed here

vite dev <-> wrangler dev > supports queues across dev sessions fails with expected [] to include 'hello from vite to wrangler' on 2 of 6 runs on main and 1 of 7 on this branch — equally present before and after, so it is independent of this change. Unlike the Windows crash it reproduces on macOS, so it should be tractable to chase separately.


  • Tests
    • Tests included/updated
    • Automated tests not possible - manual testing has been completed as follows:
    • Additional testing not necessary because:
  • Public documentation
    • Cloudflare docs PR(s):
    • Documentation not necessary because: this only changes test fixtures. There is no user-facing behaviour change, which is also why there is no changeset (@fixture/dev-registry and @fixture/shared are both private).

A picture of a cute animal (not mandatory, but encouraged)

Note

This is a contribution from an AI agent: OpenCode, claude-opus-5.

@changeset-bot

changeset-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 21edcb9

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@github-project-automation github-project-automation Bot moved this to Untriaged in workers-sdk Aug 4, 2026
@petebacondarwin
petebacondarwin force-pushed the fix/dev-registry-teardown-order branch from 44e454c to 33ebaee Compare August 4, 2026 15:11
@petebacondarwin petebacondarwin added ci:no-changeset-required Skip pull request checks for a valid changeset ci-flake Applied to PRs addressing CI flakiness labels Aug 4, 2026
@ask-bonk

ask-bonk Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Those are all service-binding tests unrelated to tail_consumers — the removal of tail_consumers from worker-entrypoint-with-assets.jsonc doesn't affect them. These tests use worker-entrypoint-with-assets for service bindings/RPC/assets, not tail.

Let me verify the removed tail_consumers from exported-handler.jsonc and external-durable-object.jsonc aren't asserted anywhere (i.e., no test asserted a tail event on those edges).

The old tests asserted tail via exported-handlerworker-entrypoint-with-assets and worker-entrypointexported-handler-with-assets. The external-durable-objectexported-handler edge was incidental and untested. Removals look safe.

I've done a comprehensive review:

  • The teardown LIFO ordering logic (fixture onTestFinished registered first → runs last; producer started last → torn down first) is sound.
  • The tail_consumers chain is acyclic: worker-entrypointexported-handler-with-assetsexported-handler. No cycle.
  • All test assertions (worker names, log prefixes, event directions) are consistent with the reversed middle edge and the config sources.
  • The stop() change correctly waits for actual process exit with a bounded 10s timeout; the tiny exit-listener race is covered by the timeout and the pre-check.
  • Removed incidental edges aren't asserted anywhere.
  • Service-binding tests are unaffected by tail_consumers removals.

No logic bugs, security issues, backward-compat violations, or incorrect API behavior. The one minor race in stop() is bounded and harmless. The PR is well-reasoned and internally consistent.

LGTM

github run

@pkg-pr-new

pkg-pr-new Bot commented Aug 4, 2026

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@15017

@cloudflare/build-output-utils

npm i https://pkg.pr.new/@cloudflare/build-output-utils@15017

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@15017

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@15017

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@15017

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@15017

miniflare

npm i https://pkg.pr.new/miniflare@15017

@cloudflare/pages-functions

npm i https://pkg.pr.new/@cloudflare/pages-functions@15017

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@15017

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@15017

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@15017

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@15017

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@15017

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@15017

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@15017

wrangler

npm i https://pkg.pr.new/wrangler@15017

commit: 21edcb9

petebacondarwin added a commit that referenced this pull request Aug 4, 2026
…per-iteration tally

Not for merge. Turbo runs the fixtures suite with --concurrency=2 and no
--continue, so an unrelated fixture flake aborts the job before dev-registry
executes (which is exactly what happened on the first run of #15017). This
harness isolates dev-registry and repeats it so a single round of CI yields a
usable sample instead of one coin flip.
petebacondarwin added a commit that referenced this pull request Aug 4, 2026
…per-iteration tally

Not for merge. Turbo runs the fixtures suite with --concurrency=2 and no
--continue, so an unrelated fixture flake aborts the job before dev-registry
executes (which is exactly what happened on the first run of #15017). This
harness isolates dev-registry and repeats it so a single round of CI yields a
usable sample instead of one coin flip.
@petebacondarwin
petebacondarwin force-pushed the fix/dev-registry-teardown-order branch from 33ebaee to 4616d98 Compare August 4, 2026 16:28
petebacondarwin added a commit that referenced this pull request Aug 4, 2026
…per-iteration tally

Not for merge. Turbo runs the fixtures suite with --concurrency=2 and no
--continue, so an unrelated fixture flake aborts the job before dev-registry
executes (which is exactly what happened on the first run of #15017). This
harness isolates dev-registry and repeats it so a single round of CI yields a
usable sample instead of one coin flip.
…teardown safe

The `Tests (Windows, fixtures)` job failed roughly half the time on
`fixtures/dev-registry`, almost always as a 50s timeout in one of the
`vite dev <-> vite dev` tests. The cause is a workerd abort during test
teardown: when a dev session is killed while another running session is
forwarding tail events to it, the surviving session's workerd calls
`std::terminate`. Miniflare then starts a replacement runtime, the Vite
plugin restarts the dev server to rebuild its module-runner sockets, and
the ~30s of churn lands on whichever test is still running.

The fixture made that shape easy to hit. `tail_consumers` formed two
cycles (`exported-handler` <-> `worker-entrypoint-with-assets` and
`worker-entrypoint` <-> `exported-handler-with-assets`), so most tests
carried a live tail edge whether or not they tested tail handlers, and no
shutdown order could keep every producer shorter-lived than its consumer.

Make the tail relationships a one-directional chain
(`worker-entrypoint` -> `exported-handler-with-assets` ->
`exported-handler`) and drop the incidental edges, which leaves tail
edges only in the tests that assert on them. Split the three
bidirectional tail tests into one test per direction so each can start
its consumer first and its producer second; Vitest tears sessions down in
LIFO order, so the producer is always killed while its consumer is still
alive. All existing tail assertions are preserved, including the
worker-name log prefixes from multi-worker Wrangler sessions.

Two supporting fixes, both of which let a session observe a peer
disappearing mid-teardown:

- The `devRegistryPath` fixture deleted the registry directory in its
  teardown, which Vitest runs *before* `onTestFinished` callbacks — i.e.
  while every dev session was still running. Register the removal as an
  `onTestFinished` callback during fixture setup so LIFO ordering runs it
  last instead.
- `runWranglerDev`'s `stop()` resolved when the kill signal had been
  delivered (on Windows, when `taskkill` exited), not when the process was
  gone. Wait for the actual exit, bounded, so sequential teardown really
  is sequential.
Reverts the temporary skip from #15018, which disabled the whole suite on
Windows to unblock other work "until the underlying issue is fixed". The
preceding commit fixes it: the timeouts were a workerd abort during teardown
(cloudflare/workerd#6913), triggered by killing a dev session that a still
running session was forwarding tail events to, and the fixture no longer sets
that up.

Windows is the only platform the dev registry flake ever appeared on, so
leaving these skipped there would mean the suite never guards the thing it
exists to guard.
@petebacondarwin
petebacondarwin force-pushed the fix/dev-registry-teardown-order branch from 4616d98 to 21edcb9 Compare August 4, 2026 17:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:no-changeset-required Skip pull request checks for a valid changeset ci-flake Applied to PRs addressing CI flakiness

Projects

Status: Untriaged

Development

Successfully merging this pull request may close these issues.

2 participants