Tinyexec 1.2.3 breaks lint-staged tests on Ubuntu

[iiroj]

Hello, upgrading lint-staged to use tinyexec@^1.2.3 is causing failures of tests in GitHub Actions running on ubuntu-latest on Node.js versions 22.2.1, 24.x and 26.x. Windows and macOS tests do pass.

Here's the related Pull Request on updates: lint-staged/lint-staged#1802

The single failing test is related to searching for configuration files, and tinyexec is used to spawn Git to list some files:

Details

exec('git', ['ls-files',
  '-z',
  '--full-name',
  '-t',
  '--cached',
  '--others',
  '--exclude-standard',
  '--',
  ':(glob)**/package.json',
  ':(glob)**/package.yaml',
  ':(glob)**/package.yml',
  ':(glob)**/.lintstagedrc',
  ':(glob)**/.lintstagedrc.json',
  ':(glob)**/.lintstagedrc.yaml',
  ':(glob)**/.lintstagedrc.yml',
  ':(glob)**/.lintstagedrc.mjs',
  ':(glob)**/.lintstagedrc.mts',
  ':(glob)**/.lintstagedrc.js',
  ':(glob)**/.lintstagedrc.ts',
  ':(glob)**/.lintstagedrc.cjs',
  ':(glob)**/.lintstagedrc.cts',
  ':(glob)**/lint-staged.config.mjs',
  ':(glob)**/lint-staged.config.mts',
  ':(glob)**/lint-staged.config.js',
  ':(glob)**/lint-staged.config.ts',
  ':(glob)**/lint-staged.config.cjs',
  ':(glob)**/lint-staged.config.cts'
])

With tinyexec 1.2.3 the command must be producing no output, which makes the test fail. Previous version 1.2.2 works ok.

I assume it's related to the destroying of streams on 'exit' introduced in #137

[43081j]

heh and that PR was only done for lint-staged 😅

do you have a link to the failure? or any reproduction?

[iiroj]

Yes, unfortunately I had this feeling given that the whole issue/PR looked AI-generated...

Failures can be seen here in the Actions log: https://github.com/lint-staged/lint-staged/actions/runs/26680743607

I'm still not sure of the original issue, but destroying the stream feels like the incorrect solution to a sub-subprocess being left open; is the real issue in the working of the subprocess logic itself, instead of its spawning? I will try to replicate the original issue directly in lint-staged but for now I need to pin tinyexec@1.2.2 to release another bugfix.

[iiroj]

It must be a timing issue because setting up similar destory logic to lint-staged fails a bit differently here: lint-staged/lint-staged#1804

EDIT: this is using pinned tinyexec@1.2.2.

[43081j]

Yes, unfortunately I had this feeling given that the whole issue/PR looked AI-generated

it did but i checked the bug myself, did some testing, etc. it really is a bug and the way you'd solve it does seem to be by destroying the stream.

this is a pain since it was only done to help lint-staged, and now apparently breaks it.

[iiroj]

For what it's worth, I can't replicate myself that ESLint with tsserver gets stuck when spawning via tinyexec (in the tinyexec repo). Also, reverting the code changes in 20e4117 while keeping the tests, still passes in the CI, so I'm not sure what the new tests do (given they also pass with the previous code): https://github.com/iiroj/tinyexec/pull/1/changes

[iiroj]

I prompted Claude Opus 4.8 (Max effort) about a spawned process's own subprocess getting stuck, and also got as the first response suggesting to destroy the output readable streams on the 'exit' event; it event used the exact same terminology so I believe this is where the fix idea came from.

However, pressing a bit further with a followup saying the solution doesn't work, Claude does continue by outputting that destroying the stream might result in loss of output, given that the I/O operations are asynchronous on UNIX (like it says on Node.js docs.

I think for now we should revert #137 and go back to the drawing board, starting with a reproducibly failing test scenario. Sorry about the confusion, I shouldn't have redirected the "fix" here from lint-staged/lint-staged#1800 before verifying it myself.

[43081j]

Yep I will revert it until we figure it out.

I'm also part way through figuring out the "real fix".

I'll revert tomorrow morning as I'm out all day

[Mearman]

Investigating further. The destroy-on-exit approach was wrong and the original repro wasn't tight enough, both fair points.

Building a deterministic test that actually fails on 1.2.2, then working out the right fix without the buffer-drain race. Will report back with findings.

[43081j]

FYI the test in #140 does fail before the fix and passes after it. So should be good help.

[Mearman]

experimenting with better reproducing the original issue in isolation over here https://github.com/Mearman/tinyexec-grandchild-mre

[Mearman]

Update.

MRE: https://github.com/Mearman/tinyexec-grandchild-mre

via-tinyexec reliably reproduces the grandchild-pipe deadlock on Linux and macOS for tinyexec 1.1.2 / 1.2.2. 1.2.3 fixes it. Deadlock side is solid.

Couldn't produce a tight standalone repro for the buffer-drain race that broke lint-staged's CI. Tried the chain that hits it in real repos (lint-staged → eslint with projectService, eslint-plugin-prettier, eslint.config.ts via jiti, standalone lint-staged.config.ts, pnpm exec, open stdin pipe), individually and combined. None triggered the data loss on Linux. The trigger in our repos was stdin pipe inheritance from the git hook environment: something inside eslint spawns a grandchild that reads from inherited stdin, never sees EOF, holds the inherited stdio open. A minimal eslint config doesn't spawn anything that does this.

@iiroj asked on #141:

Is this is an issue of tinyexec, or is the root problem that the spawned script doesn't handle closing its own child process on exit?

What actually fixed it for us was on the consumer side: <&- to close fd 0 on every pnpm/node/turbo invocation in .husky/pre-commit. The app layer can handle it.

But the chain in the wild is opaque. lint-staged uses tinyexec to spawn eslint, eslint's projectService (or plugins) spawn whatever, and the consumer of lint-staged has no insight three levels down. Asking consumers to track arbitrary grandchildren at that distance isn't practical.

Can PR a stdin-reading-grandchild fixture variant if useful: same effect on tinyexec, different reason-for-living from the stdout-holding one. Closer match to the real trigger shape. Shout.

[Mearman]

Got the buffer-drain repro after all. Concurrent x() calls are the trigger: 10 in-flight × 20 rounds reliably hits it on Ubuntu × tinyexec 1.2.3 × every Node version. Ubuntu Node 22 sample:

mode=await    losses=16/200 worst-loss=1756/5000
mode=iterator losses=51/200 worst-loss=1756/5000
distribution: min=3244 max=5000

~63 KiB lost per affected run (one kernel pipe buffer). Iterator drops 3x more often than await, matches the lint-staged 17.0.5 failure shape. Only 1.2.3, only Ubuntu.

Test: https://github.com/Mearman/tinyexec-grandchild-mre/blob/main/via-tinyexec/tests/data-loss-concurrent.mjs

Going to drop this test into a local tinyexec branch and try shaping a fix that passes it without regressing the deadlock tests. Will open a PR if it lands cleanly.

[Mearman]

Iteration converged. Branch: https://github.com/Mearman/tinyexec/tree/fix/buffer-drain-race

Per-stream idle watcher in _onExit. Timer reschedules on every chunk; only fires destroy if the stream has been quiet for exitDrainTimeout ms (configurable on Options, default 50). Normal exec calls pay no added latency because 'end' fires before the idle window expires; the grandchild case (where 'end' never fires) gets destroyed after the timeout. Cherry-picked the big-sigh test restructure to verify against the tougher 30s-grandchild fixtures; all 44 existing tests still pass.

Stress numbers on Linux Node 22, 20 concurrent × 50 rounds × 5000 lines:

mode=await    losses=0/1000 worst-loss=0
mode=iterator losses=0/1000 worst-loss=0

Same shape across the MRE matrix on ubuntu/macos/windows × Node 22/24/26: green. (Node 20 cells fail on an unrelated tsdown/unrun build issue when installing from source.)

Will open a PR if useful. Otherwise the branch is there for #140 to pull from.

[iiroj]

I no longer think we should fix this via code changes in tinyexec, since I'm still not able to replicate the original issue of typescript-eslint's languageOptions.parserOptions.projectService: true setting making ESLint get stuck when running via lint-staged. Maybe we should first figure that out before adding more and more complex code to tinyexec. This could be a "configuration error".

@Mearman I re-opened issue lint-staged/lint-staged#1800 so that we can try to reproduce the issue there.