Skip to content

fix(deploy): back geckodriver stdout log with a regular file, not a FIFO (no SIGABRT on teardown)#1209

Draft
vringar wants to merge 1 commit into
masterfrom
fix/geckodriver-fifo-sigabrt
Draft

fix(deploy): back geckodriver stdout log with a regular file, not a FIFO (no SIGABRT on teardown)#1209
vringar wants to merge 1 commit into
masterfrom
fix/geckodriver-fifo-sigabrt

Conversation

@vringar

@vringar vringar commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

The bug

geckodriver was killed with SIGABRT (exit 134) and dumped core on every browser teardown/recycle. OpenWPM routed geckodriver's stdout through a named FIFO (FirefoxLogInterceptor), whose reader — a daemon thread — could close the read end before geckodriver stopped writing. The next write() to stdout then returned EPIPE, which geckodriver's Rust runtime escalates to a panic; a Drop handler run during unwinding writes to stdout again, double-panics (panic in a destructor during cleanup), and calls abort() → SIGABRT + coredump. With N parallel browsers, each recycle produced N coredumps.

Evidence (per #1208): nine byte-for-byte identical coredumps over one crawl, arriving in bursts of 3 (= 3 parallel browsers), each containing:

thread 'webdriver dispatcher' panicked at library/core/src/panicking.rs:233:5:
failed printing to stdout: Broken pipe (os error 32)
panic in a destructor during cleanup

The fix

Back FirefoxLogInterceptor with a regular file instead of a FIFO. Writes to a regular file never raise EPIPE, so geckodriver can no longer be aborted by interceptor/teardown ordering — regardless of thread/process exit order. This removes the failure mode entirely (rather than just narrowing the race).

  • selenium_firefox.py: the interceptor creates a regular tempfile (tempfile.mkstemp), tails it (reads new lines as they appear), and stops on request via an explicit threading.Event (stop()), draining any remaining lines before unlinking the temp file. The daemon flag still guarantees the thread never blocks process exit. The now-unused mktempfifo helper is removed.
  • deploy_firefox.py: log_output points at the regular file; the interceptor is returned to the caller so teardown can stop it.
  • browser_manager.py: run_impl's finally calls webdriver_interceptor.stop() (after the driver is already quit/killed, so no write can race), draining final lines and removing the temp file promptly.

Logging is preserved: geckodriver stdout still reaches self.logger.debug as BROWSER i: driver: ....

Verification

  • Deterministic OS-level root-cause proof: a regular-file write after the reader closes succeeds; the same on a FIFO raises EPIPE — confirming both the cause and that the fix removes it.
  • Session-driven repro (direct Selenium, geckodriver 0.37.0, Firefox 152): with the regular-file interceptor, geckodriver survives the reader thread stopping mid-session, the tail captured the driver log lines, and the process exited cleanly. The FIFO path is timing-sensitive to reproduce the exact abort window on demand, but the OS-level proof and the structural argument (regular files cannot raise EPIPE) cover correctness; full validation rides on CI.
  • pytest -m pyonly green; mypy clean on both deploy_browsers files; black/isort clean; pre-commit green.

Note: the deeper cause is upstream — geckodriver/Rust should treat a broken pipe on stdout as a clean exit, not a panic-to-abort. Worth a separate Mozilla report; OpenWPM avoids triggering it entirely with this change.

Closes #1208

@codecov

codecov Bot commented Jun 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.34%. Comparing base (e438983) to head (3bfbf78).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1209      +/-   ##
==========================================
+ Coverage   62.12%   62.34%   +0.21%     
==========================================
  Files          40       40              
  Lines        3918     3909       -9     
==========================================
+ Hits         2434     2437       +3     
+ Misses       1484     1472      -12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

geckodriver was killed with SIGABRT (exit 134 + coredump) on every browser
teardown/recycle. Its stdout was routed through a named FIFO whose reader
(FirefoxLogInterceptor, a daemon thread) could close before geckodriver
stopped writing. The next write then returned EPIPE, which geckodriver's
Rust runtime escalates to a panic; a destructor run during unwinding writes
again, double-panics, and aborts.

Back the interceptor with a regular file instead. Writes to a regular file
never raise EPIPE, so geckodriver can no longer be aborted by reader/teardown
ordering.

To keep cleanup leak-proof, the interceptor opens the read end in start(),
and deploy_firefox unlinks the temp file's path immediately after opening
the write end. Both fds keep the now-anonymous inode alive until the process
exits, so no owpm_driver_*.log can leak into $TMPDIR -- even under the
multiprocess os._exit() and SIGKILL teardown paths, where no finally/atexit
hook would run. The tail thread stays a daemon and is reaped at process exit;
it buffers partial reads until a newline so a driver line is never split
across two log entries.

Add a pyonly unit test exercising the interceptor in isolation: lines reach
the logger, the temp file is unlinked-but-readable-via-fd (no leak), and
split writes are not fragmented.

Closes #1208
@vringar vringar force-pushed the fix/geckodriver-fifo-sigabrt branch from 7bb0cc7 to 3bfbf78 Compare June 21, 2026 01:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

geckodriver killed with SIGABRT (broken-pipe panic) on browser teardown — stdout logged via FIFO

1 participant