Skip to content

Fix: Python 3.14 compatibility#249

Open
griffinmilsap wants to merge 2 commits into
devfrom
fix/compat-314
Open

Fix: Python 3.14 compatibility#249
griffinmilsap wants to merge 2 commits into
devfrom
fix/compat-314

Conversation

@griffinmilsap
Copy link
Copy Markdown
Collaborator

@griffinmilsap griffinmilsap commented May 19, 2026

Closes #235

What happened

The Python 3.14 failure was caused by ezmsg subclassing concurrent.futures.ThreadPoolExecutor and overriding its private _adjust_thread_count() implementation in order to force daemon worker threads for the event loop's default executor.

That override depended on private CPython attributes used by older Python versions:

  • _initializer
  • _initargs

In Python 3.14, ThreadPoolExecutor changed its internal worker setup. The executor now uses a worker-context factory path (for example _create_worker_context()) instead of the old initializer fields. As a result, our subclass crashed at runtime with:

AttributeError: '_DaemonThreadPoolExecutor' object has no attribute '_initializer'

Why we were doing this at all

The custom executor exists to make default-executor threads daemon threads during non-strict shutdown mode.

That behavior helps ezmsg avoid hanging forever on process shutdown when blocking executor work is still outstanding. In particular, it lets the runtime exit even if some blocking work submitted through run_in_executor() does not cooperate with cancellation.

So the goal was real and useful:

  • improve shutdown robustness
  • avoid non-daemon executor threads keeping processes alive
  • preserve best-effort cleanup without requiring every blocking task to be perfectly behaved

Why this approach is fragile

The current approach subclasses a stdlib executor and overrides private implementation details.

That is brittle because:

  • CPython does not guarantee compatibility for private attributes or method internals
  • internal thread/bootstrap behavior can change across Python releases
  • fixes become version-specific and reactive
  • behavior can fragment across Python versions unless continuously maintained

In short: the shutdown intent is valid, but the mechanism is coupled to internals that are prone to breakage.

Minimal fix in this PR

This PR keeps the existing design but makes it compatible with both pre-3.14 and 3.14-style executor internals.

The compatibility patch updates _DaemonThreadPoolExecutor._adjust_thread_count() to:

  • use the 3.14-style worker-context path when available
  • fall back to the older _initializer / _initargs path on earlier versions
  • preserve daemon-thread behavior and active-task accounting

This is the smallest targeted fix to restore Python 3.14 compatibility without changing shutdown semantics.

Better long-term direction

A better design is to stop replacing the event loop's default executor with a custom ThreadPoolExecutor subclass.

Option 1: targeted daemon-thread helper (recommended)

Instead of globally swapping the loop's default executor, create a small helper for the specific blocking operations that must not prevent process exit.

For example, a helper could:

  • start a daemon threading.Thread
  • run the blocking callable in that thread
  • bridge completion or failure back to the event loop via an asyncio.Future

This keeps the stdlib executor untouched and confines the special shutdown behavior to the few places that actually need it.

Option 2: explicit daemon threads at known call sites

If only a handful of shutdown-sensitive operations need this behavior, start daemon threads directly at those sites instead of routing them through the loop's default executor.

That is even simpler and avoids a generalized executor abstraction entirely.

Option 3: custom executor implementation

We could replace the subclass with a minimal executor implementation that we own end-to-end.

This would remove the dependency on CPython private ThreadPoolExecutor internals, but it is more code and more maintenance surface than the targeted-thread approach.

Recommendation

Keep the compatibility fix in this PR because it is small and restores Python 3.14 support quickly.

Then follow up with a refactor that removes _DaemonThreadPoolExecutor entirely in favor of targeted daemon-thread handling for shutdown-sensitive blocking work. This follow-up is noted in issue #250

That would preserve the desired shutdown behavior while avoiding future breakage from stdlib internal changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

High level API is broken in 3.14

1 participant