Detach the poll thread when deattach_interrupt runs on it — don't strand its stack#10
Conversation
…and its stack pinedio_deattach_interrupt() skips pthread_join() when called from the poll thread itself (the af9bc27 self-join-deadlock fix). But skipping reclamation entirely strands the thread: it exits joinable, nothing can ever join it (the next attach_interrupt() overwrites the only handle), and glibc can never free or reuse a joinable thread's stack — ~8 MB of address space leaked per detach-from-callback. Callers that detach the interrupt from inside the callback on every radio interrupt (meshtastic firmware's RadioLibInterface ISR does) leak at interrupt rate: meshtastic/firmware#10468 measures ~9-12 stacks/min, hundreds of GB of VSZ within days, on every CH341-attached radio. pthread_detach(pthread_self()) in that branch lets glibc reclaim the stack when the thread exits, while the cannot-join-self deadlock stays fixed. The main-thread deattach path is unchanged (still joins).
|
Runtime validation on affected hardware — the patch eliminates the leak. Rebuilt meshtastic firmware
Unpatched, same box, same version: ~9–12 fresh stack pairs per minute, monotonic — 594 pairs / +4.9 GB VSZ at 66 minutes. Patched: a stable pool of ~7–8 (live threads + glibc's detached-stack cache), zero net growth across samples, stacks now demonstrably reused. (The one-time VmSize step between t0 and t+11 is 7 × 64 MB Also deployed to two Pi 5 + CH341 boxes (Debian trixie rebuild of the same recipe); multi-day soak running — I'll report if anything regresses. From my side this PR is field-validated. |
…exiting Deeper fix for the thread-stack leak class (see PR pine64#10 for the minimal one-liner). Creating and destroying a pthread per attach/detach cycle is churn no embedded target should pay — on 32-bit router-class platforms (OpenWrt mips/arm) the strand variant exhausts the address space in hours, and even leak-free churn costs a stack mmap + clone per radio interrupt. The poll thread is now created once per pinedio_init() and PARKS on a condition variable when the last interrupt detaches, resuming on the next attach. There is never a mid-life thread to join, so detaching from within the interrupt callback (which runs on this very thread) is inherently safe — the self-join hazard af9bc27 guarded against is gone by construction. The thread terminates exactly once, in pinedio_deinit() (join, or detach-self in the pathological deinit-from-callback case). Behavior change, deliberate: a pinedio_get_input() failure now latches in_error and keeps polling at the normal cadence (self-healing on a later good read) instead of silently killing the poll thread until the next full deinit/init.
|
t+54 min follow-up: VmSize is byte-identical to the t+29 sample (804,060 kB — zero growth in 25 minutes of active RX), and the two Pi 5 units deployed later are flat at ~327 MB after 30 min each. Soak continues; nothing further unless something regresses. |
…serspace#10) Pin libch341-spi-userspace to the fix-stranded-poll-thread-stacks fork commit (cff5a9f0) until pine64 PR meshtastic#10 merges upstream. Fixes the firmware#10468 pthread-stack leak on the CH341/USB path, which would hit the vm.max_map_count ceiling in ~2-3 days on a USB-radio router. Fork build: matrix trimmed to aarch64_cortex-a53 (OpenWrt One), deploy job removed (no secrets); artifacts served from the workflow run.
|
Oooh, the fix is interesting. I'll take a look. |
|
looks OK to me. @jp-bennett did you had a chance to test this? Thanks |
|
Have not been able to test yet, but the concept seems sound |
3 participants
Problem
pinedio_deattach_interrupt()skipspthread_join()when it runs on the poll thread itself — the af9bc27 guard, which correctly fixed the self-join deadlock. But skipping reclamation entirely strands the thread: it exits joinable, nothing can ever join it (thepthread_create()in the nextpinedio_attach_interrupt()overwrites the only handle), and glibc can neither free nor reuse a joinable thread's stack until join — so every detach-from-callback permanently leaks the thread's ~8 MB stack mapping.This is exactly the pattern meshtastic firmware hits: its
RadioLibInterfaceISR callsdisableInterrupt()as the first statement of the interrupt callback, which runs on this library's poll thread → deattach-from-self on every radio interrupt → one stranded 8 MB stack per interrupt. Measured in meshtastic/firmware#10468: ~9–12 stacks/min, hundreds of GB of VSZ within days, on every CH341-attached radio (Pi 4 and Pi 5, Debian and Ubuntu builds alike); SPI-attached radios (gpiod path) are unaffected. Full strace/gdb evidence chain in that issue:clone3, 30 fresh 8,454,144-byteMAP_STACKmmaps, 0 munmaps, all workers exited cleanlypthread_create: single static control block, handle overwritten per spawnFix
One line plus a comment: in the self-call branch,
pthread_detach(pthread_self()). A detached thread's stack is reclaimed by glibc at exit, and the cannot-join-self deadlock the guard was added for stays fixed. The main-thread path is unchanged (still joins).Testing status
The mechanism is pinned by live tracing on affected hardware (linked issue); I have not yet run a rebuilt binary with this patch. I operate a fleet with three leaking CH341 units (Pi 4 + Pi 5) and three SPI controls and am happy to run a patched build and report VSZ/mmap counts before/after if that helps land this.
Aside, out of scope here: the error path in
pinedio_pin_poll_thread()(in_error = true→ self-exit) can strand one stack the same way untilpinedio_deinit()joins it; a follow-up could apply the same treatment there.