[Bug]: meshtasticd (portduino/Linux) VSZ + swap grow without bound under sustained load — VmPeak ~915 GB, VmSwap ~1.78 GB after 10 days

[Nursedude]

Summary

meshtasticd (Linux/portduino build) grows its virtual address space without bound on long-uptime hosts. On two boxes in a 5-box fleet I observed VmPeak ~915 GB and VmSwap ~1.78 GB after 9–10 days of uptime, while VmRSS stayed at ~45 MB. The 1.78 GB of touched-then-swapped pages pegged one Pi 5's swap at 100% and starved the rest of the stack. A clean restart drops VmPeak back to ~280 MB and VmSwap to 0.

I'm filing this from the perspective of an operator running a small permanent NOC fleet — happy to instrument further if it helps narrow it down.

Environment

• Version: meshtasticd 2.7.15 (all five fleet boxes are on the same build)
• OS: Debian / Raspberry Pi OS bookworm (trixie on one)
• Arch: aarch64
• Hardware: mix of Pi 4B (Cortex-A72), Pi 5 (Cortex-A76), and a Pi Zero 2 W (Cortex-A53)
• Service: meshtasticd.service running long-lived as a systemd unit
• HATs: Various Meshtastic SX1262 HATs via SPI

Fleet snapshot at the time of investigation

Box	Hardware	Preset	User	Uptime	VmPeak	VmSwap	RSS
A	Pi 5	LongFast	root	9d 7h	927 GB	1.42 GB	437 MB
B	Pi 5	LongFast	root	10d 11h	915 GB	1.78 GB	45 MB
C	Pi 4B	LongFast	root	4d 11h	714 MB	1.7 MB	10 MB
D	Pi 4B	SHORT_TURBO	meshtasticd	21d 5h	714 MB	0	18 MB
E	Pi Zero 2 W	SHORT_TURBO	root	9d 10h	280 MB	1.7 MB	6 MB

The pattern hits boxes A and B; doesn't hit box D despite the longest uptime (21 days). Same firmware version on every box.

Two variables correlate with the leak in this small sample:

1. LongFast preset (higher packet rate, broader gossip) — both leaky boxes are on LongFast; both heavily-trafficked boxes leak; the two SHORT_TURBO boxes don't.
2. Heavy local API polling — boxes A and B both have a local NOC stack hitting /api/v1/fromradio?all=true and :4403 TCP on a continuous cycle. Boxes C/D/E don't have that polling pattern.

Box C is on LongFast but only 4 days uptime, so it's not yet to the threshold where A and B exhibited the leak. Worth retesting after another week.

Reproduction observations

On the leaky boxes, VmData was ~908 GB while VmRSS was 45 MB:

Name:    meshtasticd
VmPeak:  915812448 kB
VmSize:  915804192 kB
VmRSS:      45232 kB
VmData:  908017456 kB
VmSwap:   1785328 kB
Threads:        6

This suggests sparse anonymous mappings — large committed-but-untouched virtual regions that don't add up to RSS, plus a smaller (but absolute-terms still significant) ~1.78 GB of pages that DID get touched and then got pushed to swap.

Threads stayed at 6 — this isn't a thread leak.

What the operator sees

Symptom on a 4 GB Pi 5: Swap: 2.0Gi / 2.0Gi (100% full) with all other processes running normally. Once swap saturates, meshtasticd's own pages start thrashing (continuous si/so activity in vmstat) and any latency-sensitive co-tenant on the box (local map server, dashboard, monitoring stack) degrades. Eventually load climbs and the box stops servicing inbound HTTP requests in time.

Recovery

sudo systemctl restart meshtasticd

Immediately frees VmSwap back to 0 and VmPeak back to ~280 MB. Radio API on :9443 is responsive again within ~10 seconds. No data loss observed on the radio side.

Mitigation in my fleet (operator workaround, not a fix)

I deployed a weekly meshtasticd-restart.timer to bound the growth at 7 days. Treating this as a band-aid until the underlying allocation pattern is found.

Hypothesis

I'm not deep in the firmware allocator paths, but the combination of:

• VmData ~900 GB (mostly uncommitted but reserved)
• VmRSS small but VmSwap multi-GB (touched-then-cold pages)
• Only triggered under high-traffic preset + heavy local API polling

…feels consistent with either (a) growing per-packet/per-route state that gets allocated and not freed (Router history, NodeDB, MQTT queue?), or (b) a glibc/jemalloc fragmentation pattern where freed allocations leave large arenas reserved but never returned to the OS. (a) would be a real leak; (b) would be malloc behavior that's only visible because of the workload.

Happy to capture /proc/$pid/smaps, /proc/$pid/maps, or attach a pmap snapshot from a leaky box if useful — I can rebuild the leak in a few days now that we've reset. Also happy to try MALLOC_ARENA_MAX=2 as a probe if that would help separate (a) from (b).

Related

Cross-link from operator-side ops memo: this triggered a cascade where my local NOC's cloud snapshot pipeline went stale for ~3h because the map service on the leaky box stopped responding under swap pressure. Not a meshtasticd bug per se, but the swap thrash made it the proximate cause.

[github-actions]

The issue provides a detailed analysis of a potential memory leak, but it lacks device logs at this stage. Please provide logs from the Web Flasher or Meshtastic CLI to assist in further investigation. Also, it would be helpful to include the exact steps to reproduce the issue, if not already clear, and confirm whether there are any differences in firmware version used on the boxes experiencing the problem.

[Nursedude]

Fleet operator here with 5-box reproduction data + a localized root cause. Posting in response to the data request.

Reproduction

5 × Raspberry Pi running meshtasticd 2.7.15 (Bookworm, kernel 6.12.75). Same userspace build across all five.

Box	Hardware	Kernel	Radio interface	Uptime	VSZ	RSS	mmap regions
A	Pi 5 8GB	6.12.75+rpt-rpi-2712	USB (sx1262 / meshtoad-e22)	18h 35m	77 GB	175 MB	19,875
B	Pi 5 4GB	6.12.75+rpt-rpi-2712	USB (sx1262 / meshtoad-e22)	18h 40m	65 GB	31 MB	16,761
C	Pi 4 4GB	6.12.75+rpt-rpi-v8	SPI HAT (sx1262 / MeshAdv 900M30S)	5d 6h	610 MB	10 MB	314
D	Pi 4 4GB	6.12.75+rpt-rpi-v8	SPI HAT (sx1262 / MeshAdv 900M30S)	22d 0h	610 MB	18 MB	315
E	Pi 3 1GB	6.12.75+rpt-rpi-v8	SPI HAT (sx1262 / MeshAdv 900M30S)	10d 5h	273 MB	5.7 MB	300

Pi 5 boxes leak. Pi 3 + Pi 4 boxes don't. Same firmware, same chip family (sx1262), different radio interface (USB vs SPI HAT) and different kernel SoC variant. Both Pi 4 boxes are stable at 5d and 22d uptime; the Pi 5 boxes have already accumulated tens of GB virtual after <19h.

Leak signature

VSZ grows; RSS stays bounded (175 MB on a box with 77 GB VSZ). The /proc/$PID/maps of the Pi 5 process is dominated by paired anonymous regions in this exact shape:

7fec2a390000-7fec2a3a0000 ---p 00000000 00:00 0    ← 64 KB guard page (PROT_NONE)
7fec2a3a0000-7fec2aba0000 rw-p 00000000 00:00 0    ← 8 MB rw stack region
7fec2aba0000-7fec2abb0000 ---p 00000000 00:00 0    ← 64 KB guard
7fec2abb0000-7fec2b3b0000 rw-p 00000000 00:00 0    ← 8 MB stack
... (repeated ~9,800 times)

Region-size histogram (anonymous regions only):

Pi 5 box A (77 GB VSZ, 19h):
   64 KB × 9807     ← pthread guard pages
   8 MB  × 9806     ← pthread stacks
   (handful of other regions, normal)

Pi 4 box C (610 MB VSZ, 5+ days):
   64 KB × 5
   8 MB  × 5
   (and ~36 other normal regions)

That's the glibc default pthread stack layout (8 MB stack + 64 KB guard, ~8.06 MB per thread). /proc/$PID/status reports only 4 live threads on the Pi 5 — meaning ~9,800 pthread stacks belonging to ALREADY-EXITED threads are still mapped into the address space.

RSS stays bounded because stack pages mostly never get touched — the leak is virtual address space, not physical memory. Eventually the address space fills and mmap() starts failing (or systemd OOM-kills, depending on the OOMScoreAdjust).

Rate

~9,800 leaked stacks / 18.6 hours ≈ 9 thread spawns per minute on the Pi 5 boxes. Constant rate (linear VSZ growth observed). Box A and Box B have very similar rates despite different RAM (8 GB vs 4 GB).

Variables that differ between leak / no-leak

Three variables shift together between the two groups, so the report can't isolate which one is necessary:

1. SoC: Pi 5 (rpt-rpi-2712) vs Pi 3/4 (rpt-rpi-v8)
2. Radio interface: USB CDC (the e22 USB module) vs SPI (HAT)
3. Live thread count: 4 vs 2 — the Pi 5 boxes carry an extra 2 threads that I'd guess are USB-related

Best hypothesis: the USB-radio handler is spawning detached threads (or non-detached threads that aren't joined) in a loop, and on this build path the stacks aren't being unmapped on exit. The fact that Pi 4 boxes with the same chip but SPI interface don't leak strongly suggests the USB code path is the trigger.

Less likely but worth ruling out: a glibc / NPTL version difference between the Pi 5 and Pi 4 OS images that breaks pthread stack cache reuse. Both run Bookworm with the same kernel version, but the architecture-specific glibc could differ.

Workaround we're running

Weekly systemd timer that restarts meshtasticd Sunday 02:30 local. Keeps VSZ bounded at the cost of a brief radio-stack reset. Trivial unit:

[Timer]
OnCalendar=Sun *-*-* 02:30:00
Persistent=true
Unit=meshtasticd-restart.service

Sufficient for the leak rate observed (~80 GB / week peak), but obviously a band-aid.

What would help isolate

• Anyone with a Pi 5 + HAT radio (not USB): does it leak? — that would isolate the USB code path as the cause vs SoC/arch.
• Anyone with a Pi 4 + USB radio: does it leak? — same isolation, other direction.
• A pmap -x snapshot from an affected user might match this signature exactly.

Happy to provide additional diagnostics if useful — ltrace/strace -e mmap,munmap,clone on a fresh Pi 5 daemon for ~5 min should catch the leaking call site directly.

[github-actions]

This issue has not had any comment or update in the last month. If it is still relevant, please post update comments. If no comments are made, this issue will be closed automagically in 7 days.

[Nursedude]

Still reproducing — fresh data to keep this open (responding to the stale bot).

Same 5-box fleet as my 2026-05-13 comment, now on meshtasticd 2.7.24.58~alpha (Debian trixie, kernel 6.18.3x). The signature is unchanged after two firmware minor bumps and a kernel major bump:

Box	Hardware	Radio	Uptime	VSZ	RSS	anon rw mmap regions
A	Pi 5	USB (sx1262)	5d 4h	~561 GB	937 MB	71,258
B	Pi 5	USB (sx1262)	5d 4h	~453 GB	664 MB	57,567
C	Pi 4	SPI HAT	10d+	217 MB	20 MB	30
D	Pi 4	SPI HAT	8d+	289 MB	20 MB	31
E	Pi 4	SPI HAT	15d+	217 MB	9 MB	29

Leak rate on the Pi 5 + USB boxes is now ~9.5 exited-thread stacks/min (~110 GB VSZ/day) — same order as May, still strictly Pi 5 + USB; the Pi 4 + SPI boxes remain flat for weeks. So the USB-CDC code path hypothesis still stands, and the variable-isolation asks from my earlier comment (Pi 5 + HAT, or Pi 4 + USB, anyone?) are still the fastest way to pin it.

Checked the 2.7.26.x changelogs and found nothing referencing this — happy to run strace -e mmap,munmap,clone on a fresh daemon or test a candidate patch on a live Pi 5 if a dev wants data. Weekly-restart workaround still holding in the meantime.

[Nursedude]

Follow-up: I can now answer my own isolation ask from May — Pi 4 + USB radio DOES leak, which isolates the USB code path and rules out the Pi 5 SoC/kernel variant.

A sixth fleet box I hadn't included before:

• Raspberry Pi 4 Model B Rev 1.2 (not a Pi 5)
• Ubuntu 24.04.4, kernel 6.8.0-1060-raspi (not Raspberry Pi OS, not the 6.12/6.18 rpt kernels)
• meshtasticd 2.7.24.58~ppa472b14c~noble (the PPA build, not the OBS Debian build the other boxes run)
• Radio attached over USB via a CH341 serial adapter (1a86:5512) — a different USB-serial chip family than the CP210x-class sx1262 USB modules on my Pi 5 boxes

66 minutes after a daemon restart:

VmSize:  5,210,524 kB  (~4.97 GB)
VmRSS:      27,008 kB
Threads: 6
anon region histogram:
  594 × 8192 KB rw-p   ← pthread stacks
  594 ×   64 KB ---p   ← guard pages

594 leaked stacks / 66 min ≈ 9 spawns/min — the same rate as both Pi 5 + USB boxes, with the same paired 8 MB + 64 KB signature and bounded RSS.

Updated variable isolation across the 6 boxes:

Variable	Leak boxes	Clean boxes	Verdict
SoC	Pi 5 ×2, Pi 4 ×1	Pi 4 ×3	not the SoC
Kernel	6.18 rpt, 6.8 raspi	6.18 rpt	not the kernel
OS/build	Debian/OBS, Ubuntu/PPA	Debian/OBS	not the distro/build
USB-serial chip	CP210x-class, CH341	—	not chip-specific
Radio interface	USB ×3	SPI HAT ×3	USB path, 3-for-3

So: every box with a USB-attached radio leaks at ~9 thread-spawns/min regardless of SoC, kernel, distro, packaging, or USB-serial chip; every SPI-HAT box is flat for weeks. The leaking thread spawns look tied to the USB/serial handling loop itself. Still happy to run strace -e clone,mmap,munmap on a fresh daemon or test a patch.