Pin gadget MAC + self-heal dwc2 USB endpoint lockups causing intermittent connect failures

[jayofelony]

Summary

Investigation into intermittent IPv4 failures over the USB gadget on Linux hosts (enx... interface stuck at 70 (connecting) / falls back to IPv6-link-local / APIPA), starting from #27. Two fixes are included, addressing two distinct problems found along the way. Neither is a complete root-cause fix for the underlying hardware issue — see "What's still open" below.

Commit 1: Pin gadget MAC addresses and loosen CLIENT DHCP race

Without host_addr/dev_addr set in g_ether.conf, g_ether/u_ether generates a random MAC pair on every module load (every boot / gadget re-enumeration). Linux hosts name USB-Ethernet interfaces from the MAC (enx<mac>) and track NM/udev state per-device, so a new MAC every boot makes the host treat the gadget as a brand-new device each time.

• debian/rpi-usb-gadget.postinst: derives two stable, valid locally-administered/unicast MACs from the board's CPU serial (same pattern as the existing <serial> stamping) and stamps them into g_ether.conf at install time.
• rpi-usb-gadget: CLIENT_DHCP_TIMEOUT 6s → 15s; ics-watch.c: CLIENT_PROBE_WINDOW 15s → 25s — gives DHCP more time to actually bind before the CLIENT/SHARED arbitration gives up and bounces back.

This alone did not fix the core issue — see next commit.

Commit 2: Self-heal for dwc2 USB gadget endpoint lockups

What's actually happening

Correlating dmesg -T on both the Pi and a Linux host during a cold-boot repro (Pi freshly booted, host already running, then plugged in) shows this is a USB transport-layer failure, not a DHCP-readiness race:

1. Host negotiates CDC-ECM (Linux's cdc_ether driver — unlike Windows, which has no native ECM driver and always negotiates RNDIS instead, so the two OSes exercise different endpoint/transfer-handling code paths on the same dwc2 hardware).
2. Linux's usbnet driver has a hardcoded 5s TX watchdog (TX_TIMEOUT_JIFFIES in drivers/net/usb/usbnet.c). It fires repeatedly (NETDEV WATCHDOG: transmit queue 0 timed out, every ~5-6s) and force-unlinks the in-flight URB each time.
3. On the Pi side, this eventually (or in some repros, seemingly never) shows up as dwc2_hsotg_ep_stop_xfr: timeout GINTSTS.GOUTNAKEFF / timeout DOEPCTL.EPDisable — the dwc2 driver's own attempt to cleanly disable the now-dead OUT endpoint also times out.
4. Once wedged, the link does not self-recover — confirmed across a 20+ minute window with multiple host-side USB cable replugs, all failing raw enumeration (device descriptor read/64, error -110, device not accepting address, error -71, unable to enumerate USB device). Previously the only known recovery was a full power cycle of the Pi.

The fix

ics-watch now watches for this condition and self-heals by reloading g_ether (confirmed to fully recover the link within seconds — real DHCP lease obtained on the very next enumeration, no reboot needed):

• Fast path: follows the kernel ring buffer (journalctl -kf) for the two dwc2_hsotg_ep_stop_xfr/dwc2_hsotg_txfifo_flush timeout messages above, and reloads immediately on a match (guarded so it doesn't fire while carrier is already up and healthy — e.g. against a marker logged by its own prior recovery's teardown).
• Fallback path: a second, independent trigger for wedges that apparently never log anything on the Pi side at all (reproduced live: host logs its usbnet TX watchdog for 100+ seconds straight while the Pi's dwc2 driver stays completely silent). Tracks the interface's rx_packets counter: if carrier has been up for CARRIER_STUCK_TIMEOUT (10s) with zero packets actually received from the host, treats it as wedged. Deliberately RX-only, not RX+TX — ics-watch's own periodic ICS-gateway arping() probes transmit from the same interface and were masking a genuinely dead link with self-generated TX "activity" during testing.
• Both paths share a 60s cooldown to avoid a reload storm.

A note on the recovery mechanism itself

A lighter alternative was tried during development — unbinding/rebinding just the dwc2 platform device via sysfs (/sys/bus/platform/drivers/dwc2/{unbind,bind}) instead of a full rmmod/modprobe, since it goes through the same probe()/remove() → dwc2_core_reset() path without unloading the kernel module. It worked in an isolated manual test, but caused the Pi to bootloop once deployed and triggered repeatedly (every ~10-15s, under a since-reverted, more aggressive trigger threshold). The exact cause wasn't confirmed — persistent journald logging wasn't enabled yet on the test device, so the crashing boot's logs were lost — but the correlation was clear enough to revert to the plain rmmod/modprobe approach, which was validated across many real repro cycles without incident. Worth revisiting with proper crash logging in place if someone wants a lighter-weight recovery path.

What's still open

This is a mitigation for the symptom, not a fix for the root cause in dwc2 itself:

• Not confirmed: whether Windows/RNDIS genuinely never hits this independently (a fresh, isolated Windows-only cold-boot test, with the Pi power-cycled first and no prior Linux attempt in that boot, hasn't been done) vs. macOS (which also uses ECM — the APIPA reports in Incorrect SHARED mode IP address with MacOS/iOS/iPadOS #27 suggest a related-but-possibly-not-identical failure mode).
• Not confirmed: whether current Raspberry Pi kernels already carry upstream dwc2 endpoint-disable-timeout fixes, or if this is worth reporting upstream to the dwc2 maintainers directly.
• Not tried: tuning dwc2 gadget FIFO sizes (g_rx_fifo_size/g_np_tx_fifo_size/g_tx_fifo_size module params) as a possible mitigation for a FIFO-overflow-triggered lockup.
• Not tried: whether ics-watch's own probe traffic cadence in the vulnerable first ~30s after carrier-up contributes to triggering this (it adds broadcast traffic right alongside the host's own DHCP retries, mDNS, IPv6 ND, etc.).
• Bigger, unexplored option: swapping the legacy monolithic g_ether driver for a libcomposite/configfs-based ECM or RNDIS function, which has a different transfer-scheduling path that some community reports suggest doesn't hit this class of dwc2 hang. This would be a substantial change to how the postinst and g_ether.conf set up the gadget, so treating it as a separate, larger follow-up rather than folding it into this PR.

If anyone with deeper dwc2/USB gadget peripheral-mode expertise wants to dig into why the OUT endpoint wedges in the first place, that's the real prize here — this PR just makes sure it recovers automatically in the meantime instead of requiring a manual power cycle.

Testing

Reproduced live multiple times against a real Ubuntu host across several sessions (cold Pi boot into an already-running host, and cold host boot into an already-running Pi): confirmed the self-heal detects the wedge and automatically restores a working DHCP lease within seconds, with no manual intervention, across both the kmsg-marker fast path and the rx_packets fallback path independently.

[jasonfen]

Hit what looks like the same class of failure from a macOS host and posted the full data in #27 — short version: host→device delivery is essentially dead, device→host is fine.

Counters sampled simultaneously on both sides over one 280s session, Pi Zero 2 W / trixie / kernel 6.18.39, Apple Silicon host:

direction	host	device
host → Pi	netstat Opkts = 161 (froze, Oerrs=0)	usb0 rx_packets = 1
Pi → host	Ipkts = 248	tx_packets = 222

One packet arrived out of 161 the host's driver reported sending. Opkts froze at exactly 161 across five separate re-enumerations. I can't yet confirm the ep_stop_xfr/txfifo_flush signature your watchdog keys on — the Pi got unplugged before my sampling window closed — so I don't know whether this is your lockup or a different one. Rerunning with dmesg capture and will follow up either way.

Your commit 1 matches what I saw independently: six boots, six distinct gadget MACs, no host_addr/dev_addr in g_ether.conf.

One thing worth checking in that commit though. On this install g_ether.conf reads:

options g_ether idVendor=0x2E8A idProduct=0x0013 iManufacturer="Raspberry Pi Ltd." bcdDevice=0x0100 iProduct="Raspberry Pi USB Gadget" iSerialNumber=

iSerialNumber is empty — the existing <serial> substitution already produced an empty string here, so SERIAL=$(awk '/^Serial/{print $3}' /proc/cpuinfo) came back blank at install time. Most likely cause is the package being installed during image build in a chroot, where /proc/cpuinfo isn't the target board's. This image is a pwnagotchi build, so that path is plausible.

Since mac_from_hash in the new postinst hashes that same $SERIAL, an empty serial means every board derives its MACs from the same input and gets identical host/dev MACs — no longer per-board, and an outright collision if someone attaches two Pis to one host. Might be worth a fallback (/sys/firmware/devicetree/base/serial-number, machine-id, or a random-but-persisted value written on first boot) plus a guard that skips stamping if the serial is empty.

Happy to test a build of this branch on the setup above.

[jasonfen]

Correcting my earlier comment on the serial, and reporting back on the dmesg capture.

Serial: I said the awk in postinst comes back blank and guessed chroot. On the running board it isn't blank:

$ grep -i '^Serial' /proc/cpuinfo
Serial          : 000000001bda2cd5
$ cat /sys/firmware/devicetree/base/serial-number
000000001bda2cd5

So the serial is available at runtime, yet g_ether.conf still has iSerialNumber= empty. That points at the file being stamped at image build time, when /proc/cpuinfo wasn't the target board's, rather than the awk failing on a running system.

That changes what a fix needs to do. A guard that skips stamping on an empty serial protects fresh installs but does nothing for images that already shipped with a blank value, and those are the ones that will collide once mac_from_hash starts deriving from the same field. Re-stamping on first boot when the value is empty would cover both. /sys/firmware/devicetree/base/serial-number looks like a reasonable source if /proc/cpuinfo is unavailable.

dmesg: ran a full 6 minute capture with snapshots every 60s. No ep_stop_xfr, no txfifo_flush, no timeouts, no resets. dmesg is byte identical at every snapshot, nothing logged after bound driver g_ether. So whatever I'm hitting doesn't produce the messages your watcher keys on. If it is the same underlying lockup then only your rx_packets fallback would catch it.

For what it's worth the shape is: host queues 161 packets with no errors, device receives 1 with no drops and no errors, carrier stays up the whole time, usb0 is UP,LOWER_UP with 10.12.194.1/28. Full numbers in #27.

Still happy to test a build of this branch.

[jayofelony]

Wow that's a lot to read.

[jasonfen]

Following up on my two comments above. The host to device failure I reported is a macOS bug, not the lockup this PR handles, so please disregard that part.

I re-tested with the dwc2 interrupt count sampled alongside the packet counters. Across all 36 ten second intervals of a 360s run, the device's dwc2 IRQ delta exactly matched its tx_packets delta, so every interrupt was an outbound completion and none came from an inbound transfer. The gadget's controller never saw the host's packets. macOS AppleUserECM stops transmitting after exactly 161 packets per enumeration with Oerrs 0 and nothing logged. Filed as FB24614121.

That also explains why I could not find your ep_stop_xfr or txfifo_flush signature. There was nothing wrong on the device side to log.

The MAC part of my comments is unaffected and still stands. No host_addr/dev_addr in g_ether.conf gives a fresh random MAC every boot, ten distinct ones across ten boots here, and iSerialNumber= is empty on this image, so mac_from_hash would derive from a blank input. The re-stamping point in my correction above still applies.

Happy to test a build of this branch, though I can no longer offer a working macOS host as a test bed until Apple fixes theirs.