DoH upstream wedges with `dial tcp <bootstrap_ip>:443: i/o timeout` after ~3-day uptime; `ctrld reload` hangs, only full stop+start recovers

[git-jerk]

Environment

• ctrld version: 1.5.0 (2026-03-04)
• Platform: Asuswrt-Merlin 3006.102.7_2 (latest stable) on RT-AX88U Pro (ARM aarch64, 1 GB RAM)
• Install: JFFS via sh -c "$(curl -sL https://api.controld.com/dl)" -s YOUR_RESOLVER_ID forced
• Config: Control-D-managed mode (--cd flag) → single DoH upstream, timeout 5000 ms, bootstrap_ip 76.76.2.22
• Network: IPv4 only (IPv6 disabled on client devices), 1 Gbps residential ISP, average ~50 queries/min

Symptom

After ~3 days 11 hours of clean uptime, every subsequent DoH query from ctrld to https://dns.controld.com/YOUR_RESOLVER_ID fails with:

{"level":"error","error":"could not perform request: Get \"https://76.76.2.22/YOUR_RESOLVER_ID?dns=...\": dial tcp 76.76.2.22:443: i/o timeout","time":"2026-04-21T04:55:06-07:00.058","message":"[e1f340] failed to resolve query"}
{"level":"warn","time":"2026-04-21T04:55:06-07:00.058","message":"upstream \"upstream.0\" marked as down immediately (failure count: 2410)"}

Failure counter climbs into the low thousands in under an hour. No successful queries in between. verify.controld.com via nslookup still resolves (presumably from cache or the bootstrap-time resolve), but all fresh queries fail.

Key differentiator from a real network/upstream outage

From the same router shell, at the same moment, both of these succeed instantly:

$ ping -c 3 76.76.2.22
# 0% loss, 20ms RTT

$ curl --max-time 8 -sS -v 'https://dns.controld.com/YOUR_RESOLVER_ID?dns=<b64>'
# HTTP/1.1 200 OK, TLS 1.3 handshake clean, application/dns-message body returned

WAN is healthy (default route up, no interface flaps, no dnsmasq reload events in syslog). Router uptime 3d 11h, load avg 0.12, conntrack 838/300000, ~350 MB memory free. Nothing else on the box reports network trouble.

This is a ctrld-internal issue — its Go HTTP client pool is wedged — not a network/peering problem.

Reload also hangs

$ /jffs/controld/ctrld reload
[error] failed to reload ctrld: timeout waiting for ctrld reload
[notice] Service is running

Same process continues running, same wedged state. Only a full ctrld stop && ctrld start (where stop completes in ~1s cleanly) recovers. After restart: first query returns in 40 ms.

Hypothesis

Looks like a classic Go net/http2 clientConnPool wedge — a transient network event (NAT rebind, upstream NIC reset, CDN edge maintenance) leaves a half-dead HTTP/2 conn cached in the Transport. Every subsequent client.Do() attaches to it and fails on Transport.DialContext default timeout. The curl-from-same-shell test proves the network path and TLS endpoint are healthy — only the cached conn is dead.

Context from Go upstream:

• net/http: http2 clientConnPool may cache a dead connection forever (#39750)
• x/net/http2: stream close raciness leaves dead conn in pool (#59690)
• net/http: http/2 connection management magnifies the effect of network timeouts (#63422)

Note: ctrld v1.4.9 fixed the DoH3 analog ("Fixed DoH3 connections closing incorrectly, preventing recovery during network outages") and v1.5.0 fixed the DoT analog ("connection validation before reuse to prevent io.EOF errors from server-side idle timeouts"). Plain DoH over HTTP/1.1+H2 appears to be the remaining unpatched transport in this class.

Suggested fix

Configure the HTTP/2 transport's keepalive-ping parameters so the Go stdlib detects wedged conns and drops them:

import "golang.org/x/net/http2"
h2t, _ := http2.ConfigureTransports(httpTransport)
h2t.ReadIdleTimeout = 15 * time.Second
h2t.PingTimeout     = 5 * time.Second

This issues a PING frame if 15 s pass with no frame activity; if no ACK within 5 s, closes the conn. The pool re-dials a new one on next use. Same shape as the DoT/DoH3 fixes already shipped.

Related issues

• #271 — DoH3 sibling, same shape, open since 2025-12-08
• #290 — dns_watchdog watchdog goroutine termination bug (different layer but same "silently broken" class)
• #249 — "No nameservers available for query", identical error strings, closed without public root-cause documentation

Workaround in production

Until an upstream fix lands I've added:

1. A second upstream in ctrld.toml (Quad9 DoT, different transport code path, different provider) with failover_rcodes = ["SERVFAIL","REFUSED"] so queries route around the wedged upstream.
2. A 1-minute cron watchdog on the Merlin router that does a real DNS round-trip via the ctrld listener and SIGKILL + restarts after 5 consecutive failed probes.

Happy to share either config if useful.

More info

Resolver ID redacted — I can share privately if you need it for repro. Full error log snippet and the ctrld.toml available on request.

[matrix8967]

Thanks for the report. I've got a similar setup on my Asus Router, so I'll try to reproduce this.

Are you seeing this when using DoT, DoQ, or DoH3?

[git-jerk]

Confirmed — this is DoH only. upstream.0 in my ctrld.toml:

[upstream.0]
  type = 'doh'
  endpoint = 'https://dns.controld.com/<resolver_id>'
  bootstrap_ip = '76.76.2.22'
  timeout = 5000
  ip_stack = 'v4'

Not DoT, not DoQ, not DoH3. The Quad9 DoT entry I mention in the body is a second upstream I added as a workaround after the wedge — not a trigger.

Happy to share more for repro: the full ~100-line error window or a fully-redacted ctrld.toml as a gist. What helps most?

[matrix8967]

✔️ DoH only, gotcha. I just reset my ASUS routers to ctrld 1.5.0 w/ DoH...One is prod, the other is mostly idle so I'll keep an eye out for this.

Both the ctrld.toml file and the error log would be helpful if you're willing to share them.

Feel free to link to a gist, or you can email them to support at help [@] controld.com and reference this GH Issue.

[git-jerk]

Gist: https://gist.github.com/git-jerk/9f3f3d0e918d181a4c19e6af2863e333 — two files:

• ctrld.toml — current config
• log-sample.txt — real per-query JSON from /jffs/ctrld.log with routing decisions and upstream.0 → upstream.1 failover cycles. Client hostnames / MACs / query-domains stripped; RFC1918 LAN IPs (10.20.30.x) left alone.

Update since my last comment (~62h ago):

1. ctrld log view returns "runtime debug logging is not enabled" until you add log_level = "debug" + log_path = "..." to [service] in the toml. -v / -vv CLI flags alone didn't help under Merlin's $cmd & wrapper (stdout → dead pipe). Once log_path was set, routing became visible. Might be worth a note in docs/config.md.

2. First "in the wild" event at 2026-04-24 00:04 PT: 2 consecutive failed DoH probes ~60s apart, then silent recovery before my 5-fail watchdog restart threshold — classic Go net/http2 pool-poisoning in mild form, not a full wedge this time.

3. Potential ctrld issue worth looking at: my [upstream.1] with endpoint = 'tls://dns.quad9.net' (no port) silently parsed as tcp///dns.quad9.net and every failover errored with dial tcp: lookup tcp///dns.quad9.net: unknown port. No startup validation/warning — only visible at query time with log_level = "debug". Fixed by explicit :853. After fix, Layer 1 saves 5 of 6 transient upstream.0 failures silently; before fix, every one became a user-visible error. type = 'doh' URLs default to :443 via URL scheme; type = 'dot' doesn't similarly default.

Happy to trim anything in the gist differently.

[matrix8967]

Thanks - I was actually just posting an update for you when you posted your new findings.

I'll take a look at your new findings shortly and get back to you.

As for your original issue: I think I've reproduced this on my Asus Router (GT-AX6000 3006.102.7_2) w/ ctrld 1.5.0 and a plain DoH upstream. I'll get this ticketed internally for our Dev Team to have a look at this.

I think I've been largely unaffected by this since I set a custom rfetch_time in my config to 60s which creates a fresh HTTP client that bypasses the poisoned connection pool...You may see if this eases your pain with this issue while we work on a permanent fix.

Thanks for the report(s) and detective work. I'll let you know what I find about your TLS and Logging issues.