Ultra mode times out on Cloudflare/chat-widget-protected sites (most modern commercial sites)

[fediddy]

Bug

Running skillui --url <url> --mode ultra against any site that holds long-lived network connections fails with:

✗ Ultra extraction  page.goto: Timeout 30000ms exceeded.
Call log:
  - navigating to "https://<site>/", waiting until "networkidle"

Root cause

waitUntil: 'networkidle' requires zero in-flight network connections for 500ms straight. That condition never becomes true on sites that hold connections open indefinitely:

• Cloudflare Turnstile / Bot Management (3+ long-poll connections per protected page)
• Chat widgets — Tawk.to, Intercom, Drift, LiveChat, Zendesk
• Session recording — Hotjar, FullStory, LogRocket
• Remarketing pixels — Google Ads ga-audiences, Meta Pixel

That's most of the commercial web. On affected sites the 25–30s timeout fires and ultra extraction silently bails with no useful data.

Repro

skillui --url https://gironroofing.com --mode ultra

Instrumented Playwright on that page and confirmed 4 connections still open 10s after load fired:

host	role
challenges.cloudflare.com/... (×3)	Cloudflare Turnstile bot-detection iframes
www.google.com/ads/ga-audiences	Google Ads remarketing beacon (intentionally never returns)

Fix

PR #3 replaces waitUntil: 'networkidle' with waitUntil: 'load' plus a soft waitForLoadState('networkidle', { timeout: 8000 }) in try/catch across 6 ultra/computed extractor goto sites. Well-behaved sites still benefit from the network-idle wait; affected sites continue rather than silently failing.

Also adds &screenshot.fullPage=true to the Microlink screenshot URL so screenshots/homepage.png captures the entire scrollable page rather than just the 1280×720 viewport.

Related

• Fix: fix: replace networkidle with load to avoid timeouts on Cloudflare/chat/pixel sites #3

[chipchaykoffsky-ux]

+1, confirming on https://www.bolsamania.com/ (Spanish financial news site) — same Timeout 30000ms exceeded on waitUntil: 'networkidle'. The site has continuous analytics/ad pings so the network never goes idle.

Local workaround that worked: sed -i 's/waitUntil: "networkidle"/waitUntil: "domcontentloaded"/g' on all 6 page.goto calls in dist/cli.js. After patching, ultra extraction pulled 19 colors / 2 fonts / 7 scroll frames cleanly.

The approach in #3 ('load' + soft networkidle with timeout) is better than my blunt domcontentloaded swap — it preserves the wait for well-behaved sites. Worth merging.

Also worth adding a CLI flag (e.g. --wait-until <event> and --timeout <ms>) so users can override per-site without source patches.

[amaancoderx]

Fixed it already by switching from strict networkidle waits to load with a soft fallback wait, so ultra extraction no longer times out on sites with persistent connections like Cloudflare, chat widgets, or analytics beacons.