feat(web): drive the REAL browser for search + fetch — the persona can actually use the internet now#1943
Merged
Merged
Conversation
…n actually use the internet now The web hand was hollow: `web/search` used DuckDuckGo's Instant-Answer API (trivia only → count:0 for every developer query), and there was no `web/fetch` at all. So the persona was offered a search tool that returned nothing and never used it — "pretending the internet doesn't exist" because functionally, for her, it didn't. The reqwest-scrape fix I tried first got DDG's "anomaly" bot-block page (200, zero results) — the classic scraper wall. Fix: drive the host's REAL headless browser (the same Chromium the screenshotter already locates) with `--dump-dom` — Chrome loads the URL, runs its JS, prints the rendered DOM. A real browser gets real results where an HTTP scrape is blocked. - `web/browser::render_dom(url, settle)` — locate browser → `--headless=new --dump-dom` with a real UA → captured stdout, 30s timeout, fail-loud on no-browser / timeout / empty DOM. Reuses `capture::web::locate_browser` (now shared) — no duplicated browser list. - `web/search` (duckduckgo) → renders `html.duckduckgo.com/html/?q=…`, parses `result__a`/`result__snippet`, unwraps the uddg redirect. Keyless floor that ACTUALLY works; Brave key still the reliable paid path. - `web/fetch` (NEW, AiSafe) → renders any URL, strips script/style/tags, returns clean capped readable text + <title>. The "read the doc/page I found" hand — the natural partner to search. Live-verified through the persona command path: web/search → 8 real results; web/fetch → the rendered docs.rs page. Unit-tested (parse, redirect-decode, url-encode, readable-extraction, bot-block-empty). Closes the core of task #93. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo
joelteply
added a commit
that referenced
this pull request
Jul 8, 2026
…a HABIT, not just an available tool (#1944) The web tools now return real results (PR #1943), but a working tool she never reaches for is worthless. Her acting doctrine (`ACTING_BLOCK`) told her to write/run/read — but said nothing about the web, so she'd guess a library's API from a stale training memory instead of looking it up. That's exactly how a local model loses to a real dev workflow. Fix: one sentence in the acting doctrine — when she hits a library, API, error, or fact she's unsure of, SEARCH (`web/search`) and READ (`web/fetch`) the way a developer actually works; her training has a cutoff, the web doesn't; foraging then writing from what she found beats confidently shipping something wrong. Doctrine/PX, not output-steering — it names when the hand is for, same voice as the rest of the block. This is the edge canned agents don't have on real projects: she learns the current tool instead of hallucinating last year's API. [[active-acquisition-foraging]] [[px-persona-experience-tools-as-good-ux]] Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo Co-authored-by: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The web hand was hollow:
web/searchused DuckDuckGo's Instant-Answer API (trivia only → count:0 for everydeveloper query), and there was no
web/fetchat all. So the persona was offered a search tool that returnednothing and never used it — "pretending the internet doesn't exist" because functionally, for her, it didn't.
The reqwest-scrape fix I tried first got DDG's "anomaly" bot-block page (200, zero results) — the classic
scraper wall.
Fix: drive the host's REAL headless browser (the same Chromium the screenshotter already locates) with
--dump-dom— Chrome loads the URL, runs its JS, prints the rendered DOM. A real browser gets real resultswhere an HTTP scrape is blocked.
web/browser::render_dom(url, settle)— locate browser →--headless=new --dump-domwith a real UA →captured stdout, 30s timeout, fail-loud on no-browser / timeout / empty DOM. Reuses
capture::web::locate_browser(now shared) — no duplicated browser list.web/search(duckduckgo) → rendershtml.duckduckgo.com/html/?q=…, parsesresult__a/result__snippet,unwraps the uddg redirect. Keyless floor that ACTUALLY works; Brave key still the reliable paid path.
web/fetch(NEW, AiSafe) → renders any URL, strips script/style/tags, returns clean capped readable text + <title>. The "read the doc/page I found" hand — the natural partner to search.Live-verified through the persona command path: web/search → 8 real results; web/fetch → the rendered docs.rs
page. Unit-tested (parse, redirect-decode, url-encode, readable-extraction, bot-block-empty). Closes the core of
task #93.
Co-Authored-By: Claude Opus 4.8 [email protected]
Claude-Session: https://claude.ai/code/session_01LoTjvf5j3Ez13g6k8mRkFo