feat: web search + fetch tools for agents (SearXNG-backed)#661
Open
Wirasm wants to merge 1 commit into
Open
Conversation
Give every kild agent two tools so any model — including MiniMax, which has
no provider-native search — can use the web:
- webfetch: URL → markdown, in-process (turndown), keyless. Guards: 5 MB
cap, 30/120s timeouts, a UA fallback for bot walls, and an SSRF block on
private/loopback hosts.
- web_search: query → {title,url,snippet}, backed by a self-hosted SearXNG
kild only points at via KILD_SEARXNG_URL. The engine never spawns or
manages the container; the backend sits behind a SearchProvider seam, so
DDG/fastCRW/Tavily are a new impl + a switch arm later.
Both register into the worker's customTools (beside the room tools) and the
CLI's in-process fallback. webfetch is available whenever web is enabled;
web_search only when a backend is configured (else a one-line notice).
KILD_WEB=off disables both. Adds `kild web search/fetch` debug commands and
an optional infra/searxng compose for one-command bring-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Gives every kild agent two web tools so any model — including MiniMax, which has no provider-native search — can use the web:
webfetch— URL → markdown, in-process (turndown), keyless. Guards: 5 MB cap, 30/120 s timeouts, a UA fallback for bot walls, and an SSRF block on private/loopback hosts.web_search— query →{title, url, snippet}, backed by a self-hosted SearXNG kild only points at viaKILD_SEARXNG_URL. The engine never spawns or manages the container. The backend sits behind aSearchProviderseam, so DDG / fastCRW / Tavily are a new impl + a switch arm later — no tool or worker changes.How it wires in
Both tools register into the worker's
customTools(beside the room tools) and the CLI's in-process fallback, so engine and standalonekild runbehave the same.webfetchis available whenever web is enabled;web_searchonly when a backend is configured (otherwise the worker logs a one-line notice).KILD_WEB=offdisables both. No protocol or cockpit changes — the tools surface as the generic tool events the UI already renders.Adds
kild web search "<q>"/kild web fetch <url>debug commands (exercise the backend without spending agent tokens) and an optionalinfra/searxng/compose for one-command bring-up.Boundary
kild owns the tools; the search backend is external and swappable.
webfetchneeds no backend at all. This keeps the engine free of any container-lifecycle concern.Validation
bun run typecheck,bun run lint,bun run compile— all greenbun test— 49 pass (+9 new: SearXNG JSON→hits mapping, fetch markdown/html/truncation/SSRF)kild web fetch https://example.com→ clean markdown; search error paths + SSRF guard verifiedinfra/searxng/README.mdhas the bring-up + curl checkTry it
Follow-ups (deferred)
Per-session
--webtoggle + cockpit provider picker; DDG/fastCRW providers behind the seam; atextformat forwebfetch.