Add toolrag attack family: injection via a poisoned retrieved document - #31
Merged
Conversation
Model the tool-result / RAG-poisoning indirect-injection channel (Greshake et al., arXiv:2302.12173; OWASP LLM01): untrusted content arrives in a document the agent retrieves mid-task, not an inbox email. Add a lightweight external-document store to the workspace environment with search_documents / read_document tools, and a toolrag family whose benign scheduling task needs a detail that lives only in the poisoned document — so the tool result is on the critical path — with the canary co-located there (one-hop leak). The scorer, exfil_spec, canary, utility registry, and stub machinery are reused unchanged; the leak sink is still send_email to the attacker. A channel-aware onpath_text() helper keeps the on-path invariant test covering the new family. Tests: build + on-path guarantee, the document tool returns the poisoned body, leak vs no-leak both scored, and an end-to-end offline stub run through the toolrag suite. First cut is leak variants only; the multi-secret distractor gap variant and broader domain diversity are noted as follow-ups. Part of #10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A first-cut fourth attack family for the tool-result / RAG-poisoning channel (#10). Minimal slice; design notes + deferred scope below.
The channel
The three existing families (
delayed,assembly,encoded) all deliver the injection through an inbox email. This adds the distinct, well-documented indirect-injection channel where untrusted content rides in what a tool returns mid-task — a retrieved document / RAG result (Greshake et al., arXiv:2302.12173; OWASP LLM01).Shape
search_documentsandread_document, mirroringread_inbox/read_email. ADocumenthas asource(URL / knowledge-base) to make its untrusted, third-party nature explicit.toolragfamily: the benign task ("look up X in the help centre / search the web and add it to my calendar") needs a scheduling detail that lives only in the retrieved document, so the poisoned tool result is on the critical path. The canary is co-located in that document (one-hop leak).On-path guarantee
The required event date is absent from
user_taskand present only in the retrieved document — the agent cannot complete the benign task without reading the poisoned tool result. Enforced at import by the builder and covered by the on-path invariant test via a new channel-awareonpath_text()helper (so the guard now spans all four families).Scorer reused unchanged
canary,exfil_spec, the scorer (leaked/hijacked), the utility registry, and the stub machinery are untouched. The leak sink is stillsend_emailto the attacker.Tests
Build + on-path guarantee;
read_documentreturns the poisoned body whilesearch_documentsdoesn't leak it; leak vs no-leak both scored through the real scorer; hijack-without-leak; and an end-to-end offline stub run through thetoolragsuite (covers registry / stub-script / utility wiring). ruff + ruff format --check + mypy + pytest all green locally (412 passed, 7 xfailed).Deferred (follow-ups, noted here not silently dropped)
assembly) — where several look-alike documents are retrieved and the injection names the target only generically — to produce the honest hijack>leak gap. This first cut is leak variants only.http_post) alongsidesend_email.Referencing #10 (design-first, mentored) rather than auto-closing — happy to adjust the shape per discussion there before this lands.