Claude Opus 4.6 refuses structured extraction for some arXiv papers

[wconnell]

Problem

The Text Extractor agent fails with stop_reason='refusal' (zero output tokens) when processing this paper via Claude Opus 4.6. The model declines the extraction request immediately — this is not a content issue in the traditional sense but appears to be the safety classifier reacting to the prompt pattern.

Repro

Paper: https://arxiv.org/abs/2604.11661

Run:

python -m src.main process https://arxiv.org/abs/2604.11661

Observed error

Text extractor: no text block in response.
stop_reason='refusal'
block_types=[]
usage=Usage(input_tokens=26766, output_tokens=1, ...)

Hypothesis

Opus 4.6's classifier may treat the text_extractor.txt prompt pattern (large $raw_text block surrounded by --- delimiters, followed by instructions) as prompt-injection-shaped. Opus tends to be more conservative than Sonnet on these patterns.

Suggested fixes

1. Switch this agent to Sonnet 4.6 — rarely refuses simple extraction tasks and is ~5x cheaper. Opus is overkill for title/author/section parsing.
2. Restructure the prompt — put instructions first, clearly frame the text as "content to be parsed" rather than an untrusted blob, avoid raw --- delimiters around user content.
3. Add a graceful fallback — on stop_reason=='refusal', retry with Sonnet automatically.

Current state

src/agents/text_extractor.py uses claude-opus-4-6 with a messages.stream() call and Pydantic parsing. Retry logic exists (tenacity) but refusals are deterministic — retries don't help.