Skip to content

fix(tools): don't mask empty streaming tool-selection as %!w(<nil>)#65

Merged
mudler merged 1 commit into
mainfrom
fix/tool-selection-empty-content
Jul 7, 2026
Merged

fix(tools): don't mask empty streaming tool-selection as %!w(<nil>)#65
mudler merged 1 commit into
mainfrom
fix/tool-selection-empty-content

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

Image turns with a reasoning model (e.g. gemma-4-e2b via LocalAI) fail with the meaningless error:

failed to select tool: failed to pick tool: tool selection failed: failed to make a streaming decision after 3 attempts: %!w(<nil>)

%!w(<nil>) is Go telling us fmt.Errorf("...: %w", …, lastErr) was called with a nil lastErr.

Root cause (reproduced against LocalAI v4.6.2 + gemma-4-e2b)

decisionWithStreaming's empty-content retry path (content == "" && len(toolCalls) == 0) continued without setting lastErr. When every attempt hit it, the loop exhausted with lastErr == nil and rendered %!w(<nil>).

This reliably happens on image turns: the vision model streams reasoning whose length exhausts the output-token budget (vision tokens crowd the context), finishing with finish_reason=length and no visible content and no tool call. The loop then retried three identical truncations and reported the nil wrap. Verified deterministically: with a constrained budget the model emits reasoning only, content_len=0, tools=0, finish=length.

Fix

Capture finish_reason from StreamEventDone, and in the empty-content branch:

  • finish_reason=length → fail fast with an actionable error (retrying truncates identically), never a nil wrap.
  • otherwise → keep retrying, but always record a real lastErr so the final error is a real cause.

The non-streaming decision() path already accepted empty content without this retry, so only the streaming path was affected.

Tests

decision_empty_content_test.go:

  • TestDecisionWithStreamingTruncatedEmptyContent — reproduces the exact %!w(<nil>); asserts a meaningful, length-naming error and fail-fast (1 attempt).
  • TestDecisionWithStreamingEmptyContentNoFinishReason — genuinely-empty responses still error out, never nil-wrapped.

Both fail on main (RED) and pass with the fix. Existing streaming tests unchanged.

🤖 Generated with Claude Code

When the streaming tool-selection decision returned no tool calls and empty
content on every attempt, the retry loop exhausted with lastErr still nil, so
the final fmt.Errorf("...: %w", maxRetries, lastErr) rendered the literal
"%!w(<nil>)" — hiding the real cause.

This reliably bites image turns with reasoning models (e.g. gemma-4-e2b via
LocalAI): the model streams reasoning but its reasoning exhausts the output
budget before any visible content, finishing with finish_reason=length and no
content. The empty-content path then retried three identical truncations and
reported %!w(<nil>).

Capture finish_reason from StreamEventDone and, in the empty-content branch:
  - finish_reason=length: fail fast with an actionable error (retrying
    truncates identically) instead of looping into a nil wrap.
  - otherwise: keep retrying, but always record a real lastErr so the final
    error is never a nil wrap.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@mudler mudler merged commit 588db3b into main Jul 7, 2026
2 of 3 checks passed
mudler added a commit to mudler/nib that referenced this pull request Jul 7, 2026
Pulls github.com/mudler/cogito#65 (v0.11.1-0.20260707230916-588db3bcbae5):
streaming tool-selection no longer masks an empty/truncated decision as
"%!w(<nil>)". Fixes image turns where a reasoning model's reasoning exhausts
the output budget (finish_reason=length) before producing content — nib now
surfaces an actionable error instead of the meaningless nil wrap.

Co-authored-by: Ettore Di Giacinto <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants