Skip to content

feat: builtin media tools — read_image / transcribe_audio / read_video#49

Merged
mudler merged 4 commits into
masterfrom
feat/builtin-media-tools
Jul 5, 2026
Merged

feat: builtin media tools — read_image / transcribe_audio / read_video#49
mudler merged 4 commits into
masterfrom
feat/builtin-media-tools

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

What

Three builtin agent tools that let the model read a media file from the workspace and get text back — the programmatic counterpart to the user-facing attachments from #48. Each is one specialist call via the LocalAI API, so any agent model (even text-only) can use them:

Tool Delegate How
read_image(path, question?) specialist.Describe chat with an image_url part → vision model → text
transcribe_audio(path) specialist.Transcribe /audio/transcriptions → parakeet/whisper → transcript
read_video(path, question?) new specialist.DescribeVideo chat with a native video_url part → video-capable model → text

read_video needs no ffmpeg — LocalAI parses video_url content parts natively (core/http/middleware/request.go), so it mirrors read_image.

How

  • specialist.DescribeVideo mirrors Describe, sending a base64 video_url part; reuses DataURI.
  • Config gains VideoModel; the Session gains videoModel/workingDir; per-tool model selection (read_imagevisionModel, read_videovideoModel, transcribe_audiotranscribeModel; empty ⇒ LocalAI default). The visionModel field reserved in feat: multimodal attachments — file→text, transcription, native image/audio/video (+ CLI @path//attach) #48 is now consumed.
  • Tools live in chat/mediatools.go, each holding an injected delegate (the askUserTool pattern → unit-testable with fakes). Paths resolve against WorkingDir via resolveWorkspacePath (roots relative paths, matching the host read tool — not a sandbox). Errors are returned as the tool-result string so the model reads and reacts to them.
  • Gating: each tool is individually gated by s.toolEnabled(name) in the BuiltinTools allowlist and added to readOnlyTools (observational → auto-approve in default prompt mode).
  • Media calls run on the per-turn context, so Ctrl+C interrupts an in-flight description/transcription.

Testing

DescribeVideo httptest (asserts a data:video/ video_url part is sent), each tool's Run with fake delegates (arg threading, empty-path guard, delegate-error → error result), resolveWorkspacePath table test, and IsReadOnly membership. go build ./... + go test ./chat/ ./specialist/ green.

Built subagent-driven with per-task spec+quality reviews and a final whole-branch review, which caught and this PR fixes the media delegates capturing the session context instead of the per-turn context (so in-flight calls now honor interrupt).

Follow-ups

Optional per-tool config docs; frame-sampling fallback for models lacking native video input (only if a text-only-video workflow is ever needed). The desktop UI is a separate nib-desktop effort.

🤖 Generated with Claude Code

@mudler mudler merged commit 4ce5451 into master Jul 5, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants