feat(ui): render images inline in tool results by dimakis · Pull Request #374 · dimakis/mitzo

dimakis · 2026-06-09T20:57:35Z

Summary

Adds end-to-end support for rendering images from tool results (Read tool on JPEG, PNG, GIF, WebP files and generated plots)
New extractToolResultImages() extracts type: 'image' content blocks from SDK tool results that were previously discarded
Images flow through: server → WS event → protocol parser → messages reducer → <img> in ToolPill
Also handles subagent tool results with images

Changes

File	What
`packages/protocol/src/types.ts`	`ToolResultImage` type + `toolResultImages` field on block types
`packages/protocol/src/content-blocks.ts`	`extractToolResultImages()` function
`server/query-loop.ts`	Extract + emit images in `tool_result` WS events
`packages/client/src/slices/messages.ts`	`patchToolResult()` stores images, finish helpers propagate
`packages/client/src/protocol-parser.ts`	Parse images from WS messages
`frontend/src/components/ToolPill.tsx`	Render images as `<img>` in tool results
`frontend/src/styles/global.css`	`.tool-pill-result-img` styling

Test plan

Read an image file (e.g., Read tool on photo.jpg) — image renders inline in expanded tool pill
Generate a matplotlib plot via Bash, then read it — chart renders inline
Verify text-only tool results still render normally (no regression)
Verify subagent tool results with images render correctly
Check mobile layout — images should scale to fit screen width

🤖 Generated with Claude Code

Messages queued in connection.pendingSends during WS downtime (iOS background) were flushed to the wrong session after reconnect. Add clearPendingSends() and call it from newSession() and switchSession(). Co-Authored-By: Claude Opus 4.6 <[email protected]>

…dant comment Co-Authored-By: Claude Opus 4.6 <[email protected]>

The server process inherits SSH_AUTH_SOCK at startup, but macOS invalidates the socket path after sleep/wake. Resolve via launchctl getenv at session spawn time so git push/fetch always works. Co-Authored-By: Claude Opus 4.6 <[email protected]>

When Claude reads an image file (JPEG, PNG, GIF, WebP) or reads a generated plot, the SDK returns image content blocks that were previously discarded. This adds end-to-end support for extracting and rendering those images inline in the ToolPill component. Data flow: SDK tool_result → extractToolResultImages() → WS event → protocol parser → messages reducer → ToolPill <img> render. Also handles subagent tool results with images. Co-Authored-By: Claude Opus 4.6 <[email protected]>

dimakis

Centaur Review

Found 5 issue(s) (2 warning).

`packages/protocol/src/content-blocks.ts`

Solid end-to-end implementation of image rendering in tool results. Main concerns: no tests for the new extractToolResultImages function, unbounded image data size through WebSocket, and an unrelated SSH fix bundled in.

🟡 missing_tests (L50): extractToolResultImages() has no unit tests. It should be tested with: (1) string content (returns []), (2) undefined content (returns []), (3) array with image blocks, (4) array with non-image blocks, (5) image blocks missing source fields. The protocol package has no test files at all for content-blocks. [fixable]
🔵 unsafe_assumptions (L56): media_type from SDK content blocks is passed through without validation to a data: URI. While base64 data URIs are not exploitable for XSS in <img> tags (browser restricts script execution), validating against an image/* pattern would be a cheap defensive measure against unexpected MIME types. [fixable]

`server/query-loop.ts`

Solid end-to-end implementation of image rendering in tool results. Main concerns: no tests for the new extractToolResultImages function, unbounded image data size through WebSocket, and an unrelated SSH fix bundled in.

🟡 unsafe_assumptions (L1261): Image base64 data is forwarded to the client without any size limit. Text results are truncated at TOOL_RESULT_MAX_CHARS, but resultImages passes through unbounded. A tool returning a large image (e.g. a multi-MB screenshot) could produce an oversized WebSocket frame. Consider capping image data size or count. [fixable]

`packages/client/src/slices/messages.ts`

Solid end-to-end implementation of image rendering in tool results. Main concerns: no tests for the new extractToolResultImages function, unbounded image data size through WebSocket, and an unrelated SSH fix bundled in.

🔵 missing_tests (L329): The TOOL_RESULT reducer action now accepts images and threads them into toolResultImages via patchToolResult, but there are no reducer tests covering this path. The existing store test only verifies clearPendingSends behavior, not image state propagation. A reducer test should verify images survive patchToolResult → finishCurrent → rendered state. [fixable]

`server/chat.ts`

Solid end-to-end implementation of image rendering in tool results. Main concerns: no tests for the new extractToolResultImages function, unbounded image data size through WebSocket, and an unrelated SSH fix bundled in.

🔵 style (L409): The SSH agent socket resolution is unrelated to image rendering. It should be in a separate commit/PR to keep the changeset focused (the PR title is "render images inline in tool results").

dimakis · 2026-06-09T21:01:25Z

  return text;
 }

+export function extractToolResultImages(


🟡 missing_tests: extractToolResultImages() has no unit tests. It should be tested with: (1) string content (returns []), (2) undefined content (returns []), (3) array with image blocks, (4) array with non-image blocks, (5) image blocks missing source fields. The protocol package has no test files at all for content-blocks. [fixable]

dimakis · 2026-06-09T21:01:25Z

+  if (typeof content === 'string' || !Array.isArray(content)) return [];
+  const images: ToolResultImage[] = [];
+  for (const c of content) {
+    if (c.type === 'image' && c.source?.type === 'base64' && c.source.data && c.source.media_type) {


🔵 unsafe_assumptions: media_type from SDK content blocks is passed through without validation to a data: URI. While base64 data URIs are not exploitable for XSS in <img> tags (browser restricts script execution), validating against an image/* pattern would be a cheap defensive measure against unexpected MIME types. [fixable]

dimakis · 2026-06-09T21:01:25Z

            for (const block of content) {
              if (block.type === 'tool_result') {
                const resultText = extractToolResultText(block.content);
+                const resultImages = extractToolResultImages(block.content);


🟡 unsafe_assumptions: Image base64 data is forwarded to the client without any size limit. Text results are truncated at TOOL_RESULT_MAX_CHARS, but resultImages passes through unbounded. A tool returning a large image (e.g. a multi-MB screenshot) could produce an oversized WebSocket frame. Consider capping image data size or count. [fixable]

dimakis · 2026-06-09T21:01:25Z

@@ -322,6 +329,7 @@ export function messagesReducer(state: MessagesState, action: MessagesAction): M
        action.toolId,


🔵 missing_tests: The TOOL_RESULT reducer action now accepts images and threads them into toolResultImages via patchToolResult, but there are no reducer tests covering this path. The existing store test only verifies clearPendingSends behavior, not image state propagation. A reducer test should verify images survive patchToolResult → finishCurrent → rendered state. [fixable]

dimakis · 2026-06-09T21:01:25Z

  const venvPaths = getRepoConfig().resolvedVenvPaths;
  env.PATH = [...venvPaths, existingPath].join(':');

+  // Resolve the current SSH agent socket on macOS. The server process may have


🔵 style: The SSH agent socket resolution is unrelated to image rendering. It should be in a separate commit/PR to keep the changeset focused (the PR title is "render images inline in tool results").

Co-Authored-By: Claude Opus 4.6 <[email protected]>

dimakis · 2026-06-09T21:22:18Z

Centaur Review

Found 5 issue(s) (3 warning).

`packages/client/src/slices/messages.ts`

Solid feature implementation with good test coverage for the WS queue fix, but the MESSAGE_SNAPSHOT reducer is missing the new toolResultImages field (will drop images on iOS reattach), and extractToolResultImages needs unit tests.

🟡 regressions (L382): MESSAGE_SNAPSHOT reducer reconstructs StreamingBlocks but omits the new toolResultImages field (line 371-382). When an iOS client reattaches and receives a snapshot, any tool result images are silently dropped. Every other block-reconstruction path in this file (finishCurrent, finishSubagent, SUBAGENT_END) correctly copies toolResultImages. [fixable]
🔵 missing_tests (L230): patchToolResult gained a new images parameter but has no dedicated test exercising the image-patching path (patching into current, patching into finished messages, and the subagent tool result case). The existing store integration tests cover clearPendingSends but not image propagation through the reducer. [fixable]

`frontend/src/components/ToolPill.tsx`

Solid feature implementation with good test coverage for the WS queue fix, but the MESSAGE_SNAPSHOT reducer is missing the new toolResultImages field (will drop images on iOS reattach), and extractToolResultImages needs unit tests.

🟡 unsafe_assumptions (L102): The mediaType from tool results is interpolated directly into the data: URI (data:${img.mediaType};base64,...) with no allowlist. While <img> elements don't execute scripts even from data:text/html URIs, a non-image mediaType (e.g. text/html) would produce a broken image. Allowlisting to known image types (image/png, image/jpeg, image/gif, image/webp, image/svg+xml) would be more robust. [fixable]

`packages/protocol/src/content-blocks.ts`

Solid feature implementation with good test coverage for the WS queue fix, but the MESSAGE_SNAPSHOT reducer is missing the new toolResultImages field (will drop images on iOS reattach), and extractToolResultImages needs unit tests.

🟡 missing_tests (L48): The new extractToolResultImages function has no unit tests. It handles multiple edge cases (string input, undefined, missing source fields) that should be covered. Neither extractToolResultText nor extractToolResultImages have tests — this is a good opportunity to add a content-blocks.test.ts file. [fixable]

`packages/client/src/protocol-parser.ts`

Solid feature implementation with good test coverage for the WS queue fix, but the MESSAGE_SNAPSHOT reducer is missing the new toolResultImages field (will drop images on iOS reattach), and extractToolResultImages needs unit tests.

🔵 style (L341): msg.images as ToolResultImage[] | undefined is a bare type assertion on unvalidated wire data. A runtime guard (e.g. Array.isArray(msg.images) ? msg.images : undefined) would be more defensive, consistent with the (msg.isError as boolean) ?? false pattern on the adjacent line. [fixable]

dimakis · 2026-06-14T00:30:57Z

Superseded by #376 (reference-based image rendering). Closing.

dimakis and others added 4 commits June 9, 2026 19:11

fix: address review — clear all queues in switchSession, remove redun…

69edca6

…dant comment Co-Authored-By: Claude Opus 4.6 <[email protected]>

dimakis commented Jun 9, 2026

View reviewed changes

fix: address review — prettier formatting

f67be03

Co-Authored-By: Claude Opus 4.6 <[email protected]>

dimakis closed this Jun 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ui): render images inline in tool results#374

feat(ui): render images inline in tool results#374
dimakis wants to merge 5 commits into
mainfrom
feat/image-rendering

dimakis commented Jun 9, 2026

Uh oh!

dimakis left a comment

Uh oh!

dimakis Jun 9, 2026

Uh oh!

dimakis Jun 9, 2026

Uh oh!

dimakis Jun 9, 2026

Uh oh!

dimakis Jun 9, 2026

Uh oh!

dimakis Jun 9, 2026

Uh oh!

dimakis commented Jun 9, 2026

Uh oh!

dimakis commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		@@ -322,6 +329,7 @@ export function messagesReducer(state: MessagesState, action: MessagesAction): M
		action.toolId,

Conversation

dimakis commented Jun 9, 2026

Summary

Changes

Test plan

Uh oh!

dimakis left a comment

Choose a reason for hiding this comment

Centaur Review

packages/protocol/src/content-blocks.ts

server/query-loop.ts

packages/client/src/slices/messages.ts

server/chat.ts

Uh oh!

dimakis Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

dimakis Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

dimakis Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

dimakis Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

dimakis Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

dimakis commented Jun 9, 2026

Centaur Review

packages/client/src/slices/messages.ts

frontend/src/components/ToolPill.tsx

packages/protocol/src/content-blocks.ts

packages/client/src/protocol-parser.ts

Uh oh!

dimakis commented Jun 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

`packages/protocol/src/content-blocks.ts`

`server/query-loop.ts`

`packages/client/src/slices/messages.ts`

`server/chat.ts`

`packages/client/src/slices/messages.ts`

`frontend/src/components/ToolPill.tsx`

`packages/protocol/src/content-blocks.ts`

`packages/client/src/protocol-parser.ts`