Add video input support for Kimi and future multimodal models

[aebrer]

Summary

Kimi K2.7-Code supports video input via OpenAI-style video_url content blocks with base64 data URLs on the official Moonshot API. dreb currently has no video modality at all. We should add video as a first-class input type, starting with the same pattern used for the Kimi OAuth image fix in #247.

Current Behavior

• Model.input is typed as ("text" | "image")[]; there is no "video" modality.
• No VideoContent type exists; UserMessage and ToolResultMessage content only supports text and images.
• packages/coding-agent/src/cli/file-processor.ts treats video files as text attachments.
• The openai-completions provider does not serialize video_url blocks.
• The Kimi OAuth provider forces the image input capability in modifyModels(), but not video.

Proposed Behavior

• Users can attach video files (MP4, WebM, MOV, etc.) via @path or --file.
• Video attachments serialize as base64 video_url data URL blocks for models that declare video input support.
• The kimi-for-coding OAuth model exposes video capability alongside image.
• Models that do not support video receive explicit placeholder text instead of silently dropping video content (matching the image fix pattern).

Acceptance Criteria

• VideoContent type and "video" input modality added to packages/ai/src/types.ts.
• UserMessage and ToolResultMessage content unions include VideoContent.
• openai-completions provider serializes video blocks as video_url data URLs and adds placeholder text for unsupported models.
• Kimi OAuth provider (packages/ai/src/utils/oauth/kimi-coding.ts) forces video capability alongside image.
• file-processor.ts detects common video MIME types and attaches them as VideoContent.
• scripts/generate-models.ts parses video modality from upstream model sources (OpenRouter, models.dev, etc.).
• Tests added for video serialization, Kimi OAuth capability injection, and file processing.
• Documentation updated (root README, packages/coding-agent/README.md, providers.md, models.md).

Context

• Related image fix: issue Fix Kimi vision handling in coding OAuth provider #245 / PR Fix Kimi vision handling in coding OAuth provider #247
• Kimi K2.7-Code Hugging Face model card: https://huggingface.co/moonshotai/Kimi-K2.7-Code
• Video input is currently experimental and only supported on the official Moonshot API.

Technical Notes

• The Kimi K2.7-Code chat template shows video_url content blocks are rendered as <|kimi_k25_video_placeholder|>, confirming OpenAI-style video_url is the correct payload shape.
• Video files are significantly larger than images, so we will likely need a max file size limit or a separate size cap distinct from image resizing.
• This work makes dreb ready for other providers that adopt OpenAI-style video_url blocks in the future.

[aebrer]

Issue Assessment: Add video input support for Kimi and future multimodal models

1. Summary

Kimi K2.7-Code exposes video understanding through the same OpenAI Chat Completions shape already used for images: a content block of type video_url whose url is a base64 data URI (data:video/mp4;base64,...). dreb currently has no video modality, no VideoContent type, and no provider path that emits video_url blocks. Adding support means extending the unified multimodal model, plumbing video blocks through messages and tool results, and teaching the openai-completions provider (used by the Kimi OAuth endpoint) to serialize them.

2. Current state

• Type system: packages/ai/src/types.ts defines ImageContent, UserMessage.content, ToolResultMessage.content, and Model.input as text/image only. packages/agent/src/types.ts AgentToolResult.content is also text/image only.
• CLI ingestion: packages/coding-agent/src/cli/file-processor.ts plus packages/coding-agent/src/utils/mime.ts only sniff for image/jpeg|png|gif|webp; everything else is inlined as <file> text.
• Message flow: packages/coding-agent/src/core/agent-session.ts, packages/agent/src/agent.ts, and packages/agent/src/agent-loop.ts already pass arrays of content blocks, but the arrays are typed as (TextContent | ImageContent)[].
• Provider serialization: packages/ai/src/providers/openai-completions.ts maps images to image_url data URLs and has explicit placeholder text when the model lacks image support. Other providers do the same with their native image shapes, but none handle video.
• Kimi OAuth capability injection: packages/ai/src/utils/oauth/kimi-coding.ts already forces "image" into kimi-for-coding input capabilities so stale metadata does not drop vision.
• Model registry: scripts/generate-models.ts parses image/vision modality from upstream sources but never video.
• Prior art: PR 247 / issue 245 fixed the same shape of problem for images on kimi-for-coding, touching generate-models.ts, kimi-coding.ts, and openai-completions.ts.

3. Gaps

1. Core type gap: No "video" value in Model.input, no VideoContent interface, and no place for video in UserMessage, ToolResultMessage, or AgentToolResult content unions.
2. Ingestion gap: file-processor.ts cannot recognize video files; @demo.mp4 is treated as text.
3. Provider gap: Only the openai-completions path is required for the Kimi OAuth use case, but the unified model implies providers should decide whether they can emit video blocks. At minimum openai-completions.ts needs a video_url branch.
4. Capability gap: kimi-coding.ts does not force "video", and generate-models.ts does not parse video modality.
5. Testing gap: No unit or E2E tests exercise video serialization or video capability injection.
6. Docs gap: README, providers.md, and models.md describe image attachment but not video.

4. Ambiguities / open questions

• Provider scope: Should the MVP target only Kimi OAuth (openai-completions provider), or should the work also implement video for OpenAI, Gemini, Anthropic, and Amazon Bedrock in the same change? Those providers use different native shapes, so a Kimi-only slice is much smaller.
• Tool results: Should tools be allowed to return video blocks, or should the MVP restrict video to user messages only? The current architecture carries (TextContent | ImageContent)[] in tool results, so allowing video there is natural but expands the change surface.
• Video size limits: Images are resized with Photon to a 4.5 MB base64 cap. Videos cannot be resized the same way. What is the max file size? Is it provider-specific? Should dreb refuse, warn, or truncate?
• Format support: The Kimi example uses MP4. Should dreb accept WebM, MOV, AVI as well, or validate strictly to MP4?
• Transcoding: Should dreb re-encode videos to MP4/H.264, or pass the original bytes through as-is? Passing through is simpler but may fail provider validation.
• Anthropic-messages Kimi path: The kimi-coding API-key provider uses anthropic-messages API. It is unclear whether that endpoint supports video; the official video examples are on the OpenAI-compatible endpoint used by OAuth. If API-key Kimi does not support video, its model metadata should not claim it.
• Session log size: Base64 video is large. Persisting it in SessionMessageEntry JSONL could create very large session files. Should video data be stored by reference (temp path) instead of inline?

5. Scope estimate

• Kimi-only MVP (recommended first step): small-to-medium. Touches types, openai-completions.ts, kimi-coding.ts, file-processor.ts, mime.ts, a few tests, and docs.
• Full cross-provider video support: medium-to-large. Requires native video serialization for OpenAI Responses (input_video), Anthropic (video block), Gemini (inlineData with video MIME), Bedrock (video block), etc., plus provider-specific size and format rules.

6. Risks

• API validation errors: Sending video_url blocks to a model that does not support video will fail. The existing image placeholder pattern mitigates this, but only if model.input is accurate.
• Silent cost/token blow-up: Video tokens are expensive; users may attach large files without realizing the cost.
• Session bloat: Inlining base64 video in JSONL session logs can produce multi-hundred-megabyte files and slow resume/compact operations.
• Backwards compatibility: Changing UserMessage.content and ToolResultMessage.content unions is a breaking change for extensions and RPC consumers. The new VideoContent type must be exported and documented.
• Test flakiness: E2E tests that upload real video will be slow and gated on API keys; prefer unit tests for serialization and capability injection.

Recommended next step

Scope the first implementation to Kimi OAuth only with video_url data URLs, mirroring the image fix in PR 247. Once that path is proven, extend to other providers and tool-result video blocks in follow-up work.

---

Automated assessment by mach6