ONNX Engine#10
Merged
Merged
Conversation
Integrate ONNX Runtime as an alternative inference backend alongside llama.cpp, enabling GPU-accelerated inference for vision-language models with platform-specific execution providers (DirectML, CUDA, CoreML). Changes: - Add ONNX Runtime dependencies with platform-specific features (DirectML/CUDA/CoreML) - Create vlm_onnx.rs module for ONNX inference engine supporting SmolVLM models - Extend ModelManager with ONNX model download functionality from HuggingFace - Add Tauri commands for ONNX model operations (download, load, generate) - Update UI with ONNX Models tab in model selection modal - Add quantization selector (Q4, Q8, FP16) for ONNX models - Configure downloads for SmolVLM2-256M-Video-Instruct with correct HF repo structure (ONNX files in onnx/ subdirectory, config/tokenizer at root) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Implement logic to detect model backend type and route to appropriate inference engine, enabling seamless switching between llama.cpp and ONNX Runtime backends. Changes: - Update model loading useEffect to check backend type - Route to load_onnx_model for ONNX models, load_model for llama.cpp - Disable audio capability check for ONNX models (not yet supported) - Add backend detection in handleSendMessage for inference routing - Convert image data appropriately for each backend: - RGB array for llama.cpp (existing) - JPEG bytes for ONNX Runtime (new) - Call generate_onnx_response for ONNX, generate_response for llama.cpp 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Update list_downloaded_models to include ONNX models by checking for _onnx_ pattern in directory names and verifying .onnx files exist. Add frontend model ID normalization to properly match ONNX models from HuggingFace repos with normalized names (handle slashes, dashes, case). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Integrate ONNX Runtime as an alternative inference backend alongside llama.cpp, enabling GPU-accelerated inference for vision-language models with platform-specific execution providers (DirectML, CUDA, CoreML). Changes: - Add ONNX Runtime dependencies with platform-specific features (DirectML/CUDA/CoreML) - Create vlm_onnx.rs module for ONNX inference engine supporting SmolVLM models - Extend ModelManager with ONNX model download functionality from HuggingFace - Add Tauri commands for ONNX model operations (download, load, generate) - Update UI with ONNX Models tab in model selection modal - Add quantization selector (Q4, Q8, FP16) for ONNX models - Configure downloads for SmolVLM2-256M-Video-Instruct with correct HF repo structure (ONNX files in onnx/ subdirectory, config/tokenizer at root) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Add SmolVLMImageProcessor with 4x4 grid splitting + global image (17 frames total, matching SmolVLM's expected input format) - Add proper prompt expansion with grid tokens (<fake_token_around_image>, <row_X_col_Y>, <global-img>) - Auto-detect layer count from decoder model inputs - Support multiple EOS tokens (2, <end_of_utterance>, </s>) - Fix pixel coordinate order to match model expectations - Add config.json path to model loading for HuggingFace config - Add test example for VLM inference 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Remove duplicate image = "0.25" from dev-dependencies that conflicted with image = "0.24" in main dependencies, causing E0464 compilation error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
infil00p
marked this pull request as ready for review
January 4, 2026 16:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.