playing with llm WIP

[skyl]

User description

https://github.com/huggingface/candle

?

---

PR Type

• Enhancement

---

Description

• Introduces LLM-driven interpretation modal UI.

• Adds Tauri command stream_llm for token streaming.

• Implements LLM state management and token generation.

• Updates ReadingView import for new modal component.

---

Changes walkthrough 📝

	Relevant files
Enhancement	LlmInterpretation.tsx Add LLM interpretation modal component.                                    corpora-i-ching/src/components/LlmInterpretation.tsx New modal component for LLM interpretation display. Implements streaming text generation via Tauri invoke. Handles UI actions for generating and closing modal. +108/-0  ReadingView.tsx Update import to LLM modal component.                                        corpora-i-ching/src/components/ReadingView.tsx Updates import to use new LlmInterpretation. Comments out outdated InterpretationModal import. +2/-1      lib.rs Add Tauri command for LLM token streaming.                              corpora-i-ching/src-tauri/src/lib.rs Adds Tauri command stream_llm for streaming tokens. Integrates new LLM state wrapped in Mutex. Registers new command in the Tauri invoke handler. +22/-0    llm.rs New module for LLM integration and text generation.            corpora-i-ching/src-tauri/src/llm.rs Introduces LlmState struct for LLM and tokenization. Implements token generation and streaming logic. Loads model and tokenizer with external dependencies. +73/-0
Configuration changes	Cargo.toml Update Cargo.toml with LLM dependencies.                                  corpora-i-ching/src-tauri/Cargo.toml Adds dependencies: candle-core and candle-transformers. +2/-0

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Error Handling The token streaming error in the LLM generation handler is only logged to the console. Consider providing user feedback in the UI when token streaming fails. try { // Invoke the stream_llm command with the channel await invoke('stream_llm', { prompt: "Provide an I Ching reading interpretation.", channel, }); } catch (error) { console.error("Failed to stream LLM:", error); } Token Streaming The LLM token generation and streaming mechanism should be reviewed for edge cases, including proper handling of EOS tokens and decoding errors to ensure consistent stream delivery. pub fn generate( &mut self, prompt: &str, max_tokens: usize, sender: tauri::ipc::Channel<String>, ) -> anyhow::Result<()> { // Tokenize the input prompt let encoding = self .tokenizer .encode(prompt, true) .map_err(|e| anyhow::anyhow!(e))?; let mut tokens = encoding.get_ids().to_vec(); // Generate tokens one at a time and stream to the frontend for _ in 0..max_tokens { let input = Tensor::new(&tokens, &self.model.device())?.unsqueeze(0)?; let logits = self.model.forward(&input)?; let logits = logits.squeeze(0)?; let next_token = self.logits_processor.sample(&logits)?; tokens.push(next_token); // Decode the token to text let decoded = self .tokenizer .decode(&[next_token], true) .map_err(|e| anyhow::anyhow!(e))?; // Stream the token to the frontend sender.send(decoded)?; // Check for EOS token (adjust based on your model's EOS token ID) if next_token == self.tokenizer.token_to_id("</s>").unwrap_or(2) { break; } } Ok(()) }

[github-actions]

PR Code Suggestions ✨

No code suggestions found for the PR.

[skyl]

https://x.com/i/grok?conversation=1908930219802603898