[Feature]:❗Implement AI based Tool Selection

[MC-Meesh]

Replace Pattern Matching with AI-Driven Tool Selection for @enteract Commands

Issue Summary

Currently, @enteract commands use basic pattern matching to select MCP tools, which severely limits their capability for complex requests. Meanwhile, regular AI chat has a sophisticated AI-driven tool calling system that's much more capable but not exposed to users. We need to unify these systems to provide better tool selection and execution.

❗Proper MCP should exist but "@enteract" is a shortcut for regex - users don't have access to real MCP. Should unify if possible. The regex is reliable but limited.

First validate

Current Architecture Problems

Pattern Matching System (@enteract)

Location: src/composables/mcpService.ts - parseActionFromMessage()

Current Implementation:

// Very basic string matching
if (lowerMessage.includes('screenshot')) {
  actions.push({ toolName: 'take_screenshot', parameters: {} })
}
if (lowerMessage.includes('click') && coordMatch) {
  actions.push({ toolName: 'click_at', parameters: { x: 100, y: 200 } })
}

Limitations:

• ❌ Hardcoded string matching (includes('screenshot'))
• ❌ Cannot handle complex multi-step requests
• ❌ Parameter extraction is primitive (basic regex only)
• ❌ No context understanding
• ❌ Cannot combine multiple tools intelligently
• ❌ Fails on requests like: "take a screenshot then click on the red button in the top right"

AI Tool Calling System (Regular Chat)

Location: src-tauri/src/ollama.rs - process_tool_calls() and build_mcp_system_prompt()

Current Implementation:

// AI model generates structured tool calls
let system_prompt = "You can use tools like: TOOL_CALL: take_screenshot {}"
// Model generates: "I'll help you. TOOL_CALL: take_screenshot {}"
let tool_call_pattern = regex::Regex::new(r"TOOL_CALL:\s*(\w+)\s*(\{[^}]*\})").ok()?;

Capabilities:

• ✅ Handles complex multi-step requests
• ✅ Understands context and intent
• ✅ Generates appropriate parameters automatically
• ✅ Can combine multiple tools intelligently
• ✅ Provides explanations and reasoning

Proposed Solution

Replace the pattern matching system with AI-driven tool selection for @enteract commands.

Option 1: Enhance @enteract with AI Planning (Recommended)

New Flow:

1. User sends @enteract take a screenshot and click login
2. Frontend sends request to AI model for tool planning
3. AI generates structured tool execution plan with TOOL_CALL: format
4. Backend parses and executes AI-generated plan sequentially
5. Results displayed to user with progress updates

Option 2: Expose Existing AI Tool System

Make the existing generate_mcp_enabled_response available through user commands (e.g., @ai instead of @enteract).

Implementation Plan

Phase 1: Backend Changes

Files to modify: src-tauri/src/ollama.rs

1. Create new command: generate_mcp_tool_plan

   #[tauri::command]
   pub async fn generate_mcp_tool_plan(
       user_request: String,
       mcp_session_id: String,
       planning_session_id: String,
       // ... other params
   ) -> Result<(), String>

2. Enhanced system prompt for tool planning:

   const TOOL_PLANNING_PROMPT: &str = r#"
   You are a tool execution planner. Analyze the user request and create a step-by-step plan using available tools.
   
   Available tools: {tool_list}
   
   For each tool you want to use, format as:
   TOOL_CALL: tool_name {"param1": "value1", "param2": "value2"}
   
   Provide clear explanations for each step.
   "#;

3. Event emission for tool plans:

   app_handle.emit(&format!("mcp-tool-plan-{}", planning_session_id), tool_plan)

Phase 2: Frontend Changes

Files to modify: src/composables/mcpService.ts

1. Replace pattern matching in parseActionFromMessage():

   // OLD: Pattern matching
   private static parseActionFromMessage(message: string, availableTools: any[])
   
   // NEW: AI-driven planning
   private static async generateToolPlan(message: string, sessionId: string)

2. Add event listeners for AI-generated plans:

   static listenForToolPlan(aiSessionId: string, thinkingMessageId: number, mcpSessionId: string)

3. Sequential execution of AI-generated tool plans:

   static async executeToolPlan(toolPlan: any, sessionId: string, messageId: number)

Phase 3: UI/UX Improvements

1. Progress indicators during multi-step execution
2. Plan preview before execution (for complex requests)
3. Better error handling and recovery
4. Approval system for high-risk tool combinations

Files to Modify

Backend (Rust)

• src-tauri/src/ollama.rs - Add generate_mcp_tool_plan command
• src-tauri/src/system_prompts.rs - Add tool planning prompt
• src-tauri/src/lib.rs - Register new command

Frontend (TypeScript)

• src/composables/mcpService.ts - Replace pattern matching with AI planning
• src/composables/agentService.ts - Update @enteract routing if needed

Example Improvements

Before (Pattern Matching)

User: "@enteract take a screenshot and click on the login button"
Result: ❌ Only takes screenshot (can't parse "click on login button")

After (AI-Driven)

User: "@enteract take a screenshot and click on the login button"
AI Plan: 
  1. TOOL_CALL: take_screenshot {}
  2. TOOL_CALL: find_text {"text": "login"}  
  3. TOOL_CALL: click_on_text {"text": "login"}
Result: ✅ Takes screenshot, finds login button, clicks it

Testing Requirements

Unit Tests

• AI tool plan generation with various request types
• Tool plan parsing and validation
• Sequential execution error handling
• Event emission and listening

Integration Tests

• End-to-end @enteract flow with AI planning
• Complex multi-step tool execution
• Error recovery and user feedback
• Performance with large tool sets

User Testing

• Complex request handling (screenshots + clicks + typing)
• Natural language tool requests
• Error scenarios and user experience
• Performance and responsiveness

Success Criteria

1. Functionality: @enteract can handle complex multi-step requests like "take a screenshot, find the search box, type 'hello world', and press enter"

2. Performance: AI planning completes within 2-3 seconds for typical requests

3. User Experience: Clear progress indicators and error messages for tool execution

4. Reliability: 95%+ success rate for well-formed tool requests

5. Backward Compatibility: Existing simple @enteract commands continue to work

Priority

High Priority - This directly impacts the core user experience of the MCP system and unlocks significantly more powerful automation capabilities.

Estimated Effort

2-3 days for core implementation + testing

• Day 1: Backend AI planning system
• Day 2: Frontend integration and event handling
• Day 3: Testing, refinement, and UX improvements