feat(desktop-context): add approved desktop context capsules#388
Draft
hiqiancheng wants to merge 4 commits into
Draft
feat(desktop-context): add approved desktop context capsules#388hiqiancheng wants to merge 4 commits into
hiqiancheng wants to merge 4 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the approved read-only desktop context path for TouchAI invocation context:
builtin__get_desktop_contextapprovalinclude: string[]fields for safe metadata, selected text, clipboard, screenshot metadata/image, capabilities, and redactionsbuiltin__get_desktop_contexttool-call expansion instead of the user message bubble, with screenshot preview first and raw complete tool JSON after itxcap, persists them under the desktop-context artifact store, attaches image results to the model, and updates turn artifacts for history replaymain,popup-*,tray-menu) during capture and restore them immediately, preserving remote desktop and screen sharing compatibilityRelated issue or RFC
AI assistance disclosure
Testing evidence
Commands run locally:
pnpm --filter @touchai/desktop test:e2ebuilt the desktop app successfully, then failed before running specs because the local machine does not havemsedgedriver.exeinstalled/configured:TDD: the latest follow-up added focused regression tests for default selected-text redaction, prompt injection, approval boundaries, user-message rendering removal, and desktop-context tool-call expansion before the final implementation was completed. Earlier feature work was not fully test-first.
Risk notes
AgentService, runtime, MCP, or schema impact: AgentService prompt/runtime/tool execution now carries an optional bound desktop context capsule for the current turn; built-in tool execution can persist an updated approved screenshot artifact after tool approval.session_turn_context_artifactstable and updates artifact upsert behavior so approved screenshot metadata can replace the pending metadata for the bound capsule.xcapfor screenshot/window capture; Windows selected-text support uses UI Automation plus read-only Win32 text-control fallback; macOS/Linux selected-text extraction returns explicit unsupported/permission reasons until a verified platform backend is added.set_content_protected(true)from the search surface; screenshot capture briefly hides and restores TouchAI transient windows, with blur suppression during the capture window to avoid treating the temporary hide as a user dismiss.Screenshots or recordings
Not included. This change affects tool-call rendering and approval/runtime behavior; the new desktop-context tool expansion is covered by
tests/components/DesktopContextToolCallItem.test.ts.Checklist
[WIP]or similar title prefixes.AgentService, runtime, MCP, or schema boundaries, there is an accepted RFC.pnpm test:prfor this code PR, or this is a docs-only change.pnpm test:coverage:rustor relied on CI coverage evidence.pnpm test:e2elocally or documented why CI is the first valid proof.