[PowerOCR][WIP] migrate PowerOCR to WinUI 3 - #49431
Draft
moooyo wants to merge 29 commits into
Draft
Conversation
Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
- Add IBitmapPreprocessor / BitmapPreprocessor with padding and scale logic - Add PreparedBitmap value holder (IDisposable) - Add IOcrRecognizer interface and WindowsOcrRecognizer adapter (Windows.Media.Ocr) - Add OcrExtractionRequest record and ITextExtractorService interface - Add TextExtractorService orchestrating preprocessing, OCR, and formatting - Add BitmapPreprocessorTests (5 tests) and TextExtractorServiceTests (6 tests) - All 21 unit tests pass Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Replace the WPF application shell with a headless WinUI 3 process foundation: - New csproj targeting WinUI 3 / Windows App SDK with AOT compatibility - Explicit Program.cs entry point with GPO check, mutex, and Runner PID - WinUI App.xaml composition root with DI (Microsoft.Extensions.DependencyInjection) - ActivationService + NativeEventListener for named event handling - ThrottledActionInvoker rewritten with DispatcherQueueTimer - KeyboardMonitor updated to use IActivationService - UserSettings/IUserSettings implement IDisposable, MEF removed - SettingsDeepLink for PowerOCR-local settings launch - ResourceLoaderInstance for .resw-based localization - Resources.resw with all UI strings for Task 4 overlay - Deleted all WPF production files (overlay, imaging, models, event monitor) - Simplified app.manifest for DPI awareness Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Implement Task 4: display capture and multi-window WinUI overlays. - DisplayCapture model owns GDI+ bitmap and SoftwareBitmapSource - IScreenCaptureService captures each DisplayArea via CopyFromScreen - OverlayManager subscribes IActivationService, enumerates displays, captures all before showing, handles capture failure with error overlay - OCROverlay WindowEx: borderless topmost per-display window - OverlayPage: screenshot + dim layer + InfoBar + Cancel button - OverlayWindowFactory uses ActivatorUtilities.CreateInstance - Services registered in App.OnLaunched DI container Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
…c handler - Dispose SoftwareBitmapSource in DisplayCapture.Dispose(); IDisposable confirmed by successful compile - Guard window creation/activation in OverlayManager.ShowAsync with try/catch; on failure close created windows, dispose captures, cancel/dispose CTS, reset _sessionActive - Replace fire-and-forget ShowAsync call with async void handler that awaits ShowAsync and catches/logs unexpected exceptions, calling CloseAll to recover session state Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>`nCopilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Implement selection, commands, OCR, clipboard, and settings for the WinUI PowerOCR overlays (Task 5). - Add OverlaySessionViewModel with observable language/mode/state properties - Add IClipboardService/ClipboardService for WinRT clipboard writes - Add CursorClipper for physical cursor clipping via P/Invoke - Replace minimal page with full four-mask selection UI, toolbar, and context menu - Implement pointer capture with Shift-translate, DPI mapping via SelectionGeometry - Implement keyboard shortcuts (Escape/S/T/1-9), dynamic language flyout - Complete OCR orchestration: word/region/table/single-line modes - Register ClipboardService and transient OverlaySessionViewModel in DI - Add localized CancelMenuItem text, x:Uid for all buttons - Replace hardcoded Cancel with localized toolbar icons Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
- Add PointerCaptureLost and PointerCanceled handlers on the selection Canvas so that losing capture mid-drag resets _isSelecting, restores the toolbar, releases any remaining capture, and calls CursorClipper.UnClip(). - Refactor the duplicate cleanup code in PointerReleased into a shared EndSelectionCleanup(Pointer?) helper used by all three termination paths. - Add CursorClipper.UnClip() at the start of OverlayManager.CloseAll, covering Escape, settings, successful capture, native termination, and external window-close paths. Co-authored-by: Copilot <[email protected]>
Dispose region/table/single-line crops on OCR failure or cancellation while leaving the shared word-mode capture bitmap owned by the session. Co-authored-by: Copilot <[email protected]>
- Replace class-name/index OCR language lookup with By.AccessibilityId("OCRLanguageMenuItem_0") for stable selection
- Add PowerOCR.ToolbarModes test: verifies SingleLineToggleButton and TableToggleButton Selected state and Escape dismissal
- Stabilize TextSelectionAndClipboardTest: use RegionClickCanvas + Pane.Drag(bounds-aware) instead of raw Selenium drag from (0,0)
- Remove PerformSeleniumDrag helper and unused OpenQA.Selenium.Interactions import
- Expand tests-checklist-text-extractor.md: add WinUI theming, toolbar a11y, capture modes, multi-monitor, mixed DPI, activation paths, and settings deep link
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
…er, and verification paths - Update native launcher to WinUI3Apps\PowerToys.PowerOCR.exe - Update ESRP signing entries for WinUI3Apps paths + Core.dll - Move verification entries from criticalSignedFiles to winUI3SignedFiles - Remove obsolete satellite assembly component from Resources.wxs - Regenerate WinUI3Applications.wxs with PowerOCR file components Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Address final whole-branch review findings on the WinUI 3 migration: - Recompute overlay full-cover masks from real Loaded/SizeChanged dimensions instead of a 1920x1080 pre-layout fallback, so 1440p/4K lower areas dim on the first rendered layout; preserve in-progress selection visuals on later size events. - Split OverlayManager.CaptureAsync OCR and clipboard error boundaries so OCR failures always surface OcrFailed and clipboard failures surface ClipboardFailed, preserving cancellation and IsProcessing behavior. - Restore OrdinalIgnoreCase for preferred-language NativeName matching to match the original WPF behavior. - Re-check _sessionActive and the session token after asynchronous display capture and dispose locally captured images if the session was torn down, before creating overlay windows. Co-authored-by: Copilot <[email protected]> Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]> Copilot-Session: e5ba1f54-e354-456c-bd2a-c716a143eb97
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of the Pull Request
Migrates Text Extractor (
PowerOCR) from WPF to WinUI 3 while preserving its Runner integration, settings contract, activation behavior, telemetry, and UI Automation identifiers.WindowExoverlays, one per display, with mixed-DPI and negative-origin coordinate handling.PowerOCR.Corelibrary..resxto.resw/x:Uid, and updates process lifetime, Named Event dispatch, clipboard handling, and deterministic resource cleanup.WinUI3Appsand updates the native launcher, signing manifest, installer resources, and installation verification list.Part of the WinUI 3 migration effort tracked in #46463.
PR Checklist
PowerToys.PowerOCR.Core.dlland the WinUI3Apps pathsinstaller/PowerToysSetupVNext/Resources.wxsPowerToys.slnxand uses existing test discoveryDetailed Description of the Pull Request / Additional comments
Core and project structure
src/modules/PowerOCR/PowerOCR.Core/andPowerOCR.Core.UnitTests/.UseWPF,System.Windows, WPF imaging types, MEF construction, and the WPF-onlyCommon.UIdependency.src/modules/PowerOCR/PowerOCR/PowerOCR.csprojto an unpackaged, self-contained WinUI 3 application with XAML underPowerOCRXAML/.Overlay and OCR flow
DisplayArea.CombinedGeometryselection clip with four mask rectangles plus a selectionBorder.Lifetime and compatibility
PowerOCRModuleInterface.dllin the install root and launchesWinUI3Apps\PowerToys.PowerOCR.exe.Validation Steps Performed
PowerOCR.Core.UnitTestsfor x64 Debug and ran all tests withvstest.console.exe: 23/23 passed.PowerOCR-UITestsfor x64 Debug successfully.PowerOCRfor x64 Release and ARM64 Release successfully.PowerOCRModuleInterfacefor x64 Release and ARM64 Release successfully.PowerToys.PowerOCR.Core.dllandPowerToys.PowerOCR.pri, are generated in the expected locations.