Skip to content

[PowerOCR][WIP] migrate PowerOCR to WinUI 3 - #49431

Draft
moooyo wants to merge 29 commits into
mainfrom
yuleng/powerocr-winui3
Draft

[PowerOCR][WIP] migrate PowerOCR to WinUI 3#49431
moooyo wants to merge 29 commits into
mainfrom
yuleng/powerocr-winui3

Conversation

@moooyo

@moooyo moooyo commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary of the Pull Request

Migrates Text Extractor (PowerOCR) from WPF to WinUI 3 while preserving its Runner integration, settings contract, activation behavior, telemetry, and UI Automation identifiers.

  • Replaces the WPF overlay with native WinUI 3 WindowEx overlays, one per display, with mixed-DPI and negative-origin coordinate handling.
  • Extracts OCR, bitmap preparation, text formatting, table analysis, and selection geometry into the UI-independent PowerOCR.Core library.
  • Migrates localization from .resx to .resw/x:Uid, and updates process lifetime, Named Event dispatch, clipboard handling, and deterministic resource cleanup.
  • Moves managed Text Extractor outputs into WinUI3Apps and updates the native launcher, signing manifest, installer resources, and installation verification list.
  • Adds Core unit tests and updates the existing PowerOCR UI automation/checklist for the WinUI controls.

Part of the WinUI 3 migration effort tracked in #46463.

PR Checklist

  • Closes: Migrate Text Extractor to WinUI3 #49656
  • Communication: I've discussed this with core contributors already. If the work hasn't been agreed, this work might be rejected
  • Tests: Added/updated and all pass — Core tests pass, but interactive UI tests and several visual/multi-monitor checks remain environment-blocked as detailed below
  • Localization: All end-user-facing strings can be localized
  • Dev docs: Added/updated
  • New binaries: Added on the required places
    • JSON for signing for PowerToys.PowerOCR.Core.dll and the WinUI3Apps paths
    • WXS for installer — handled by the existing WinUI3Apps component generator; removed the obsolete PowerOCR satellite-resource component from installer/PowerToysSetupVNext/Resources.wxs
    • YML for CI pipeline — N/A; the new test project is included in PowerToys.slnx and uses existing test discovery
    • YML for signed pipeline — N/A; no per-module pipeline entry is required
  • Documentation updated: If checked, please file a pull request on our docs repo and link it here: #xxx — N/A; no end-user behavior change

Detailed Description of the Pull Request / Additional comments

Core and project structure

  • Adds src/modules/PowerOCR/PowerOCR.Core/ and PowerOCR.Core.UnitTests/.
  • Removes production UseWPF, System.Windows, WPF imaging types, MEF construction, and the WPF-only Common.UI dependency.
  • Converts src/modules/PowerOCR/PowerOCR/PowerOCR.csproj to an unpackaged, self-contained WinUI 3 application with XAML under PowerOCRXAML/.

Overlay and OCR flow

  • Captures every display before showing overlays, then creates one borderless topmost overlay per DisplayArea.
  • Uses cached physical-pixel screenshots for clicked-word, region, single-line, and table OCR modes.
  • Converts WinUI DIP pointer coordinates through each window's rasterization scale and supports displays with negative virtual-desktop origins.
  • Replaces the WPF CombinedGeometry selection clip with four mask rectangles plus a selection Border.
  • Synchronizes language and formatting state across all overlay windows and preserves Escape, S, T, and D1-D9 keyboard behavior.

Lifetime and compatibility

  • Preserves the existing process name, mutex, Runner PID argument, GPO behavior, Named Events, standalone keyboard hook, settings JSON, telemetry events, and existing AutomationIds.
  • Adds cancellation-aware Named Event handling and deterministic disposal for windows, screenshots, WinRT image sources, OCR crops, pointer capture, and cursor clipping.
  • Keeps PowerOCRModuleInterface.dll in the install root and launches WinUI3Apps\PowerToys.PowerOCR.exe.

Validation Steps Performed

  • Built PowerOCR.Core.UnitTests for x64 Debug and ran all tests with vstest.console.exe: 23/23 passed.
  • Built PowerOCR-UITests for x64 Debug successfully.
  • Built PowerOCR for x64 Release and ARM64 Release successfully.
  • Built PowerOCRModuleInterface for x64 Release and ARM64 Release successfully.
  • Ran XAML Styler against the branch: 3 PowerOCR XAML files processed with no changes.
  • Confirmed the required x64/ARM64 outputs, including PowerToys.PowerOCR.Core.dll and PowerToys.PowerOCR.pri, are generated in the expected locations.
  • Ran build+sideload checklist validation against the local WinUI 3 bits: 15 PASS, 0 FAIL, 16 BLOCKED.
    • Blocked items require an attached interactive desktop, WinAppDriver v1.2.1, multiple displays/mixed DPI, real pointer drag/input, or visual theme inspection.
    • No product defects were observed in the checks that could be executed.
  • The legacy UI test runtime remains blocked because WinAppDriver is unavailable and the current RDP input desktop is detached; the tests compile successfully.

Yu Leng (from Dev Box) and others added 23 commits July 21, 2026 15:11
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
- Add IBitmapPreprocessor / BitmapPreprocessor with padding and scale logic
- Add PreparedBitmap value holder (IDisposable)
- Add IOcrRecognizer interface and WindowsOcrRecognizer adapter (Windows.Media.Ocr)
- Add OcrExtractionRequest record and ITextExtractorService interface
- Add TextExtractorService orchestrating preprocessing, OCR, and formatting
- Add BitmapPreprocessorTests (5 tests) and TextExtractorServiceTests (6 tests)
- All 21 unit tests pass

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Replace the WPF application shell with a headless WinUI 3 process foundation:
- New csproj targeting WinUI 3 / Windows App SDK with AOT compatibility
- Explicit Program.cs entry point with GPO check, mutex, and Runner PID
- WinUI App.xaml composition root with DI (Microsoft.Extensions.DependencyInjection)
- ActivationService + NativeEventListener for named event handling
- ThrottledActionInvoker rewritten with DispatcherQueueTimer
- KeyboardMonitor updated to use IActivationService
- UserSettings/IUserSettings implement IDisposable, MEF removed
- SettingsDeepLink for PowerOCR-local settings launch
- ResourceLoaderInstance for .resw-based localization
- Resources.resw with all UI strings for Task 4 overlay
- Deleted all WPF production files (overlay, imaging, models, event monitor)
- Simplified app.manifest for DPI awareness

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Implement Task 4: display capture and multi-window WinUI overlays.

- DisplayCapture model owns GDI+ bitmap and SoftwareBitmapSource
- IScreenCaptureService captures each DisplayArea via CopyFromScreen
- OverlayManager subscribes IActivationService, enumerates displays,
  captures all before showing, handles capture failure with error overlay
- OCROverlay WindowEx: borderless topmost per-display window
- OverlayPage: screenshot + dim layer + InfoBar + Cancel button
- OverlayWindowFactory uses ActivatorUtilities.CreateInstance
- Services registered in App.OnLaunched DI container

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
…c handler

- Dispose SoftwareBitmapSource in DisplayCapture.Dispose(); IDisposable confirmed by successful compile
- Guard window creation/activation in OverlayManager.ShowAsync with try/catch; on failure close created windows, dispose captures, cancel/dispose CTS, reset _sessionActive
- Replace fire-and-forget ShowAsync call with async void handler that awaits ShowAsync and catches/logs unexpected exceptions, calling CloseAll to recover session state

Co-authored-by: Copilot <[email protected]>
Co-authored-by: Copilot <[email protected]>`nCopilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Implement selection, commands, OCR, clipboard, and settings for the WinUI PowerOCR overlays (Task 5).

- Add OverlaySessionViewModel with observable language/mode/state properties
- Add IClipboardService/ClipboardService for WinRT clipboard writes
- Add CursorClipper for physical cursor clipping via P/Invoke
- Replace minimal page with full four-mask selection UI, toolbar, and context menu
- Implement pointer capture with Shift-translate, DPI mapping via SelectionGeometry
- Implement keyboard shortcuts (Escape/S/T/1-9), dynamic language flyout
- Complete OCR orchestration: word/region/table/single-line modes
- Register ClipboardService and transient OverlaySessionViewModel in DI
- Add localized CancelMenuItem text, x:Uid for all buttons
- Replace hardcoded Cancel with localized toolbar icons

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
- Add PointerCaptureLost and PointerCanceled handlers on the selection Canvas so that losing capture mid-drag resets _isSelecting, restores the toolbar, releases any remaining capture, and calls CursorClipper.UnClip().
- Refactor the duplicate cleanup code in PointerReleased into a shared EndSelectionCleanup(Pointer?) helper used by all three termination paths.
- Add CursorClipper.UnClip() at the start of OverlayManager.CloseAll, covering Escape, settings, successful capture, native termination, and external window-close paths.

Co-authored-by: Copilot <[email protected]>
Dispose region/table/single-line crops on OCR failure or cancellation while leaving the shared word-mode capture bitmap owned by the session.

Co-authored-by: Copilot <[email protected]>
- Replace class-name/index OCR language lookup with By.AccessibilityId("OCRLanguageMenuItem_0") for stable selection
- Add PowerOCR.ToolbarModes test: verifies SingleLineToggleButton and TableToggleButton Selected state and Escape dismissal
- Stabilize TextSelectionAndClipboardTest: use RegionClickCanvas + Pane.Drag(bounds-aware) instead of raw Selenium drag from (0,0)
- Remove PerformSeleniumDrag helper and unused OpenQA.Selenium.Interactions import
- Expand tests-checklist-text-extractor.md: add WinUI theming, toolbar a11y, capture modes, multi-monitor, mixed DPI, activation paths, and settings deep link

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
…er, and verification paths

- Update native launcher to WinUI3Apps\PowerToys.PowerOCR.exe
- Update ESRP signing entries for WinUI3Apps paths + Core.dll
- Move verification entries from criticalSignedFiles to winUI3SignedFiles
- Remove obsolete satellite assembly component from Resources.wxs
- Regenerate WinUI3Applications.wxs with PowerOCR file components

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
Address final whole-branch review findings on the WinUI 3 migration:

- Recompute overlay full-cover masks from real Loaded/SizeChanged dimensions instead of a 1920x1080 pre-layout fallback, so 1440p/4K lower areas dim on the first rendered layout; preserve in-progress selection visuals on later size events.
- Split OverlayManager.CaptureAsync OCR and clipboard error boundaries so OCR failures always surface OcrFailed and clipboard failures surface ClipboardFailed, preserving cancellation and IsProcessing behavior.
- Restore OrdinalIgnoreCase for preferred-language NativeName matching to match the original WPF behavior.
- Re-check _sessionActive and the session token after asynchronous display capture and dispose locally captured images if the session was torn down, before creating overlay windows.

Co-authored-by: Copilot <[email protected]>
Copilot-Session: 64307790-5118-4071-9308-5e9ee9836310
@github-actions github-actions Bot added Area-Localization issues regarding to Localization the application Product-Text Extractor This refers to the Text Extractor PowerToy labels Jul 21, 2026
@moooyo moooyo changed the title refactor(text-extractor): migrate PowerOCR to WinUI 3 [PowerOCR][WIP] migrate PowerOCR to WinUI 3 Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area-Localization issues regarding to Localization the application Product-Text Extractor This refers to the Text Extractor PowerToy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate Text Extractor to WinUI3

2 participants