Skip to content

Commit 34992ef

Browse files
JE-ChenJE-Chen
committed
Add advanced TextPattern: find / select / read text attributes
ax_text shipped whole-range reads but couldn't search a control's text, select a found range, or read formatting. Round out TextPattern: find_control_text searches real content (not OCR) via FindText, select_control_text finds + selects a range so the next keystrokes replace it, and control_text_attributes reads font/size/bold/italic/ colour. Extends the backend ABC + Windows UIA backend and the ax_text facade.
1 parent c44cb3b commit 34992ef

13 files changed

Lines changed: 388 additions & 14 deletions

File tree

‎WHATS_NEW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-25) — Advanced TextPattern (find / select / read attributes)
4+
5+
Search a control's text, select a match to replace it, and read font/colour formatting. Full reference: [`docs/source/Eng/doc/new_features/v200_features_doc.rst`](docs/source/Eng/doc/new_features/v200_features_doc.rst).
6+
7+
- **`find_control_text` / `select_control_text` / `control_text_attributes`** (`AC_find_control_text`, `AC_select_control_text`, `AC_control_text_attributes`): `ax_text` shipped the three whole-range *reads*, but couldn't search for a substring, select a found range, or read text formatting — needed to assert "the error word is red and bold" or to place the selection at matched text before typing. This rounds out TextPattern: `find_control_text` searches the real content (not OCR) via `FindText`, `select_control_text` finds + selects a range so the next keystrokes replace it, and `control_text_attributes` reads `{font_name, font_size, bold, italic, foreground_color}`. Dispatched through the injectable accessibility backend seam (headless-testable via a fake backend; real UIA in the Windows backend). No `PySide6`.
8+
39
## What's new (2026-06-25) — MSAA Bridge for Legacy Controls (LegacyIAccessible)
410

511
Automate the long tail of old Win32 controls that expose nothing via modern UIA. Full reference: [`docs/source/Eng/doc/new_features/v199_features_doc.rst`](docs/source/Eng/doc/new_features/v199_features_doc.rst).

‎docs/source/Eng/doc/new_features/v200_features_doc.rst‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
Advanced TextPattern — Find / Select / Read Attributes
2+
======================================================
3+
4+
``ax_text`` shipped the three whole-range *reads* (document / selection / visible
5+
text). It could not **search** for a substring, **select** a found range, or read
6+
text **formatting attributes** — needed to assert "the error word is red and
7+
bold" or to place the caret / selection at matched text before typing. This rounds
8+
TextPattern out from "dump the text" to "interrogate and manipulate" it.
9+
10+
* :func:`find_control_text` — whether ``text`` occurs in the control
11+
(TextPattern.FindText, searches the real content, not OCR),
12+
* :func:`select_control_text` — find ``text`` and select its range, so the next
13+
keystrokes replace it (FindText + Select),
14+
* :func:`control_text_attributes` — the selection's font / colour formatting.
15+
16+
Each is a thin dispatch onto the injectable ``accessibility.backends.get_backend()``
17+
seam — headless-testable via a fake backend; the real UIA calls live in the
18+
Windows backend. Imports no ``PySide6``.
19+
20+
Headless API
21+
------------
22+
23+
.. code-block:: python
24+
25+
from je_auto_control import (find_control_text, select_control_text,
26+
control_text_attributes, type_text)
27+
28+
if find_control_text("TODO", name="Editor"):
29+
select_control_text("TODO", name="Editor") # selection now spans "TODO"
30+
type_text("DONE") # replaces it
31+
32+
control_text_attributes(name="Editor")
33+
# {"font_name": "Consolas", "font_size": 11.0, "bold": True,
34+
# "italic": False, "foreground_color": 16711680}
35+
36+
``ignore_case`` (default ``True``) controls the search. The control is located by
37+
``name`` / ``role`` / ``app_name`` / ``automation_id`` (same as the other
38+
TextPattern reads). ``find_control_text`` / ``select_control_text`` return
39+
``bool``; ``control_text_attributes`` returns the formatting dict (values may be
40+
``None`` where the range spans mixed formatting) or ``None`` if not found.
41+
42+
Executor commands
43+
-----------------
44+
45+
``AC_find_control_text`` / ``AC_select_control_text`` (``text`` / ``ignore_case``)
46+
and ``AC_control_text_attributes`` (``{found, attributes}``). They are exposed as
47+
the matching ``ac_*`` MCP tools (find / attributes read-only, select destructive)
48+
and as Script Builder commands under **Native UI**.

‎docs/source/Zh/doc/new_features/v200_features_doc.rst‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
進階 TextPattern——搜尋 / 選取 / 讀取屬性
2+
========================================
3+
4+
``ax_text`` 先前提供三種整段*讀取*(document / selection / visible 文字)。它無法**搜尋**子字串、
5+
**選取**找到的範圍,或讀取文字的**格式屬性**——而這些正是斷言「錯誤字是紅色且粗體」或在輸入前
6+
把游標 / 選取定位到匹配文字所需。本次把 TextPattern 從「傾印文字」擴充為「查詢與操作」文字。
7+
8+
* :func:`find_control_text` ——``text`` 是否出現在控制項中(TextPattern.FindText,搜尋真正的
9+
內容,而非 OCR),
10+
* :func:`select_control_text` ——找到 ``text`` 並選取其範圍,讓接下來的按鍵取代它
11+
(FindText + Select),
12+
* :func:`control_text_attributes` ——選取範圍的字型 / 顏色格式。
13+
14+
每個都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派——可透過注入 fake
15+
backend 進行無頭測試;真正的 UIA 呼叫位於 Windows 後端。不匯入 ``PySide6``。
16+
17+
無頭 API
18+
--------
19+
20+
.. code-block:: python
21+
22+
from je_auto_control import (find_control_text, select_control_text,
23+
control_text_attributes, type_text)
24+
25+
if find_control_text("TODO", name="Editor"):
26+
select_control_text("TODO", name="Editor") # 選取範圍現在涵蓋 "TODO"
27+
type_text("DONE") # 取代它
28+
29+
control_text_attributes(name="Editor")
30+
# {"font_name": "Consolas", "font_size": 11.0, "bold": True,
31+
# "italic": False, "foreground_color": 16711680}
32+
33+
``ignore_case``(預設 ``True``)控制搜尋。控制項以 ``name`` / ``role`` / ``app_name`` /
34+
``automation_id`` 定位(與其他 TextPattern 讀取相同)。``find_control_text`` /
35+
``select_control_text`` 回傳 ``bool``;``control_text_attributes`` 回傳格式字典(範圍跨越混合
36+
格式時某些值可能為 ``None``),找不到則回傳 ``None``。
37+
38+
執行器指令
39+
----------
40+
41+
``AC_find_control_text`` / ``AC_select_control_text``(``text`` / ``ignore_case``)與
42+
``AC_control_text_attributes``(``{found, attributes}``)。皆以對應的 ``ac_*`` MCP 工具
43+
(find / attributes 為唯讀、select 為破壞性)及 Script Builder 指令(位於 **Native UI** 分類下)
44+
形式提供。

‎je_auto_control/__init__.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,9 +53,10 @@
5353
collapse_control, control_expand_state, control_range, expand_control,
5454
scroll_control_into_view, select_control_item, set_control_range,
5555
)
56-
# Native text reads via UIA TextPattern (document / selection / visible)
56+
# Native text via UIA TextPattern (read / find / select / attributes)
5757
from je_auto_control.utils.ax_text import (
58-
get_control_text, get_selected_text, get_visible_text,
58+
control_text_attributes, find_control_text, get_control_text,
59+
get_selected_text, get_visible_text, select_control_text,
5960
)
6061
# Readable / addressable a11y-tree post-processing (role names + node paths)
6162
from je_auto_control.utils.ax_tree_walk import (
@@ -1681,6 +1682,7 @@ def start_autocontrol_gui(*args, **kwargs):
16811682
"select_control_item", "control_range", "set_control_range",
16821683
"scroll_control_into_view",
16831684
"get_control_text", "get_selected_text", "get_visible_text",
1685+
"find_control_text", "select_control_text", "control_text_attributes",
16841686
"control_type_name", "humanize_role", "humanize_tree",
16851687
"assign_node_paths", "find_by_path",
16861688
"is_interactive_role", "tab_order", "audit_focus_order", "focus_control",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1681,6 +1681,25 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:
16811681
fields=fields,
16821682
description="Read full text via TextPattern (multiline / document safe).",
16831683
))
1684+
specs.append(CommandSpec(
1685+
"AC_find_control_text", "Native UI", "Find Text in Control",
1686+
fields=(FieldSpec("text", FieldType.STRING),
1687+
FieldSpec("ignore_case", FieldType.BOOL, optional=True,
1688+
default=True)) + fields,
1689+
description="Whether text occurs in a control (TextPattern.FindText).",
1690+
))
1691+
specs.append(CommandSpec(
1692+
"AC_select_control_text", "Native UI", "Select Text in Control",
1693+
fields=(FieldSpec("text", FieldType.STRING),
1694+
FieldSpec("ignore_case", FieldType.BOOL, optional=True,
1695+
default=True)) + fields,
1696+
description="Find + select text in a control (FindText + Select).",
1697+
))
1698+
specs.append(CommandSpec(
1699+
"AC_control_text_attributes", "Native UI", "Get Text Attributes",
1700+
fields=fields,
1701+
description="Read selection formatting (font/size/bold/italic/colour).",
1702+
))
16841703
specs.append(CommandSpec(
16851704
"AC_get_selected_text", "Native UI", "Get Selected Text",
16861705
fields=fields,

‎je_auto_control/utils/accessibility/backends/base.py‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,28 @@ def visible_text(self, name: Optional[str] = None, role: Optional[str] = None,
123123
"""Return only the on-screen text of the control (TextPattern), or None."""
124124
self._unsupported("visible_text")
125125

126+
def find_text(self, text: str = "", ignore_case: bool = True,
127+
name: Optional[str] = None, role: Optional[str] = None,
128+
app_name: Optional[str] = None,
129+
automation_id: Optional[str] = None) -> bool:
130+
"""Return whether ``text`` occurs in the control (TextPattern.FindText)."""
131+
self._unsupported("find_text")
132+
133+
def select_text(self, text: str = "", ignore_case: bool = True,
134+
name: Optional[str] = None, role: Optional[str] = None,
135+
app_name: Optional[str] = None,
136+
automation_id: Optional[str] = None) -> bool:
137+
"""Find ``text`` and select its range (TextPattern.FindText + Select)."""
138+
self._unsupported("select_text")
139+
140+
def text_attributes(self, name: Optional[str] = None,
141+
role: Optional[str] = None, app_name: Optional[str] = None,
142+
automation_id: Optional[str] = None,
143+
) -> Optional[Dict[str, Any]]:
144+
"""Return formatting of the control's selection — ``{font_name, font_size,
145+
bold, italic, foreground_color}`` (TextPattern attributes), or None."""
146+
self._unsupported("text_attributes")
147+
126148
# --- keyboard focus ----------------------------------------------------
127149

128150
def set_focus(self, name: Optional[str] = None, role: Optional[str] = None,

‎je_auto_control/utils/accessibility/backends/windows_backend.py‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,48 @@ def visible_text(self, name=None, role=None, app_name=None,
445445
except (OSError, AttributeError):
446446
return None
447447

448+
def _find_range(self, text, ignore_case, name, role, app_name, automation_id):
449+
"""Find ``text`` in the control's document range (TextPattern.FindText)."""
450+
pattern = self._text_pattern(name, role, app_name, automation_id)
451+
if pattern is None:
452+
return None
453+
try:
454+
return pattern.DocumentRange.FindText(str(text), False,
455+
bool(ignore_case))
456+
except (OSError, AttributeError):
457+
return None
458+
459+
def find_text(self, text="", ignore_case=True, name=None, role=None,
460+
app_name=None, automation_id=None) -> bool:
461+
return self._find_range(text, ignore_case, name, role, app_name,
462+
automation_id) is not None
463+
464+
def select_text(self, text="", ignore_case=True, name=None, role=None,
465+
app_name=None, automation_id=None) -> bool:
466+
found = self._find_range(text, ignore_case, name, role, app_name,
467+
automation_id)
468+
if not found:
469+
return False
470+
try:
471+
found.Select()
472+
return True
473+
except (OSError, AttributeError):
474+
return False
475+
476+
def text_attributes(self, name=None, role=None, app_name=None,
477+
automation_id=None) -> Optional[Dict[str, Any]]:
478+
pattern = self._text_pattern(name, role, app_name, automation_id)
479+
if pattern is None:
480+
return None
481+
try:
482+
selection = pattern.GetSelection()
483+
text_range = (selection.GetElement(0)
484+
if selection and int(selection.Length or 0) > 0
485+
else pattern.DocumentRange)
486+
except (OSError, AttributeError):
487+
return None
488+
return _read_text_attributes(text_range)
489+
448490
def set_focus(self, name=None, role=None, app_name=None,
449491
automation_id=None) -> bool:
450492
raw = self._find_raw(name, role, app_name, automation_id)
@@ -512,6 +554,33 @@ def _as_text(value) -> str:
512554
return str(value or "")
513555

514556

557+
# UIA TextPattern attribute ids (UIAutomationClient AttributeId range).
558+
_TEXT_ATTR_FONT_NAME = 40005
559+
_TEXT_ATTR_FONT_SIZE = 40006
560+
_TEXT_ATTR_FONT_WEIGHT = 40007
561+
_TEXT_ATTR_FOREGROUND = 40008
562+
_TEXT_ATTR_IS_ITALIC = 40014
563+
564+
565+
def _attr(text_range, attribute_id, cast):
566+
try:
567+
return cast(text_range.GetAttributeValue(attribute_id))
568+
except (OSError, AttributeError, ValueError, TypeError):
569+
return None
570+
571+
572+
def _read_text_attributes(text_range) -> Dict[str, Any]:
573+
"""Read font / colour formatting of a TextRange into a plain dict."""
574+
weight = _attr(text_range, _TEXT_ATTR_FONT_WEIGHT, int)
575+
return {
576+
"font_name": _attr(text_range, _TEXT_ATTR_FONT_NAME, _as_text),
577+
"font_size": _attr(text_range, _TEXT_ATTR_FONT_SIZE, float),
578+
"bold": (weight >= 700) if isinstance(weight, int) else None,
579+
"italic": _attr(text_range, _TEXT_ATTR_IS_ITALIC, bool),
580+
"foreground_color": _attr(text_range, _TEXT_ATTR_FOREGROUND, int),
581+
}
582+
583+
515584
# (key, LegacyIAccessiblePattern attribute, cast) for the MSAA bridge read.
516585
_LEGACY_READS = (
517586
("name", "CurrentName", _as_text),

‎je_auto_control/utils/ax_text/__init__.py‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1-
"""Native text reading via the UI Automation TextPattern (document/selection/visible)."""
1+
"""Native text via the UI Automation TextPattern (read / find / select / attributes)."""
22
from je_auto_control.utils.ax_text.ax_text import (
3-
get_control_text, get_selected_text, get_visible_text,
3+
control_text_attributes, find_control_text, get_control_text,
4+
get_selected_text, get_visible_text, select_control_text,
45
)
56

67
__all__ = [
78
"get_control_text", "get_selected_text", "get_visible_text",
9+
"find_control_text", "select_control_text", "control_text_attributes",
810
]

‎je_auto_control/utils/ax_text/ax_text.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
platform by injecting a fake backend; the real UIA calls live in the Windows
1818
backend. Imports no ``PySide6``.
1919
"""
20-
from typing import Optional
20+
from typing import Any, Dict, Optional
2121

2222

2323
def _backend():
@@ -59,3 +59,34 @@ def get_visible_text(name: Optional[str] = None, role: Optional[str] = None,
5959
"""
6060
return _backend().visible_text(name=name, role=role, app_name=app_name,
6161
automation_id=automation_id)
62+
63+
64+
def find_control_text(text: str, *, ignore_case: bool = True,
65+
name: Optional[str] = None, role: Optional[str] = None,
66+
app_name: Optional[str] = None,
67+
automation_id: Optional[str] = None) -> bool:
68+
"""Return whether ``text`` occurs in the control (TextPattern.FindText)."""
69+
return _backend().find_text(str(text), bool(ignore_case), name=name,
70+
role=role, app_name=app_name,
71+
automation_id=automation_id)
72+
73+
74+
def select_control_text(text: str, *, ignore_case: bool = True,
75+
name: Optional[str] = None, role: Optional[str] = None,
76+
app_name: Optional[str] = None,
77+
automation_id: Optional[str] = None) -> bool:
78+
"""Find ``text`` and select its range — position the caret / selection before
79+
typing to replace it (TextPattern.FindText + Select); True on success."""
80+
return _backend().select_text(str(text), bool(ignore_case), name=name,
81+
role=role, app_name=app_name,
82+
automation_id=automation_id)
83+
84+
85+
def control_text_attributes(name: Optional[str] = None, role: Optional[str] = None,
86+
app_name: Optional[str] = None,
87+
automation_id: Optional[str] = None,
88+
) -> Optional[Dict[str, Any]]:
89+
"""Return the control selection's formatting — ``{font_name, font_size, bold,
90+
italic, foreground_color}`` (TextPattern attributes), or None if not found."""
91+
return _backend().text_attributes(name=name, role=role, app_name=app_name,
92+
automation_id=automation_id)

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2582,6 +2582,38 @@ def _get_control_text(name: Optional[str] = None, role: Optional[str] = None,
25822582
automation_id=automation_id)}
25832583

25842584

2585+
def _find_control_text(text: str, ignore_case: Any = True,
2586+
name: Optional[str] = None, role: Optional[str] = None,
2587+
app_name: Optional[str] = None,
2588+
automation_id: Optional[str] = None) -> bool:
2589+
"""Adapter: whether text occurs in a control (TextPattern.FindText)."""
2590+
from je_auto_control.utils.ax_text import find_control_text
2591+
return find_control_text(str(text), ignore_case=bool(ignore_case), name=name,
2592+
role=role, app_name=app_name,
2593+
automation_id=automation_id)
2594+
2595+
2596+
def _select_control_text(text: str, ignore_case: Any = True,
2597+
name: Optional[str] = None, role: Optional[str] = None,
2598+
app_name: Optional[str] = None,
2599+
automation_id: Optional[str] = None) -> bool:
2600+
"""Adapter: find + select text in a control (TextPattern.FindText + Select)."""
2601+
from je_auto_control.utils.ax_text import select_control_text
2602+
return select_control_text(str(text), ignore_case=bool(ignore_case),
2603+
name=name, role=role, app_name=app_name,
2604+
automation_id=automation_id)
2605+
2606+
2607+
def _control_text_attributes(name: Optional[str] = None, role: Optional[str] = None,
2608+
app_name: Optional[str] = None,
2609+
automation_id: Optional[str] = None) -> Dict[str, Any]:
2610+
"""Adapter: read a control selection's font/colour formatting (TextPattern)."""
2611+
from je_auto_control.utils.ax_text import control_text_attributes
2612+
attrs = control_text_attributes(name=name, role=role, app_name=app_name,
2613+
automation_id=automation_id)
2614+
return {"found": attrs is not None, "attributes": attrs}
2615+
2616+
25852617
def _get_selected_text(name: Optional[str] = None, role: Optional[str] = None,
25862618
app_name: Optional[str] = None,
25872619
automation_id: Optional[str] = None) -> Dict[str, Any]:
@@ -6528,6 +6560,9 @@ def __init__(self):
65286560
"AC_legacy_info": _legacy_info,
65296561
"AC_legacy_default_action": _legacy_default_action,
65306562
"AC_get_control_text": _get_control_text,
6563+
"AC_find_control_text": _find_control_text,
6564+
"AC_select_control_text": _select_control_text,
6565+
"AC_control_text_attributes": _control_text_attributes,
65316566
"AC_get_selected_text": _get_selected_text,
65326567
"AC_get_visible_text": _get_visible_text,
65336568
"AC_read_table": _read_table,

0 commit comments

Comments
 (0)