Skip to content

Commit 7e7f901

Browse files
authored
Merge pull request #453 from Integration-Automation/dev
Release: input-fidelity lane completion (act_in_view, act_with_mode) v221-v222
2 parents 8ba055d + dcf86b7 commit 7e7f901

16 files changed

Lines changed: 632 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,18 @@
22

33
## What's new (2026-06-26)
44

5+
### Trial and Force Action Modes (Playwright-style)
6+
7+
Dry-run "is this control ready?" without clicking, or force a click past the gate. Full reference: [`docs/source/Eng/doc/new_features/v222_features_doc.rst`](docs/source/Eng/doc/new_features/v222_features_doc.rst).
8+
9+
- **`act_with_mode`** (`AC_act_with_mode`): `actionability.act_when_ready` only waits-then-acts. Real flows need two more modes Playwright codified: **trial** (run every actionability check but *don't* act — a side-effect-free "would this work?" dry run) and **force** (skip the checks and act now — the escape hatch when the gate misjudges a control as occluded/disabled). `act_with_mode` adds both alongside the default `auto`, over the same injectable seams as the gate, returning `{mode, acted, actionable, reason, point, result}`. Reuses `actionability.wait_actionable`; fully testable without a screen. Completes the ROUND-15 input-fidelity lane (7/7). No `PySide6`.
10+
11+
### Act In View — Scroll to a Target, Then Act When Actionable
12+
13+
Click the row three pages down: scroll it into view, then gate on actionability before clicking. Full reference: [`docs/source/Eng/doc/new_features/v221_features_doc.rst`](docs/source/Eng/doc/new_features/v221_features_doc.rst).
14+
15+
- **`act_in_view` / `ScrollPlan`** (`AC_act_in_view`): two reliability primitives stayed separate — `scroll_find.scroll_until_visible` brings an off-screen target on-screen, and `actionability.act_when_ready` waits for it to be visible/stable/enabled/unoccluded before acting. A real "click the off-screen row" step needs both. `act_in_view` composes them: scroll until the target is located, then run the actionability gate at its point and perform the action. `ScrollPlan` bundles the scroll search + its `locator`/`scroller` seams so the call stays within the argument limit; the actionability probes (`region_sampler`/`enabled_probe`/`hit_tester`) and gate `config` are injectable too, so the whole flow is testable without a screen. Closes the input-fidelity lane's composition gap. No `PySide6`.
16+
517
### Template-Free Element Proposal (Pixels to Elements)
618

719
Get a clean numbered element list straight from the screen when there's no accessibility tree. Full reference: [`docs/source/Eng/doc/new_features/v220_features_doc.rst`](docs/source/Eng/doc/new_features/v220_features_doc.rst).
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
Act In View — Scroll to a Target, Then Act When Actionable
2+
==========================================================
3+
4+
Two reliability primitives stayed separate: ``scroll_find.scroll_until_visible``
5+
brings an off-screen target on-screen, and ``actionability.act_when_ready`` waits
6+
for a target to be visible / stable / enabled / unoccluded before acting. A real
7+
"click the row three pages down" step needs *both* — scroll to it, then gate
8+
before clicking. ``act_in_view`` composes them into one call.
9+
10+
* :class:`ScrollPlan` — bundles the scroll search (``kind`` / ``direction`` /
11+
``max_scrolls`` / ``scroll_amount``) and its injectable ``locator`` /
12+
``scroller`` seams, so the composed call stays within a sane argument count.
13+
* :func:`act_in_view` — scroll until the target is found, then run the
14+
actionability gate at its location and perform ``action`` on it.
15+
16+
Every seam — the scroll locator / scroller, the action, the actionability probes
17+
(``region_sampler`` / ``enabled_probe`` / ``hit_tester``) and the gate ``config``
18+
— is injectable, so the whole flow is testable without a screen. Reuses
19+
:func:`scroll_find.scroll_until_visible` and
20+
:func:`actionability.act_when_ready`. Imports no ``PySide6``.
21+
22+
Headless API
23+
------------
24+
25+
.. code-block:: python
26+
27+
from je_auto_control import act_in_view, ScrollPlan
28+
29+
# Scroll down to the "Submit" button image, then click it once it's actionable
30+
act_in_view("submit.png", lambda point: click(point[0], point[1]),
31+
scroll=ScrollPlan(kind="image", direction="down",
32+
max_scrolls=20))
33+
34+
``act_in_view`` returns ``{acted, coords, scrolls, result}`` (``result`` is the
35+
action's return value) and raises ``AutoControlActionException`` if the target
36+
never comes into view. Pass ``enabled_probe`` / ``hit_tester`` / ``config`` to
37+
have the actionability gate actually wait for the control to be enabled and
38+
unoccluded before the action fires — otherwise it acts as soon as the target is
39+
located.
40+
41+
Executor commands
42+
-----------------
43+
44+
``AC_act_in_view`` (``target`` + ``kind`` / ``direction`` / ``max_scrolls`` /
45+
``scroll_amount`` / ``button`` → ``{acted, coords, scrolls}``) scrolls a template
46+
or text target into view and clicks it. It is the matching ``ac_act_in_view`` MCP
47+
tool and a Script Builder command under **Flow**. :func:`act_in_view` (which
48+
takes an arbitrary action and the actionability probes) is the Python-API
49+
surface.
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
Trial and Force Action Modes (Playwright-style)
2+
===============================================
3+
4+
``actionability.act_when_ready`` has one behaviour: wait for the target to be
5+
actionable, then act (or raise on timeout). Real flows need two more modes that
6+
Playwright codified:
7+
8+
* **trial** — run every actionability check but *don't* perform the action; just
9+
report whether it *would* have acted. The dry run for "is this control ready?"
10+
without side effects.
11+
* **force** — skip the checks and act *now*, the deliberate escape hatch when the
12+
gate is wrong (a control the heuristics misjudge as occluded / disabled).
13+
14+
:func:`act_with_mode` adds both alongside the default gated (``auto``) behaviour,
15+
over the same injectable seams as the gate, so each mode is testable without a
16+
screen. Reuses :func:`actionability.wait_actionable`. Imports no ``PySide6``.
17+
18+
Headless API
19+
------------
20+
21+
.. code-block:: python
22+
23+
from je_auto_control import act_with_mode
24+
25+
bbox = lambda: (x, y, w, h)
26+
click = lambda point: do_click(point[0], point[1])
27+
28+
act_with_mode(click, bbox, mode="auto") # gate, then click if ready
29+
report = act_with_mode(click, bbox, mode="trial") # dry run, never clicks
30+
if report["actionable"]:
31+
...
32+
act_with_mode(click, bbox, mode="force") # click now, no checks
33+
34+
Every mode returns ``{mode, acted, actionable, reason, point, result}``:
35+
``acted`` says whether the action ran, ``actionable`` / ``reason`` come from the
36+
gate (``trial`` reports these without acting), and ``result`` is the action's
37+
return value. The actionability probes (``region_sampler`` / ``enabled_probe`` /
38+
``hit_tester``) and ``config`` are forwarded to the gate as usual. An unknown
39+
``mode`` raises ``ValueError``.
40+
41+
Executor commands
42+
-----------------
43+
44+
``AC_act_with_mode`` (``x`` / ``y`` + ``mode`` / ``button`` → ``{mode, acted,
45+
actionable, reason, point}``) clicks a point under the chosen mode — ``trial``
46+
is a dry-run probe that never clicks, ``force`` clicks unconditionally. It is the
47+
matching ``ac_act_with_mode`` MCP tool and a Script Builder command under
48+
**Flow**. :func:`act_with_mode` (which takes an arbitrary action) is the
49+
Python-API surface.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
在視野內操作——捲動到目標,再於可操作時動作
2+
============================================
3+
4+
兩個可靠性原語原本各自獨立:``scroll_find.scroll_until_visible`` 把螢幕外的目標捲進畫面,
5+
``actionability.act_when_ready`` 則在目標可見 / 穩定 / 啟用 / 未被遮擋前等待再動作。真實的
6+
「點選下三頁的那一列」步驟需要*兩者*——先捲到它,再閘控後才點擊。``act_in_view`` 把它們組合成單一呼叫。
7+
8+
* :class:`ScrollPlan` ——把捲動搜尋(``kind`` / ``direction`` / ``max_scrolls`` /
9+
``scroll_amount``)與其可注入的 ``locator`` / ``scroller`` 接縫打包,讓組合後的呼叫維持在合理的參數數量內。
10+
* :func:`act_in_view` ——捲動直到找到目標,接著在其位置執行 actionability 閘控,並對其執行 ``action``。
11+
12+
每個接縫——捲動的 locator / scroller、action、actionability 探針(``region_sampler`` /
13+
``enabled_probe`` / ``hit_tester``)與閘控 ``config``——皆可注入,故整個流程能在沒有螢幕的情況下測試。
14+
重用 :func:`scroll_find.scroll_until_visible` 與 :func:`actionability.act_when_ready`。不匯入 ``PySide6``。
15+
16+
無頭 API
17+
--------
18+
19+
.. code-block:: python
20+
21+
from je_auto_control import act_in_view, ScrollPlan
22+
23+
# 向下捲動到「Submit」按鈕影像,於可操作時點擊
24+
act_in_view("submit.png", lambda point: click(point[0], point[1]),
25+
scroll=ScrollPlan(kind="image", direction="down",
26+
max_scrolls=20))
27+
28+
``act_in_view`` 回傳 ``{acted, coords, scrolls, result}``(``result`` 為 action 的回傳值),
29+
若目標始終未進入畫面則丟出 ``AutoControlActionException``。傳入 ``enabled_probe`` / ``hit_tester`` /
30+
``config`` 可讓 actionability 閘控真正等到控制項已啟用且未被遮擋才觸發動作——否則一旦定位到目標即動作。
31+
32+
執行器指令
33+
----------
34+
35+
``AC_act_in_view``(``target`` 加上 ``kind`` / ``direction`` / ``max_scrolls`` /
36+
``scroll_amount`` / ``button`` → ``{acted, coords, scrolls}``)把 template 或文字目標捲入畫面並點擊。
37+
以對應的 ``ac_act_in_view`` MCP 工具及 Script Builder 指令(位於 **Flow** 分類下)形式提供。
38+
:func:`act_in_view`(接受任意 action 與 actionability 探針)則是 Python API 介面。
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
試行與強制動作模式(Playwright 風格)
2+
=====================================
3+
4+
``actionability.act_when_ready`` 只有一種行為:等待目標可操作,再動作(或逾時丟例外)。真實流程還需要
5+
Playwright 定義的另外兩種模式:
6+
7+
* **trial(試行)**——執行每一項 actionability 檢查,但*不*真正動作;只回報它*是否會*動作。
8+
「這個控制項準備好了嗎?」的無副作用乾跑。
9+
* **force(強制)**——跳過檢查,*立即*動作;當閘控判斷錯誤(把控制項誤判為被遮擋 / 停用)時的刻意逃生口。
10+
11+
:func:`act_with_mode` 在預設的閘控(``auto``)行為之外加上這兩種,使用與閘控相同的可注入接縫,
12+
故每種模式都能在沒有螢幕的情況下測試。重用 :func:`actionability.wait_actionable`。不匯入 ``PySide6``。
13+
14+
無頭 API
15+
--------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import act_with_mode
20+
21+
bbox = lambda: (x, y, w, h)
22+
click = lambda point: do_click(point[0], point[1])
23+
24+
act_with_mode(click, bbox, mode="auto") # 閘控後若就緒則點擊
25+
report = act_with_mode(click, bbox, mode="trial") # 乾跑,絕不點擊
26+
if report["actionable"]:
27+
...
28+
act_with_mode(click, bbox, mode="force") # 立即點擊,不檢查
29+
30+
每種模式皆回傳 ``{mode, acted, actionable, reason, point, result}``:``acted`` 表示動作是否執行,
31+
``actionable`` / ``reason`` 來自閘控(``trial`` 不動作即回報這些),``result`` 為 action 的回傳值。
32+
actionability 探針(``region_sampler`` / ``enabled_probe`` / ``hit_tester``)與 ``config`` 一如往常轉發給閘控。
33+
未知的 ``mode`` 會丟出 ``ValueError``。
34+
35+
執行器指令
36+
----------
37+
38+
``AC_act_with_mode``(``x`` / ``y`` 加上 ``mode`` / ``button`` → ``{mode, acted,
39+
actionable, reason, point}``)以所選模式點擊一個點——``trial`` 是絕不點擊的乾跑探測,``force`` 無條件點擊。
40+
以對應的 ``ac_act_with_mode`` MCP 工具及 Script Builder 指令(位於 **Flow** 分類下)形式提供。
41+
:func:`act_with_mode`(接受任意 action)則是 Python API 介面。

je_auto_control/__init__.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,10 @@
151151
)
152152
# Propose a clean element list from raw pixels (template-free)
153153
from je_auto_control.utils.element_proposal import propose_elements, tag_kinds
154+
# Scroll a target into view, then act on it once it is actionable
155+
from je_auto_control.utils.act_in_view import ScrollPlan, act_in_view
156+
# Trial / force action modes over the actionability gate
157+
from je_auto_control.utils.act_modes import act_with_mode
154158
# Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
155159
from je_auto_control.utils.clipboard_rich_formats import (
156160
build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
@@ -1782,6 +1786,7 @@ def start_autocontrol_gui(*args, **kwargs):
17821786
"localize_changes", "rank_changes",
17831787
"classify_widget", "box_features", "classify_icon",
17841788
"propose_elements", "tag_kinds",
1789+
"act_in_view", "ScrollPlan", "act_with_mode",
17851790
"build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
17861791
"set_clipboard_rtf", "get_clipboard_rtf",
17871792
"set_clipboard_csv", "get_clipboard_csv",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4522,6 +4522,35 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None:
45224522
),
45234523
description="Index where a busy/idle series first settles idle.",
45244524
))
4525+
specs.append(CommandSpec(
4526+
"AC_act_in_view", "Flow", "Act In View (scroll + click)",
4527+
fields=(
4528+
FieldSpec("target", FieldType.STRING,
4529+
placeholder="template path or text"),
4530+
FieldSpec("kind", FieldType.STRING, optional=True,
4531+
default="image", placeholder="image / text"),
4532+
FieldSpec("direction", FieldType.STRING, optional=True,
4533+
default="down", placeholder="up / down"),
4534+
FieldSpec("max_scrolls", FieldType.INT, optional=True, default=10),
4535+
FieldSpec("scroll_amount", FieldType.INT, optional=True,
4536+
default=3),
4537+
FieldSpec("button", FieldType.STRING, optional=True,
4538+
default="left"),
4539+
),
4540+
description="Scroll a target into view, then click it when actionable.",
4541+
))
4542+
specs.append(CommandSpec(
4543+
"AC_act_with_mode", "Flow", "Click with Mode (auto/trial/force)",
4544+
fields=(
4545+
FieldSpec("x", FieldType.INT, placeholder="x"),
4546+
FieldSpec("y", FieldType.INT, placeholder="y"),
4547+
FieldSpec("mode", FieldType.STRING, optional=True, default="auto",
4548+
placeholder="auto / trial / force"),
4549+
FieldSpec("button", FieldType.STRING, optional=True,
4550+
default="left"),
4551+
),
4552+
description="Click a point under an action mode (gate / dry-run / force).",
4553+
))
45254554
specs.append(CommandSpec(
45264555
"AC_simulate_cvd", "Image", "Simulate Colour-Vision Deficiency",
45274556
fields=(
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Scroll a target into view, then act on it once it is actionable."""
2+
from je_auto_control.utils.act_in_view.act_in_view import (
3+
ScrollPlan, act_in_view,
4+
)
5+
6+
__all__ = ["act_in_view", "ScrollPlan"]
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
"""Scroll a target into view, then act on it only once it is actionable.
2+
3+
Two reliability primitives the framework already had stayed separate:
4+
``scroll_find.scroll_until_visible`` brings an off-screen target on-screen, and
5+
``actionability.act_when_ready`` waits for a target to be visible / stable /
6+
enabled / unoccluded before acting. A real "click the row three pages down" step
7+
needs *both* — scroll to it, then gate before clicking. ``act_in_view`` composes
8+
them into one call.
9+
10+
* :class:`ScrollPlan` — bundles the scroll search (``kind`` / ``direction`` /
11+
``max_scrolls`` / ``scroll_amount``) and its injectable ``locator`` /
12+
``scroller`` seams, so the composed call stays within a sane argument count.
13+
* :func:`act_in_view` — scroll until the target is found, then run the
14+
actionability gate at its location and perform ``action`` on it.
15+
16+
All seams (locator / scroller / action / actionability probes / clock) are
17+
injectable, so the whole flow is testable without a screen. Reuses
18+
:func:`scroll_find.scroll_until_visible` and
19+
:func:`actionability.act_when_ready`. Imports no ``PySide6``.
20+
"""
21+
from dataclasses import dataclass
22+
from typing import Any, Callable, Dict, List, Optional
23+
24+
from je_auto_control.utils.actionability import GateConfig, act_when_ready
25+
from je_auto_control.utils.exception.exceptions import AutoControlActionException
26+
from je_auto_control.utils.scroll_find import scroll_until_visible
27+
from je_auto_control.utils.scroll_find.scroll_find import Locator, Scroller
28+
29+
30+
@dataclass
31+
class ScrollPlan:
32+
"""How to scroll while searching for the target (with injectable seams)."""
33+
34+
kind: str = "image"
35+
direction: str = "down"
36+
max_scrolls: int = 10
37+
scroll_amount: int = 3
38+
locator: Optional[Locator] = None
39+
scroller: Optional[Scroller] = None
40+
41+
42+
def act_in_view(target: str, action: Callable[[List[int]], Any], *,
43+
scroll: Optional[ScrollPlan] = None,
44+
region_sampler: Optional[Callable[[Any], Any]] = None,
45+
enabled_probe: Optional[Callable[[], Optional[bool]]] = None,
46+
hit_tester: Optional[Callable[[List[int]], bool]] = None,
47+
config: Optional[GateConfig] = None) -> Dict[str, Any]:
48+
"""Scroll ``target`` into view, gate on actionability, then ``action`` it.
49+
50+
Scrolls per ``scroll`` (a :class:`ScrollPlan`) until ``target`` is located,
51+
then runs :func:`actionability.act_when_ready` at the found point and calls
52+
``action(center_point)``. Raises ``AutoControlActionException`` if the target
53+
never comes into view. The actionability probes / ``config`` are injectable
54+
and forwarded to the gate. Returns ``{acted, coords, scrolls, result}``.
55+
"""
56+
plan = scroll if scroll is not None else ScrollPlan()
57+
found = scroll_until_visible(
58+
target, kind=plan.kind, direction=plan.direction,
59+
max_scrolls=plan.max_scrolls, scroll_amount=plan.scroll_amount,
60+
locator=plan.locator, scroller=plan.scroller)
61+
if not found["found"]:
62+
raise AutoControlActionException(
63+
f"target {target!r} not in view after {found['scrolls']} scrolls")
64+
cx, cy = int(found["coords"][0]), int(found["coords"][1])
65+
result = act_when_ready(action, lambda: (cx, cy, 1, 1),
66+
region_sampler=region_sampler,
67+
enabled_probe=enabled_probe, hit_tester=hit_tester,
68+
config=config)
69+
return {"acted": True, "coords": [cx, cy], "scrolls": found["scrolls"],
70+
"result": result}
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
"""Trial and force action modes over the actionability gate."""
2+
from je_auto_control.utils.act_modes.act_modes import ACT_MODES, act_with_mode
3+
4+
__all__ = ["act_with_mode", "ACT_MODES"]

0 commit comments

Comments
 (0)