Skip to content

Commit ea124d3

Browse files
committed
Add act_with_mode: trial and force action modes over the actionability gate
act_when_ready only waits-then-acts. Real flows need two more modes Playwright codified: trial (run every actionability check but DON'T act - a side-effect-free dry run) and force (skip checks, act now). act_with_mode adds both alongside the default auto, over the same injectable seams as the gate, so each mode is testable without a screen. Reuses wait_actionable.
1 parent 629cccb commit ea124d3

11 files changed

Lines changed: 312 additions & 1 deletion

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
## What's new (2026-06-26)
44

5+
### Trial and Force Action Modes (Playwright-style)
6+
7+
Dry-run "is this control ready?" without clicking, or force a click past the gate. Full reference: [`docs/source/Eng/doc/new_features/v222_features_doc.rst`](docs/source/Eng/doc/new_features/v222_features_doc.rst).
8+
9+
- **`act_with_mode`** (`AC_act_with_mode`): `actionability.act_when_ready` only waits-then-acts. Real flows need two more modes Playwright codified: **trial** (run every actionability check but *don't* act — a side-effect-free "would this work?" dry run) and **force** (skip the checks and act now — the escape hatch when the gate misjudges a control as occluded/disabled). `act_with_mode` adds both alongside the default `auto`, over the same injectable seams as the gate, returning `{mode, acted, actionable, reason, point, result}`. Reuses `actionability.wait_actionable`; fully testable without a screen. Completes the ROUND-15 input-fidelity lane (7/7). No `PySide6`.
10+
511
### Act In View — Scroll to a Target, Then Act When Actionable
612

713
Click the row three pages down: scroll it into view, then gate on actionability before clicking. Full reference: [`docs/source/Eng/doc/new_features/v221_features_doc.rst`](docs/source/Eng/doc/new_features/v221_features_doc.rst).
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
Trial and Force Action Modes (Playwright-style)
2+
===============================================
3+
4+
``actionability.act_when_ready`` has one behaviour: wait for the target to be
5+
actionable, then act (or raise on timeout). Real flows need two more modes that
6+
Playwright codified:
7+
8+
* **trial** — run every actionability check but *don't* perform the action; just
9+
report whether it *would* have acted. The dry run for "is this control ready?"
10+
without side effects.
11+
* **force** — skip the checks and act *now*, the deliberate escape hatch when the
12+
gate is wrong (a control the heuristics misjudge as occluded / disabled).
13+
14+
:func:`act_with_mode` adds both alongside the default gated (``auto``) behaviour,
15+
over the same injectable seams as the gate, so each mode is testable without a
16+
screen. Reuses :func:`actionability.wait_actionable`. Imports no ``PySide6``.
17+
18+
Headless API
19+
------------
20+
21+
.. code-block:: python
22+
23+
from je_auto_control import act_with_mode
24+
25+
bbox = lambda: (x, y, w, h)
26+
click = lambda point: do_click(point[0], point[1])
27+
28+
act_with_mode(click, bbox, mode="auto") # gate, then click if ready
29+
report = act_with_mode(click, bbox, mode="trial") # dry run, never clicks
30+
if report["actionable"]:
31+
...
32+
act_with_mode(click, bbox, mode="force") # click now, no checks
33+
34+
Every mode returns ``{mode, acted, actionable, reason, point, result}``:
35+
``acted`` says whether the action ran, ``actionable`` / ``reason`` come from the
36+
gate (``trial`` reports these without acting), and ``result`` is the action's
37+
return value. The actionability probes (``region_sampler`` / ``enabled_probe`` /
38+
``hit_tester``) and ``config`` are forwarded to the gate as usual. An unknown
39+
``mode`` raises ``ValueError``.
40+
41+
Executor commands
42+
-----------------
43+
44+
``AC_act_with_mode`` (``x`` / ``y`` + ``mode`` / ``button`` → ``{mode, acted,
45+
actionable, reason, point}``) clicks a point under the chosen mode — ``trial``
46+
is a dry-run probe that never clicks, ``force`` clicks unconditionally. It is the
47+
matching ``ac_act_with_mode`` MCP tool and a Script Builder command under
48+
**Flow**. :func:`act_with_mode` (which takes an arbitrary action) is the
49+
Python-API surface.
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
試行與強制動作模式(Playwright 風格)
2+
=====================================
3+
4+
``actionability.act_when_ready`` 只有一種行為:等待目標可操作,再動作(或逾時丟例外)。真實流程還需要
5+
Playwright 定義的另外兩種模式:
6+
7+
* **trial(試行)**——執行每一項 actionability 檢查,但*不*真正動作;只回報它*是否會*動作。
8+
「這個控制項準備好了嗎?」的無副作用乾跑。
9+
* **force(強制)**——跳過檢查,*立即*動作;當閘控判斷錯誤(把控制項誤判為被遮擋 / 停用)時的刻意逃生口。
10+
11+
:func:`act_with_mode` 在預設的閘控(``auto``)行為之外加上這兩種,使用與閘控相同的可注入接縫,
12+
故每種模式都能在沒有螢幕的情況下測試。重用 :func:`actionability.wait_actionable`。不匯入 ``PySide6``。
13+
14+
無頭 API
15+
--------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import act_with_mode
20+
21+
bbox = lambda: (x, y, w, h)
22+
click = lambda point: do_click(point[0], point[1])
23+
24+
act_with_mode(click, bbox, mode="auto") # 閘控後若就緒則點擊
25+
report = act_with_mode(click, bbox, mode="trial") # 乾跑,絕不點擊
26+
if report["actionable"]:
27+
...
28+
act_with_mode(click, bbox, mode="force") # 立即點擊,不檢查
29+
30+
每種模式皆回傳 ``{mode, acted, actionable, reason, point, result}``:``acted`` 表示動作是否執行,
31+
``actionable`` / ``reason`` 來自閘控(``trial`` 不動作即回報這些),``result`` 為 action 的回傳值。
32+
actionability 探針(``region_sampler`` / ``enabled_probe`` / ``hit_tester``)與 ``config`` 一如往常轉發給閘控。
33+
未知的 ``mode`` 會丟出 ``ValueError``。
34+
35+
執行器指令
36+
----------
37+
38+
``AC_act_with_mode``(``x`` / ``y`` 加上 ``mode`` / ``button`` → ``{mode, acted,
39+
actionable, reason, point}``)以所選模式點擊一個點——``trial`` 是絕不點擊的乾跑探測,``force`` 無條件點擊。
40+
以對應的 ``ac_act_with_mode`` MCP 工具及 Script Builder 指令(位於 **Flow** 分類下)形式提供。
41+
:func:`act_with_mode`(接受任意 action)則是 Python API 介面。

je_auto_control/__init__.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,8 @@
153153
from je_auto_control.utils.element_proposal import propose_elements, tag_kinds
154154
# Scroll a target into view, then act on it once it is actionable
155155
from je_auto_control.utils.act_in_view import ScrollPlan, act_in_view
156+
# Trial / force action modes over the actionability gate
157+
from je_auto_control.utils.act_modes import act_with_mode
156158
# Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
157159
from je_auto_control.utils.clipboard_rich_formats import (
158160
build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
@@ -1784,7 +1786,7 @@ def start_autocontrol_gui(*args, **kwargs):
17841786
"localize_changes", "rank_changes",
17851787
"classify_widget", "box_features", "classify_icon",
17861788
"propose_elements", "tag_kinds",
1787-
"act_in_view", "ScrollPlan",
1789+
"act_in_view", "ScrollPlan", "act_with_mode",
17881790
"build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
17891791
"set_clipboard_rtf", "get_clipboard_rtf",
17901792
"set_clipboard_csv", "get_clipboard_csv",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4539,6 +4539,18 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None:
45394539
),
45404540
description="Scroll a target into view, then click it when actionable.",
45414541
))
4542+
specs.append(CommandSpec(
4543+
"AC_act_with_mode", "Flow", "Click with Mode (auto/trial/force)",
4544+
fields=(
4545+
FieldSpec("x", FieldType.INT, placeholder="x"),
4546+
FieldSpec("y", FieldType.INT, placeholder="y"),
4547+
FieldSpec("mode", FieldType.STRING, optional=True, default="auto",
4548+
placeholder="auto / trial / force"),
4549+
FieldSpec("button", FieldType.STRING, optional=True,
4550+
default="left"),
4551+
),
4552+
description="Click a point under an action mode (gate / dry-run / force).",
4553+
))
45424554
specs.append(CommandSpec(
45434555
"AC_simulate_cvd", "Image", "Simulate Colour-Vision Deficiency",
45444556
fields=(
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
"""Trial and force action modes over the actionability gate."""
2+
from je_auto_control.utils.act_modes.act_modes import ACT_MODES, act_with_mode
3+
4+
__all__ = ["act_with_mode", "ACT_MODES"]
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
"""Trial and force action modes over the actionability gate (Playwright-style).
2+
3+
``actionability.act_when_ready`` has one behaviour: wait for the target to be
4+
actionable, then act (or raise on timeout). Real flows need two more modes that
5+
Playwright codified:
6+
7+
* **trial** — run every actionability check but *don't* perform the action; just
8+
report whether it *would* have acted. The dry-run for "is this control ready?"
9+
without side effects.
10+
* **force** — skip the checks and act *now*, for the deliberate escape hatch when
11+
the gate is wrong (a control the heuristics misjudge as occluded / disabled).
12+
13+
``act_with_mode`` adds both alongside the default gated behaviour, over the same
14+
injectable seams as the gate, so each mode is testable without a screen. Reuses
15+
:func:`actionability.wait_actionable`. Imports no ``PySide6``.
16+
"""
17+
from typing import Any, Callable, Dict, List, Optional
18+
19+
from je_auto_control.utils.actionability import wait_actionable
20+
from je_auto_control.utils.actionability.actionability import _center
21+
22+
# The supported action modes.
23+
ACT_MODES = ("auto", "trial", "force")
24+
25+
26+
def _force_act(action: Callable[[List[int]], Any],
27+
bbox_provider: Callable[[], Any]) -> Dict[str, Any]:
28+
"""Act at the target centre with no actionability checks."""
29+
bbox = bbox_provider()
30+
if not bbox:
31+
return {"mode": "force", "acted": False, "actionable": True,
32+
"reason": "no target", "point": None, "result": None}
33+
point = _center(bbox)
34+
return {"mode": "force", "acted": True, "actionable": True,
35+
"reason": "forced", "point": point, "result": action(point)}
36+
37+
38+
def act_with_mode(action: Callable[[List[int]], Any],
39+
bbox_provider: Callable[[], Any], *, mode: str = "auto",
40+
region_sampler: Optional[Callable[[Any], Any]] = None,
41+
enabled_probe: Optional[Callable[[], Optional[bool]]] = None,
42+
hit_tester: Optional[Callable[[List[int]], bool]] = None,
43+
config: Optional[Any] = None) -> Dict[str, Any]:
44+
"""Perform ``action`` on a target under a ``mode`` (auto / trial / force).
45+
46+
``auto`` waits for the actionability gate and acts only if it passes;
47+
``trial`` runs the gate but never acts (a dry run); ``force`` acts at once
48+
with no checks. Returns ``{mode, acted, actionable, reason, point, result}``.
49+
Raises ``ValueError`` for an unknown ``mode``.
50+
"""
51+
if mode not in ACT_MODES:
52+
raise ValueError(f"unknown act mode: {mode!r}")
53+
if mode == "force":
54+
return _force_act(action, bbox_provider)
55+
report = wait_actionable(bbox_provider, region_sampler=region_sampler,
56+
enabled_probe=enabled_probe, hit_tester=hit_tester,
57+
config=config)
58+
point = list(report.point) if report.point is not None else None
59+
base = {"mode": mode, "actionable": report.actionable,
60+
"reason": report.reason, "point": point}
61+
if mode == "trial" or not report.actionable:
62+
return {**base, "acted": False, "result": None}
63+
return {**base, "acted": True, "result": action(report.point)}

je_auto_control/utils/executor/action_executor.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2979,6 +2979,18 @@ def _act_in_view(target: Any, kind: Any = "image", direction: Any = "down",
29792979
"scrolls": out["scrolls"]}
29802980

29812981

2982+
def _act_with_mode(x: Any, y: Any, mode: Any = "auto",
2983+
button: Any = "left") -> Dict[str, Any]:
2984+
"""Adapter: click a point under an action mode (auto / trial / force)."""
2985+
from je_auto_control.utils.act_modes import act_with_mode
2986+
out = act_with_mode(
2987+
lambda point: click_mouse(str(button), int(point[0]), int(point[1])),
2988+
lambda: (int(x), int(y), 1, 1), mode=str(mode))
2989+
return {"mode": out["mode"], "acted": out["acted"],
2990+
"actionable": out["actionable"], "reason": out["reason"],
2991+
"point": out["point"]}
2992+
2993+
29822994
def _normalize_ext(target: str) -> Dict[str, Any]:
29832995
"""Adapter: the lowercased extension of a path / bare ext (pure)."""
29842996
from je_auto_control.utils.file_assoc import normalize_ext
@@ -7025,6 +7037,7 @@ def __init__(self):
70257037
"AC_propose_elements": _propose_elements,
70267038
"AC_tag_kinds": _tag_kinds,
70277039
"AC_act_in_view": _act_in_view,
7040+
"AC_act_with_mode": _act_with_mode,
70287041
"AC_normalize_ext": _normalize_ext,
70297042
"AC_file_association": _file_association,
70307043
"AC_get_control_text": _get_control_text,

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1913,6 +1913,20 @@ def smart_wait_tools() -> List[MCPTool]:
19131913
handler=h.act_in_view,
19141914
annotations=SIDE_EFFECT_ONLY,
19151915
),
1916+
MCPTool(
1917+
name="ac_act_with_mode",
1918+
description=("Click point (x, y) under an action 'mode': 'auto' "
1919+
"(gate then click), 'trial' (gate but DON'T click — "
1920+
"dry run), or 'force' (click with no checks). Returns "
1921+
"{mode, acted, actionable, reason, point}."),
1922+
input_schema=schema({"x": {"type": "integer"},
1923+
"y": {"type": "integer"},
1924+
"mode": {"type": "string"},
1925+
"button": {"type": "string"}},
1926+
required=["x", "y"]),
1927+
handler=h.act_with_mode,
1928+
annotations=SIDE_EFFECT_ONLY,
1929+
),
19161930
]
19171931

19181932

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -816,6 +816,11 @@ def act_in_view(target, kind="image", direction="down", max_scrolls=10,
816816
button)
817817

818818

819+
def act_with_mode(x, y, mode="auto", button="left"):
820+
from je_auto_control.utils.executor.action_executor import _act_with_mode
821+
return _act_with_mode(x, y, mode, button)
822+
823+
819824
def normalize_ext(target):
820825
from je_auto_control.utils.executor.action_executor import _normalize_ext
821826
return _normalize_ext(target)

0 commit comments

Comments
 (0)