Skip to content

Commit 5b4eb97

Browse files
JE-ChenJE-Chen
authored
Merge pull request #393 from Integration-Automation/feat/match-ensemble-batch
Add match_ensemble: multi-template consensus matching
2 parents 3c8c8bb + 82d0344 commit 5b4eb97

15 files changed

Lines changed: 320 additions & 0 deletions

File tree

‎README/WHATS_NEW_zh-CN.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-06-24) — 多模板共识匹配
4+
5+
把同一目标的多个参考裁切投票成单一可信位置。完整参考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
6+
7+
- **`match_ensemble` / `vote_centers`**(`AC_match_ensemble`、`AC_vote_centers`):一个按钮以多种状态呈现(默认/悬停/按下)但是单一逻辑目标;`ab_locator` 只选一个策略、`match_template(scales=...)` 只扫一个模板——两者都不融合多参考。本功能匹配每个参考,聚类命中中心,只有在 ≥ `min_votes` 个于 `agree_px` 内一致时才接受,返回 `{point, votes, n_candidates, spread}`——减少换肤/动画 UI 的误判。重用 `visual_match.match_template` + `grounding_consensus`;`vote_centers` 为纯投票核心。不导入 `PySide6`。
8+
39
## 本次更新 (2026-06-24) — 逐步评审特征 + 规则式步骤评分
410

511
把为代理步骤评分所需的证据打包,并内建规则式评分器。完整参考:[`docs/source/Zh/doc/new_features/v177_features_doc.rst`](../docs/source/Zh/doc/new_features/v177_features_doc.rst)。

‎README/WHATS_NEW_zh-TW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-06-24) — 多模板共識比對
4+
5+
把同一目標的多個參考裁切投票成單一可信位置。完整參考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
6+
7+
- **`match_ensemble` / `vote_centers`**(`AC_match_ensemble`、`AC_vote_centers`):一個按鈕以多種狀態呈現(預設/懸停/按下)但是單一邏輯目標;`ab_locator` 只選一個策略、`match_template(scales=...)` 只掃一個模板——兩者都不融合多參考。本功能比對每個參考,聚類命中中心,只有在 ≥ `min_votes` 個於 `agree_px` 內一致時才接受,回傳 `{point, votes, n_candidates, spread}`——減少換膚/動畫 UI 的誤判。重用 `visual_match.match_template` + `grounding_consensus`;`vote_centers` 為純投票核心。不匯入 `PySide6`。
8+
39
## 本次更新 (2026-06-24) — 逐步評審特徵 + 規則式步驟評分
410

511
把為代理步驟評分所需的證據打包,並內建規則式評分器。完整參考:[`docs/source/Zh/doc/new_features/v177_features_doc.rst`](../docs/source/Zh/doc/new_features/v177_features_doc.rst)。

‎WHATS_NEW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-24) — Multi-Template Consensus Matching
4+
5+
Vote several reference crops of one target into a single trustworthy location. Full reference: [`docs/source/Eng/doc/new_features/v178_features_doc.rst`](docs/source/Eng/doc/new_features/v178_features_doc.rst).
6+
7+
- **`match_ensemble` / `vote_centers`** (`AC_match_ensemble`, `AC_vote_centers`): a button renders in several states (default/hover/pressed) but is one logical target; `ab_locator` picks one strategy and `match_template(scales=...)` sweeps one template — neither fuses multiple references. This matches each reference, clusters the hit centres, and accepts a target only when ≥ `min_votes` agree within `agree_px`, returning `{point, votes, n_candidates, spread}` — cutting false positives on themed/animated UI. Reuses `visual_match.match_template` + `grounding_consensus`; `vote_centers` is the pure voting core. No `PySide6`.
8+
39
## What's new (2026-06-24) — Per-Step Critic Features + Rule-Based Step Scorer
410

511
Bundle the evidence to score an agent step, with a built-in rule-based scorer. Full reference: [`docs/source/Eng/doc/new_features/v177_features_doc.rst`](docs/source/Eng/doc/new_features/v177_features_doc.rst).

‎docs/source/Eng/doc/new_features/v178_features_doc.rst‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
Multi-Template Consensus Matching
2+
=================================
3+
4+
A button often renders in several states — default / hover / pressed / disabled — yet it is a
5+
single logical target. ``ab_locator`` A/B-tests *which single strategy wins* and
6+
``match_template(scales=...)`` sweeps *one* template across scales — neither fuses *multiple
7+
reference crops* into one vote. ``match_ensemble`` matches each reference, clusters the hit
8+
centres and accepts a target only when at least ``min_votes`` references agree within
9+
``agree_px`` — sharply cutting false positives on themed / animated UI.
10+
11+
The voting core (``vote_centers``) is pure-stdlib and reuses ``grounding_consensus`` for the
12+
clustering, so it is unit-testable with no image; only ``match_ensemble`` itself calls
13+
``visual_match.match_template`` (testable on a synthetic injected ``haystack``). Imports no
14+
``PySide6``.
15+
16+
Headless API
17+
------------
18+
19+
.. code-block:: python
20+
21+
from je_auto_control import match_ensemble, vote_centers
22+
23+
# three reference crops of the same button (default / hover / pressed)
24+
result = match_ensemble(["btn_default.png", "btn_hover.png", "btn_pressed.png"],
25+
min_score=0.8, agree_px=10, min_votes=2)
26+
if result:
27+
click(*result["point"]) # {point, votes, n_candidates, spread}
28+
29+
# or vote hit centres you already have
30+
vote_centers([[100, 100], [102, 98], [400, 400]], agree_px=10, min_votes=2)
31+
32+
``match_ensemble`` returns ``{point, votes, n_candidates, spread}`` for the consensus location,
33+
or ``None`` if fewer than ``min_votes`` references agree. ``vote_centers`` is the pure voting
34+
step over candidate centres you supply.
35+
36+
Executor commands
37+
-----------------
38+
39+
``AC_match_ensemble`` (``templates`` / ``min_score`` / ``agree_px`` / ``min_votes`` /
40+
``region`` → ``{found, result}``) and ``AC_vote_centers`` (``centers`` / ``agree_px`` /
41+
``min_votes`` → ``{found, result}``). They are exposed as the MCP tools ``ac_match_ensemble`` /
42+
``ac_vote_centers`` (read-only) and as the Script Builder commands **Match Ensemble (vote
43+
references)** / **Vote Centers (consensus)** under **Image**.

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@ Comprehensive guides for all AutoControl features.
200200
doc/new_features/v175_features_doc
201201
doc/new_features/v176_features_doc
202202
doc/new_features/v177_features_doc
203+
doc/new_features/v178_features_doc
203204
doc/ocr_backends/ocr_backends_doc
204205
doc/observability/observability_doc
205206
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v178_features_doc.rst‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
多模板共識比對
2+
==============
3+
4+
一個按鈕常以多種狀態呈現——預設 / 懸停 / 按下 / 停用——但它是單一邏輯目標。``ab_locator``
5+
A/B 測試*哪個單一策略勝出*,``match_template(scales=...)`` 跨縮放掃描*一個*模板——兩者都不把
6+
*多個參考裁切*融合成一次投票。``match_ensemble`` 比對每個參考,聚類命中中心,只有在至少
7+
``min_votes`` 個參考於 ``agree_px`` 內一致時才接受目標——大幅減少換膚 / 動畫 UI 上的誤判。
8+
9+
投票核心(``vote_centers``)為純標準函式庫,並重用 ``grounding_consensus`` 做聚類,因此可在
10+
無影像下單元測試;只有 ``match_ensemble`` 本身呼叫 ``visual_match.match_template``(可於注入的
11+
合成 ``haystack`` 上測試)。不匯入 ``PySide6``。
12+
13+
無頭 API
14+
--------
15+
16+
.. code-block:: python
17+
18+
from je_auto_control import match_ensemble, vote_centers
19+
20+
# 同一按鈕的三個參考裁切(預設 / 懸停 / 按下)
21+
result = match_ensemble(["btn_default.png", "btn_hover.png", "btn_pressed.png"],
22+
min_score=0.8, agree_px=10, min_votes=2)
23+
if result:
24+
click(*result["point"]) # {point, votes, n_candidates, spread}
25+
26+
# 或對你已有的命中中心投票
27+
vote_centers([[100, 100], [102, 98], [400, 400]], agree_px=10, min_votes=2)
28+
29+
``match_ensemble`` 回傳共識位置的 ``{point, votes, n_candidates, spread}``,或在少於
30+
``min_votes`` 個參考一致時回傳 ``None``。``vote_centers`` 是對你提供的候選中心的純投票步驟。
31+
32+
執行器指令
33+
----------
34+
35+
``AC_match_ensemble``(``templates`` / ``min_score`` / ``agree_px`` / ``min_votes`` /
36+
``region`` → ``{found, result}``)與 ``AC_vote_centers``(``centers`` / ``agree_px`` /
37+
``min_votes`` → ``{found, result}``)。兩者以 MCP 工具 ``ac_match_ensemble`` /
38+
``ac_vote_centers``(唯讀)及 Script Builder 指令 **Match Ensemble (vote references)** /
39+
**Vote Centers (consensus)**(位於 **Image** 分類下)形式提供。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -200,6 +200,7 @@ AutoControl 所有功能的完整使用指南。
200200
doc/new_features/v175_features_doc
201201
doc/new_features/v176_features_doc
202202
doc/new_features/v177_features_doc
203+
doc/new_features/v178_features_doc
203204
doc/ocr_backends/ocr_backends_doc
204205
doc/observability/observability_doc
205206
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,10 @@
291291
from je_auto_control.utils.edge_match import (
292292
chamfer_distance, edge_match, edge_match_all,
293293
)
294+
# Multi-template consensus matching (vote references onto one location)
295+
from je_auto_control.utils.match_ensemble import (
296+
match_ensemble, vote_centers,
297+
)
294298
# Otsu auto-thresholding for template matching (no hand-tuned min_score)
295299
from je_auto_control.utils.match_autothresh import (
296300
auto_threshold, match_auto,
@@ -1274,6 +1278,8 @@ def start_autocontrol_gui(*args, **kwargs):
12741278
"edge_match",
12751279
"edge_match_all",
12761280
"chamfer_distance",
1281+
"match_ensemble",
1282+
"vote_centers",
12771283
"SubPixelMatch",
12781284
"match_subpixel",
12791285
"refine_peak",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -407,6 +407,30 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
407407
),
408408
description="Match with a sub-pixel-refined centre (drag / slider precision).",
409409
))
410+
specs.append(CommandSpec(
411+
"AC_match_ensemble", "Image", "Match Ensemble (vote references)",
412+
fields=(
413+
FieldSpec("templates", FieldType.STRING,
414+
placeholder='["default.png", "hover.png", "pressed.png"]'),
415+
FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
416+
min_value=0.0, max_value=1.0),
417+
FieldSpec("agree_px", FieldType.INT, optional=True, default=10),
418+
FieldSpec("min_votes", FieldType.INT, optional=True, default=2),
419+
FieldSpec("region", FieldType.STRING, optional=True,
420+
placeholder=_REGION_PLACEHOLDER),
421+
),
422+
description="Vote several reference crops onto one consensus location.",
423+
))
424+
specs.append(CommandSpec(
425+
"AC_vote_centers", "Image", "Vote Centers (consensus)",
426+
fields=(
427+
FieldSpec("centers", FieldType.STRING,
428+
placeholder="[[100, 100], [102, 98]]"),
429+
FieldSpec("agree_px", FieldType.INT, optional=True, default=10),
430+
FieldSpec("min_votes", FieldType.INT, optional=True, default=2),
431+
),
432+
description="Vote candidate hit centres into one consensus target.",
433+
))
410434
specs.append(CommandSpec(
411435
"AC_grid_cells", "Image", "Grid Cells (coarse grounding)",
412436
fields=(

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3400,6 +3400,32 @@ def _match_subpixel(template: str, min_score: Any = 0.0, region: Any = None,
34003400
"match": match.to_dict() if match else None}
34013401

34023402

3403+
def _vote_centers(centers: Any, agree_px: Any = 10,
3404+
min_votes: Any = 2) -> Dict[str, Any]:
3405+
"""Adapter: vote candidate hit centres into a consensus target."""
3406+
import json
3407+
from je_auto_control.utils.match_ensemble import vote_centers
3408+
if isinstance(centers, str):
3409+
centers = json.loads(centers)
3410+
result = vote_centers(centers, agree_px=float(agree_px),
3411+
min_votes=int(min_votes))
3412+
return {"found": result is not None, "result": result}
3413+
3414+
3415+
def _match_ensemble(templates: Any, min_score: Any = 0.8, agree_px: Any = 10,
3416+
min_votes: Any = 2, region: Any = None) -> Dict[str, Any]:
3417+
"""Adapter: vote several template references onto one consensus location."""
3418+
import json
3419+
from je_auto_control.utils.match_ensemble import match_ensemble
3420+
if isinstance(templates, str):
3421+
templates = json.loads(templates)
3422+
if isinstance(region, str):
3423+
region = json.loads(region) if region.strip() else None
3424+
result = match_ensemble(templates, region=region, min_score=float(min_score),
3425+
agree_px=float(agree_px), min_votes=int(min_votes))
3426+
return {"found": result is not None, "result": result}
3427+
3428+
34033429
def _region_arg(value: Any) -> Optional[List[int]]:
34043430
"""Coerce a JSON-string / list region arg into a list of ints, or None."""
34053431
import json
@@ -6123,6 +6149,8 @@ def __init__(self):
61236149
"AC_edge_match": _edge_match,
61246150
"AC_edge_match_all": _edge_match_all,
61256151
"AC_match_subpixel": _match_subpixel,
6152+
"AC_match_ensemble": _match_ensemble,
6153+
"AC_vote_centers": _vote_centers,
61266154
"AC_grid_cells": _grid_cells,
61276155
"AC_cell_for_point": _cell_for_point,
61286156
"AC_point_for_cell": _point_for_cell,

0 commit comments

Comments
 (0)