Merge pull request #394 from Integration-Automation/feat/color-match-batch

JE-Chen · web-flow · commit 8410eab1a058 · 2026-06-24T07:26:10.000+08:00
Add color_match: colour-aware (HSV) template matching
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 色彩感知模板匹配(HSV)
+
+区分形状相同的红色与绿色状态点。完整参考:[`docs/source/Zh/doc/new_features/v179_features_doc.rst`](../docs/source/Zh/doc/new_features/v179_features_doc.rst)。
+
+- **`match_color` / `match_color_all`**(`AC_match_color`、`AC_match_color_all`):`visual_match` 每个匹配器都先转灰阶,故形状相同的红 vs 绿无法区分;`color_region` 找已知颜色的 blob 却无法对多色字形做模板匹配。本功能在 HSV 色相/饱和度上以色彩*距离*度量(`TM_SQDIFF_NORMED`——相关会把绝对色相正规化掉,使红→绿边与黑→蓝边同分)。重用 `color_region` 的 RGB 加载器 + `visual_match` 的 resize/NMS/`Match`。`channels` 默认 `("h","s")`(平坦饱和度目标用 `("h",)`);纯色 blob 请用 `find_color_region`。不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 多模板共识匹配
 
 把同一目标的多个参考裁切投票成单一可信位置。完整参考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 色彩感知樣板比對(HSV)
+
+區分形狀相同的紅色與綠色狀態點。完整參考:[`docs/source/Zh/doc/new_features/v179_features_doc.rst`](../docs/source/Zh/doc/new_features/v179_features_doc.rst)。
+
+- **`match_color` / `match_color_all`**(`AC_match_color`、`AC_match_color_all`):`visual_match` 每個比對器都先轉灰階,故形狀相同的紅 vs 綠無法區分;`color_region` 找已知顏色的 blob 卻無法對多色字形做樣板比對。本功能在 HSV 色相/飽和度上以色彩*距離*度量(`TM_SQDIFF_NORMED`——相關會把絕對色相正規化掉,使紅→綠邊與黑→藍邊同分)。重用 `color_region` 的 RGB 載入器 + `visual_match` 的 resize/NMS/`Match`。`channels` 預設 `("h","s")`(平坦飽和度目標用 `("h",)`);純色 blob 請用 `find_color_region`。不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 多模板共識比對
 
 把同一目標的多個參考裁切投票成單一可信位置。完整參考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Colour-Aware Template Matching (HSV)
+
+Tell a red status dot from a green one of identical shape. Full reference: [`docs/source/Eng/doc/new_features/v179_features_doc.rst`](docs/source/Eng/doc/new_features/v179_features_doc.rst).
+
+- **`match_color` / `match_color_all`** (`AC_match_color`, `AC_match_color_all`): every `visual_match` matcher grayscales first, so red vs green of identical shape is indistinguishable; `color_region` finds known-colour blobs but can't template-match a multi-colour glyph. This matches on HSV hue/saturation with a colour-*distance* metric (`TM_SQDIFF_NORMED` — correlation would normalise away the absolute hue, scoring a red→green edge same as black→blue). Reuses `color_region`'s RGB loaders + `visual_match`'s resize/NMS/`Match`. `channels` default `("h","s")` (use `("h",)` for flat-saturation targets); for solid blobs use `find_color_region`. No `PySide6`.
+
 ## What's new (2026-06-24) — Multi-Template Consensus Matching
 
 Vote several reference crops of one target into a single trustworthy location. Full reference: [`docs/source/Eng/doc/new_features/v178_features_doc.rst`](docs/source/Eng/doc/new_features/v178_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v179_features_doc.rst b/docs/source/Eng/doc/new_features/v179_features_doc.rst
@@ -0,0 +1,47 @@
+Colour-Aware Template Matching (HSV)
+====================================
+
+Every matcher in ``visual_match`` converts to grayscale first, so a red versus green status
+indicator of identical shape is *indistinguishable* to ``match_template`` — the discriminating
+signal is thrown away. ``color_region`` finds blobs of a *known* colour but cannot
+template-match a multi-colour glyph by appearance. ``color_match`` matches on the HSV
+hue / saturation channels using a colour-*distance* metric (``TM_SQDIFF_NORMED``, not a
+correlation — correlation normalises away the absolute hue, so a red→green edge and a
+black→blue edge would score the same), locating colour-discriminated targets that grayscale
+matching collapses.
+
+It reuses ``color_region``'s RGB loaders and ``visual_match``'s resize / NMS / ``Match``. The
+``haystack`` is injectable; the search is unit-testable on synthetic arrays. Imports no
+``PySide6``.
+
+Note: like any window metric, a *flat* single-colour patch has no per-channel variance — for
+solid colour blobs use ``find_color_region``; ``color_match`` is for targets with colour
+*structure*.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import match_color, match_color_all
+
+    # locate a red status chip, not the green one of the same shape
+    hit = match_color("status_red.png", channels=("h", "s"), min_score=0.7)
+    if hit:
+        click(*hit.center)
+
+    for m in match_color_all("tag_green.png", channels=("h",)):
+        print(m.center, m.score)
+
+``match_color`` returns the best ``Match`` over the chosen ``channels`` (default ``("h", "s")``;
+use ``("h",)`` for flat-saturation targets), or ``None``. ``match_color_all`` returns every
+match at or above ``min_score`` with overlaps removed by NMS.
+
+Executor commands
+-----------------
+
+``AC_match_color`` (``template`` / ``channels`` / ``min_score`` / ``scales`` / ``region`` →
+``{found, match}``) and ``AC_match_color_all`` (adds ``max_results`` / ``nms_iou`` →
+``{count, matches}``). They are exposed as the MCP tools ``ac_match_color`` /
+``ac_match_color_all`` (read-only) and as the Script Builder commands **Match Template
+(colour/HSV)** / **Match Template All (colour/HSV)** under **Image**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -201,6 +201,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v176_features_doc
    doc/new_features/v177_features_doc
    doc/new_features/v178_features_doc
+   doc/new_features/v179_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v179_features_doc.rst b/docs/source/Zh/doc/new_features/v179_features_doc.rst
@@ -0,0 +1,42 @@
+色彩感知樣板比對(HSV)
+========================
+
+``visual_match`` 的每個比對器都先轉灰階,因此形狀相同的紅色與綠色狀態指示燈對
+``match_template`` 而言*無法區分*——具辨別力的訊號被丟棄了。``color_region`` 找*已知*顏色的
+blob,卻無法以外觀對多色字形做樣板比對。``color_match`` 在 HSV 色相 / 飽和度通道上以色彩
+*距離*度量(``TM_SQDIFF_NORMED``,而非相關——相關會把絕對色相正規化掉,使紅→綠邊與黑→藍邊
+得分相同)進行比對,定位灰階比對會塌掉的色彩辨識目標。
+
+本功能重用 ``color_region`` 的 RGB 載入器與 ``visual_match`` 的 resize / NMS / ``Match``。
+``haystack`` 可注入;搜尋可在合成陣列上單元測試。不匯入 ``PySide6``。
+
+註:如同任何視窗度量,*平坦*的單色 patch 在各通道無變異——純色 blob 請用
+``find_color_region``;``color_match`` 適用於有色彩*結構*的目標。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import match_color, match_color_all
+
+    # 定位紅色狀態 chip,而非同形狀的綠色
+    hit = match_color("status_red.png", channels=("h", "s"), min_score=0.7)
+    if hit:
+        click(*hit.center)
+
+    for m in match_color_all("tag_green.png", channels=("h",)):
+        print(m.center, m.score)
+
+``match_color`` 在選定 ``channels``(預設 ``("h", "s")``;平坦飽和度目標用 ``("h",)``)中回傳
+最佳 ``Match``,或 ``None``。``match_color_all`` 回傳所有達到 ``min_score`` 的匹配,重疊以 NMS
+移除。
+
+執行器指令
+----------
+
+``AC_match_color``(``template`` / ``channels`` / ``min_score`` / ``scales`` / ``region`` →
+``{found, match}``)與 ``AC_match_color_all``(另加 ``max_results`` / ``nms_iou`` →
+``{count, matches}``)。兩者以 MCP 工具 ``ac_match_color`` / ``ac_match_color_all``(唯讀)及
+Script Builder 指令 **Match Template (colour/HSV)** / **Match Template All (colour/HSV)**
+(位於 **Image** 分類下)形式提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -201,6 +201,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v176_features_doc
    doc/new_features/v177_features_doc
    doc/new_features/v178_features_doc
+   doc/new_features/v179_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -295,6 +295,10 @@
 from je_auto_control.utils.match_ensemble import (
     match_ensemble, vote_centers,
 )
+# Colour-aware template matching on HSV channels
+from je_auto_control.utils.color_match import (
+    match_color, match_color_all,
+)
 # Otsu auto-thresholding for template matching (no hand-tuned min_score)
 from je_auto_control.utils.match_autothresh import (
     auto_threshold, match_auto,
@@ -1280,6 +1284,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "chamfer_distance",
     "match_ensemble",
     "vote_centers",
+    "match_color",
+    "match_color_all",
     "SubPixelMatch",
     "match_subpixel",
     "refine_peak",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -431,6 +431,33 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Vote candidate hit centres into one consensus target.",
     ))
+    specs.append(CommandSpec(
+        "AC_match_color", "Image", "Match Template (colour/HSV)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("channels", FieldType.STRING, optional=True,
+                      placeholder='["h", "s"]'),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.7,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Match by colour on HSV channels (red vs green, not grayscale).",
+    ))
+    specs.append(CommandSpec(
+        "AC_match_color_all", "Image", "Match Template All (colour/HSV)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("channels", FieldType.STRING, optional=True,
+                      placeholder='["h", "s"]'),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.7,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("max_results", FieldType.INT, optional=True, default=20),
+            FieldSpec("nms_iou", FieldType.FLOAT, optional=True, default=0.3,
+                      min_value=0.0, max_value=1.0),
+        ),
+        description="Find every colour (HSV) match of a template (NMS-deduped).",
+    ))
     specs.append(CommandSpec(
         "AC_grid_cells", "Image", "Grid Cells (coarse grounding)",
         fields=(
diff --git a/je_auto_control/utils/color_match/__init__.py b/je_auto_control/utils/color_match/__init__.py
@@ -0,0 +1,6 @@
+"""Colour-aware template matching on HSV channels."""
+from je_auto_control.utils.color_match.color_match import (
+    match_color, match_color_all,
+)
+
+__all__ = ["match_color", "match_color_all"]
diff --git a/je_auto_control/utils/color_match/color_match.py b/je_auto_control/utils/color_match/color_match.py
@@ -0,0 +1,95 @@
+"""Colour-aware template matching on HSV channels.
+
+Every matcher in ``visual_match`` converts to grayscale first, so a red versus green status
+indicator of identical shape is *indistinguishable* to ``match_template`` — the discriminating
+signal is thrown away. ``color_region`` finds blobs of a *known* colour but cannot template-match
+a multi-colour glyph by appearance. ``color_match`` correlates on the HSV hue / saturation
+channels (hue is illumination-invariant), so it locates colour-discriminated targets — a
+coloured chip, a syntax-highlighted token, a status light with structure — that grayscale
+matching collapses.
+
+It reuses ``color_region``'s RGB loaders and ``visual_match``'s resize / NMS / ``Match`` (no new
+image or matching code). The ``haystack`` is injectable; the search is unit-testable on synthetic
+arrays. OpenCV + NumPy are imported lazily. Imports no ``PySide6``.
+
+Note: like any NCC, a *flat* single-colour patch has no per-channel variance to correlate; for
+solid colour blobs use ``color_region``. ``color_match`` is for targets with colour *structure*.
+"""
+from typing import Any, List, Optional, Sequence
+
+from je_auto_control.utils.color_region.color_region import _grab_rgb, _to_rgb
+from je_auto_control.utils.visual_match.visual_match import Match, _nms, _resize
+
+ImageSource = Any
+_CHANNEL_INDEX = {"h": 0, "s": 1, "v": 2}
+
+
+def _hsv(source, region, is_haystack: bool):
+    import cv2
+    rgb = (_to_rgb(source) if source is not None
+           else _grab_rgb(region)) if is_haystack else _to_rgb(source)
+    return cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
+
+
+def _score_map(template_hsv, haystack_hsv, channels: Sequence[str]):
+    """Per-channel colour-distance score (higher = better) over the chosen channels.
+
+    Uses ``TM_SQDIFF_NORMED`` (not a correlation): correlation methods normalise away the
+    *absolute* hue, so a red→green edge and a black→blue edge correlate identically. Squared
+    distance keeps the absolute colour, so red is told from green. Score is ``1 - mean(sqdiff)``;
+    a flat channel (no variance, sqdiff undefined) contributes the worst score.
+    """
+    import cv2
+    import numpy as np
+    accumulator = None
+    for channel in channels:
+        index = _CHANNEL_INDEX[channel]
+        result = cv2.matchTemplate(haystack_hsv[:, :, index],
+                                   template_hsv[:, :, index], cv2.TM_SQDIFF_NORMED)
+        result = np.nan_to_num(result, nan=1.0, posinf=1.0)
+        accumulator = result if accumulator is None else accumulator + result
+    return 1.0 - accumulator / len(channels)
+
+
+def match_color(template: ImageSource, *, haystack: Optional[ImageSource] = None,
+                region: Optional[Sequence[int]] = None,
+                channels: Sequence[str] = ("h", "s"),
+                scales: Sequence[float] = (1.0,),
+                min_score: float = 0.7) -> Optional[Match]:
+    """Return the best colour (HSV-channel) match at or above ``min_score``, or ``None``."""
+    import cv2
+    template_hsv = _hsv(template, None, is_haystack=False)
+    haystack_hsv = _hsv(haystack, region, is_haystack=True)
+    best: Optional[Match] = None
+    for scale in scales:
+        scaled = _resize(template_hsv, float(scale))
+        if scaled.shape[0] > haystack_hsv.shape[0] \
+                or scaled.shape[1] > haystack_hsv.shape[1]:
+            continue
+        _, max_val, _, max_loc = cv2.minMaxLoc(
+            _score_map(scaled, haystack_hsv, channels))
+        if max_val >= min_score and (best is None or max_val > best.score):
+            best = Match(int(max_loc[0]), int(max_loc[1]), scaled.shape[1],
+                         scaled.shape[0], round(float(max_val), 4), float(scale))
+    return best
+
+
+def match_color_all(template: ImageSource, *,
+                    haystack: Optional[ImageSource] = None,
+                    region: Optional[Sequence[int]] = None,
+                    channels: Sequence[str] = ("h", "s"), min_score: float = 0.7,
+                    max_results: int = 20, nms_iou: float = 0.3) -> List[Match]:
+    """Return every colour match >= ``min_score`` (scale 1.0), overlaps removed (NMS)."""
+    import numpy as np
+    template_hsv = _hsv(template, None, is_haystack=False)
+    haystack_hsv = _hsv(haystack, region, is_haystack=True)
+    if template_hsv.shape[0] > haystack_hsv.shape[0] \
+            or template_hsv.shape[1] > haystack_hsv.shape[1]:
+        return []
+    score_map = _score_map(template_hsv, haystack_hsv, channels)
+    height, width = template_hsv.shape[:2]
+    ys, xs = np.nonzero(score_map >= float(min_score))
+    candidates = [Match(int(x), int(y), width, height,
+                        round(float(score_map[y, x]), 4), 1.0)
+                  for y, x in zip(ys, xs)]
+    return _nms(candidates, float(nms_iou))[:int(max_results)]
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3426,6 +3426,39 @@ def _match_ensemble(templates: Any, min_score: Any = 0.8, agree_px: Any = 10,
     return {"found": result is not None, "result": result}
 
 
+def _match_color(template: str, channels: Any = None, min_score: Any = 0.7,
+                 scales: Any = None, region: Any = None) -> Dict[str, Any]:
+    """Adapter: best colour (HSV-channel) template match on the screen."""
+    import json
+    from je_auto_control.utils.color_match import match_color
+    if isinstance(channels, str):
+        channels = json.loads(channels) if channels.strip() else None
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    match = match_color(template, region=region,
+                        channels=tuple(channels) if channels else ("h", "s"),
+                        scales=_seq_arg(scales, (1.0,)), min_score=float(min_score))
+    return {"found": match is not None,
+            "match": match.to_dict() if match else None}
+
+
+def _match_color_all(template: str, channels: Any = None, min_score: Any = 0.7,
+                     max_results: Any = 20, nms_iou: Any = 0.3,
+                     region: Any = None) -> Dict[str, Any]:
+    """Adapter: every colour (HSV-channel) match on the screen (NMS)."""
+    import json
+    from je_auto_control.utils.color_match import match_color_all
+    if isinstance(channels, str):
+        channels = json.loads(channels) if channels.strip() else None
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    matches = match_color_all(template, region=region,
+                              channels=tuple(channels) if channels else ("h", "s"),
+                              min_score=float(min_score),
+                              max_results=int(max_results), nms_iou=float(nms_iou))
+    return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
+
+
 def _region_arg(value: Any) -> Optional[List[int]]:
     """Coerce a JSON-string / list region arg into a list of ints, or None."""
     import json
@@ -6151,6 +6184,8 @@ def __init__(self):
             "AC_match_subpixel": _match_subpixel,
             "AC_match_ensemble": _match_ensemble,
             "AC_vote_centers": _vote_centers,
+            "AC_match_color": _match_color,
+            "AC_match_color_all": _match_color_all,
             "AC_grid_cells": _grid_cells,
             "AC_cell_for_point": _cell_for_point,
             "AC_point_for_cell": _point_for_cell,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
diff --git a/test/unit_test/headless/test_color_match_batch.py b/test/unit_test/headless/test_color_match_batch.py
