Skip to content

Commit 538a6b4

Browse files
authored
Merge pull request #407 from Integration-Automation/feat/scale-detect-batch
Add scale_detect: infer display scale / visual DPI from a template
2 parents debb186 + f70d7d4 commit 538a6b4

11 files changed

Lines changed: 363 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-24) — Display-Scale / Visual-DPI Detection
4+
5+
Infer which display scale (DPI) a template renders at — and how confidently. Full reference: [`docs/source/Eng/doc/new_features/v189_features_doc.rst`](docs/source/Eng/doc/new_features/v189_features_doc.rst).
6+
7+
- **`detect_scale` / `scale_sweep`** (`AC_detect_scale`, `AC_scale_sweep`): a template cropped at 100% scale won't match on a 150%-DPI machine, and `match_template` returns only the single best match — discarding the per-scale scores. This keeps the whole profile: `scale_sweep` scores the template at every scale, and `detect_scale` reports the winning scale as a DPI inference (`scale_percent`) with a confidence `margin` (how far it beats the runner-up). Reuses `visual_match._score_map` per scale; source is any ndarray / path / PIL image (or the live screen); scales default to the common Windows values. cv2/numpy lazily imported. No `PySide6`.
8+
39
## What's new (2026-06-24) — Image Quality Scoring (sharpness / contrast / brightness gate)
410

511
Refuse to OCR a blurry or washed-out frame — score quality and gate before recognition. Full reference: [`docs/source/Eng/doc/new_features/v188_features_doc.rst`](docs/source/Eng/doc/new_features/v188_features_doc.rst).
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Display-Scale / Visual-DPI Detection
2+
====================================
3+
4+
A template cropped at 100% display scale will not match pixel-for-pixel on a
5+
machine running at 150% DPI — everything is 1.5x bigger. ``visual_match.
6+
match_template`` *can* sweep scales, but it returns only the single best match's
7+
location and throws the per-scale scores away. ``scale_detect`` keeps the whole
8+
profile: it scores the template against the haystack at a range of scales and
9+
reports **which scale wins, by how much**, so an automation can infer the
10+
effective UI scale / DPI and how confident that inference is.
11+
12+
* :func:`scale_sweep` — the per-scale score profile (every scale's best match),
13+
* :func:`detect_scale` — the winning scale as a DPI inference with a confidence
14+
margin.
15+
16+
It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface,
17+
oriented higher = better) for each scale, so the source is any ndarray / path /
18+
PIL image (or the live screen). cv2 / numpy are lazily imported. Imports no
19+
``PySide6``.
20+
21+
Headless API
22+
------------
23+
24+
.. code-block:: python
25+
26+
from je_auto_control import detect_scale, scale_sweep
27+
28+
detect_scale("button.png", "screen.png")
29+
# {"scale": 1.5, "scale_percent": 150, "score": 0.98, "center": [...],
30+
# "margin": 0.62, "candidates": [...]}
31+
32+
scale_sweep("button.png", scales=[1.0, 1.25, 1.5, 1.75, 2.0])
33+
# [{"scale": 1.0, "score": .., "center": [..]}, {"scale": 1.25, ...}, ...]
34+
35+
``scales`` defaults to the common Windows display scales
36+
``(1.0, 1.25, 1.5, 1.75, 2.0)``. ``margin`` is how far the winning scale beats the
37+
runner-up — a low margin means the inference is ambiguous. Scales at which the
38+
template is larger than the haystack are skipped; ``detect_scale`` returns
39+
``None`` when none fit. Omit ``haystack`` to match against the live screen
40+
(``region`` applies to that grab).
41+
42+
Executor commands
43+
-----------------
44+
45+
``AC_detect_scale`` and ``AC_scale_sweep`` (``template`` / ``haystack`` /
46+
``region`` / ``scales`` / ``method``). They are exposed as read-only ``ac_*`` MCP
47+
tools and as Script Builder commands under **Image**.
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
顯示縮放 / 視覺 DPI 偵測
2+
=======================
3+
4+
在 100% 顯示縮放下裁切的模板,在 150% DPI 的機器上不會逐像素吻合——一切都放大了 1.5 倍。
5+
``visual_match.match_template`` *可以* 掃過多個縮放,但它只回傳單一最佳吻合的位置,並把各縮放的
6+
分數丟棄。``scale_detect`` 保留整個剖面:它在一系列縮放下對 haystack 評分模板,並回報**哪個縮放
7+
勝出、勝出多少**,讓自動化能推測有效的 UI 縮放 / DPI,以及該推測的信心。
8+
9+
* :func:`scale_sweep` ——逐縮放的分數剖面(每個縮放的最佳吻合),
10+
* :func:`detect_scale` ——勝出的縮放作為 DPI 推測,並附信心 margin。
11+
12+
它對每個縮放重用 ``visual_match._score_map``(完整的 ``matchTemplate`` 表面,方向為越高越好),
13+
因此來源可為任何 ndarray / 路徑 / PIL 影像(或存活螢幕)。cv2 / numpy 為延遲匯入。不匯入
14+
``PySide6``。
15+
16+
無頭 API
17+
--------
18+
19+
.. code-block:: python
20+
21+
from je_auto_control import detect_scale, scale_sweep
22+
23+
detect_scale("button.png", "screen.png")
24+
# {"scale": 1.5, "scale_percent": 150, "score": 0.98, "center": [...],
25+
# "margin": 0.62, "candidates": [...]}
26+
27+
scale_sweep("button.png", scales=[1.0, 1.25, 1.5, 1.75, 2.0])
28+
# [{"scale": 1.0, "score": .., "center": [..]}, {"scale": 1.25, ...}, ...]
29+
30+
``scales`` 預設為常見的 Windows 顯示縮放 ``(1.0, 1.25, 1.5, 1.75, 2.0)``。``margin`` 是勝出縮放
31+
領先次佳者的幅度——margin 低代表推測模稜兩可。模板大於 haystack 的縮放會被略過;當沒有任何縮放
32+
吻合時 ``detect_scale`` 回傳 ``None``。省略 ``haystack`` 即對存活螢幕比對(``region`` 套用於該
33+
擷取)。
34+
35+
執行器指令
36+
----------
37+
38+
``AC_detect_scale`` 與 ``AC_scale_sweep``(``template`` / ``haystack`` / ``region`` /
39+
``scales`` / ``method``)。皆以唯讀 ``ac_*`` MCP 工具及 Script Builder 指令(位於 **Image**
40+
分類下)形式提供。

je_auto_control/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,8 @@
8282
from je_auto_control.utils.image_quality import (
8383
image_quality, is_blurry, quality_gate,
8484
)
85+
# Display-scale / visual-DPI detection (per-scale match profile)
86+
from je_auto_control.utils.scale_detect import detect_scale, scale_sweep
8587
# VLM element locator (headless)
8688
from je_auto_control.utils.vision import (
8789
VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1657,6 +1659,7 @@ def start_autocontrol_gui(*args, **kwargs):
16571659
"list_clipboard_formats", "clipboard_formats",
16581660
"plan_file_drop", "drop_files",
16591661
"image_quality", "is_blurry", "quality_gate",
1662+
"detect_scale", "scale_sweep",
16601663
# VLM locator
16611664
"VLMNotAvailableError", "locate_by_description", "click_by_description",
16621665
"verify_description",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -763,6 +763,30 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
763763
),
764764
description="Pass/fail an image for OCR readability with named issues.",
765765
))
766+
specs.append(CommandSpec(
767+
"AC_detect_scale", "Image", "Detect Display Scale (DPI)",
768+
fields=(
769+
FieldSpec("template", FieldType.FILE_PATH),
770+
FieldSpec("haystack", FieldType.FILE_PATH, optional=True),
771+
FieldSpec("region", FieldType.STRING, optional=True,
772+
placeholder=_REGION_PLACEHOLDER),
773+
FieldSpec("scales", FieldType.STRING, optional=True,
774+
placeholder="[1.0, 1.25, 1.5, 1.75, 2.0]"),
775+
),
776+
description="Infer the display scale a template renders at (visual DPI).",
777+
))
778+
specs.append(CommandSpec(
779+
"AC_scale_sweep", "Image", "Scale Sweep (per-scale scores)",
780+
fields=(
781+
FieldSpec("template", FieldType.FILE_PATH),
782+
FieldSpec("haystack", FieldType.FILE_PATH, optional=True),
783+
FieldSpec("region", FieldType.STRING, optional=True,
784+
placeholder=_REGION_PLACEHOLDER),
785+
FieldSpec("scales", FieldType.STRING, optional=True,
786+
placeholder="[1.0, 1.25, 1.5, 1.75, 2.0]"),
787+
),
788+
description="Per-scale match-score profile of a template.",
789+
))
766790
specs.append(CommandSpec(
767791
"AC_changed_regions", "Image", "Changed Regions (motion)",
768792
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4298,6 +4298,35 @@ def _quality_gate(source: Any = None, region: Any = None,
42984298
min_contrast=float(min_contrast))
42994299

43004300

4301+
def _coerce_scales(scales: Any):
4302+
"""Normalise a scales argument (JSON '[1.0,1.5]' string / list / None)."""
4303+
import json
4304+
if isinstance(scales, str):
4305+
return json.loads(scales) if scales.strip() else None
4306+
return scales
4307+
4308+
4309+
def _detect_scale(template: Any, haystack: Any = None, region: Any = None,
4310+
scales: Any = None,
4311+
method: str = "ccoeff_normed") -> Dict[str, Any]:
4312+
"""Adapter: infer the display scale a template renders at (visual DPI)."""
4313+
from je_auto_control.utils.scale_detect import detect_scale
4314+
result = detect_scale(template, haystack, region=_coerce_region(region),
4315+
scales=_coerce_scales(scales), method=str(method))
4316+
return {"found": result is not None, "result": result}
4317+
4318+
4319+
def _scale_sweep(template: Any, haystack: Any = None, region: Any = None,
4320+
scales: Any = None,
4321+
method: str = "ccoeff_normed") -> Dict[str, Any]:
4322+
"""Adapter: per-scale match-score profile of a template."""
4323+
from je_auto_control.utils.scale_detect import scale_sweep
4324+
return {"sweep": scale_sweep(template, haystack,
4325+
region=_coerce_region(region),
4326+
scales=_coerce_scales(scales),
4327+
method=str(method))}
4328+
4329+
43014330
def _image_histogram(source: Any = None, bins: Any = 32, space: str = "hsv",
43024331
region: Any = None) -> Dict[str, Any]:
43034332
"""Adapter: per-channel colour histogram of an image / the screen."""
@@ -6522,6 +6551,8 @@ def __init__(self):
65226551
"AC_drop_files": _drop_files,
65236552
"AC_image_quality": _image_quality,
65246553
"AC_quality_gate": _quality_gate,
6554+
"AC_detect_scale": _detect_scale,
6555+
"AC_scale_sweep": _scale_sweep,
65256556
"AC_image_histogram": _image_histogram,
65266557
"AC_histogram_changed": _histogram_changed,
65276558
"AC_changed_regions": _changed_regions,

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3411,6 +3411,38 @@ def img_histogram_tools() -> List[MCPTool]:
34113411
handler=h.quality_gate,
34123412
annotations=READ_ONLY,
34133413
),
3414+
MCPTool(
3415+
name="ac_detect_scale",
3416+
description=("Infer the display scale a 'template' renders at (visual "
3417+
"DPI) by scoring it against 'haystack' (default screen) "
3418+
"across 'scales'. Returns {found, result:{scale, "
3419+
"scale_percent, score, center, margin, candidates}}."),
3420+
input_schema=schema({
3421+
"template": {"type": "string"},
3422+
"haystack": {"type": "string"},
3423+
"region": {"type": "array", "items": {"type": "integer"}},
3424+
"scales": {"type": "array", "items": {"type": "number"}},
3425+
"method": {"type": "string"}},
3426+
required=["template"]),
3427+
handler=h.detect_scale,
3428+
annotations=READ_ONLY,
3429+
),
3430+
MCPTool(
3431+
name="ac_scale_sweep",
3432+
description=("Per-scale match-score profile of a 'template' against "
3433+
"'haystack' (default screen): {sweep:[{scale, score, x, "
3434+
"y, width, height, center}]} — the raw scores match_"
3435+
"template discards."),
3436+
input_schema=schema({
3437+
"template": {"type": "string"},
3438+
"haystack": {"type": "string"},
3439+
"region": {"type": "array", "items": {"type": "integer"}},
3440+
"scales": {"type": "array", "items": {"type": "number"}},
3441+
"method": {"type": "string"}},
3442+
required=["template"]),
3443+
handler=h.scale_sweep,
3444+
annotations=READ_ONLY,
3445+
),
34143446
]
34153447

34163448

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2520,6 +2520,18 @@ def quality_gate(source=None, region=None, min_sharpness=100.0,
25202520
return _quality_gate(source, region, min_sharpness, min_contrast)
25212521

25222522

2523+
def detect_scale(template, haystack=None, region=None, scales=None,
2524+
method="ccoeff_normed"):
2525+
from je_auto_control.utils.executor.action_executor import _detect_scale
2526+
return _detect_scale(template, haystack, region, scales, method)
2527+
2528+
2529+
def scale_sweep(template, haystack=None, region=None, scales=None,
2530+
method="ccoeff_normed"):
2531+
from je_auto_control.utils.executor.action_executor import _scale_sweep
2532+
return _scale_sweep(template, haystack, region, scales, method)
2533+
2534+
25232535
def image_histogram(source=None, bins=32, space="hsv", region=None):
25242536
from je_auto_control.utils.executor.action_executor import _image_histogram
25252537
return _image_histogram(source, bins, space, region)
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Detect the display scale / visual DPI a template renders at (per-scale profile)."""
2+
from je_auto_control.utils.scale_detect.scale_detect import (
3+
detect_scale, scale_sweep,
4+
)
5+
6+
__all__ = ["detect_scale", "scale_sweep"]
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
"""Detect the display scale a template renders at (visual DPI).
2+
3+
A template cropped at 100% display scale will not match pixel-for-pixel on a
4+
machine running at 150% DPI — everything is 1.5x bigger. ``visual_match.
5+
match_template`` *can* sweep scales, but it returns only the single best match's
6+
location and throws the per-scale scores away. ``scale_detect`` keeps the whole
7+
profile: it scores the template against the haystack at a range of scales and
8+
reports *which scale wins, by how much*, so an automation can infer the effective
9+
UI scale / DPI and how confident that inference is.
10+
11+
It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface,
12+
oriented higher = better) for each scale, so the source is any ndarray / path /
13+
PIL image (or the live screen). cv2 / numpy are lazily imported. Imports no
14+
``PySide6``.
15+
"""
16+
from typing import Any, Dict, List, Optional, Sequence
17+
18+
ImageSource = Any
19+
# Common Windows display scales (100% / 125% / 150% / 175% / 200%).
20+
_DEFAULT_SCALES = (1.0, 1.25, 1.5, 1.75, 2.0)
21+
22+
23+
def _score_at(template: ImageSource, haystack: Optional[ImageSource],
24+
region: Optional[Sequence[int]], method: str,
25+
scale: float) -> Optional[Dict[str, Any]]:
26+
import cv2
27+
from je_auto_control.utils.visual_match.visual_match import _score_map
28+
score_map, tmpl = _score_map(template, haystack, region=region,
29+
method=method, scale=scale)
30+
if score_map is None:
31+
return None # template larger than haystack at this scale
32+
_min_v, max_v, _min_loc, max_loc = cv2.minMaxLoc(score_map)
33+
height, width = int(tmpl.shape[0]), int(tmpl.shape[1])
34+
x, y = int(max_loc[0]), int(max_loc[1])
35+
return {"scale": float(scale), "score": float(max_v), "x": x, "y": y,
36+
"width": width, "height": height,
37+
"center": [x + width // 2, y + height // 2]}
38+
39+
40+
def scale_sweep(template: ImageSource, haystack: Optional[ImageSource] = None, *,
41+
region: Optional[Sequence[int]] = None,
42+
scales: Optional[Sequence[float]] = None,
43+
method: str = "ccoeff_normed") -> List[Dict[str, Any]]:
44+
"""Score ``template`` against the haystack at each scale.
45+
46+
Returns ``[{scale, score, x, y, width, height, center}]`` (best match per
47+
scale), skipping scales at which the template is larger than the haystack.
48+
"""
49+
chosen = tuple(scales) if scales else _DEFAULT_SCALES
50+
results = []
51+
for scale in chosen:
52+
entry = _score_at(template, haystack, region, method, float(scale))
53+
if entry is not None:
54+
results.append(entry)
55+
return results
56+
57+
58+
def detect_scale(template: ImageSource, haystack: Optional[ImageSource] = None, *,
59+
region: Optional[Sequence[int]] = None,
60+
scales: Optional[Sequence[float]] = None,
61+
method: str = "ccoeff_normed") -> Optional[Dict[str, Any]]:
62+
"""Infer the display scale ``template`` renders at (its visual DPI).
63+
64+
Returns ``{scale, scale_percent, score, center, margin, candidates}`` — the
65+
winning scale, its percentage, the match score, and ``margin`` (how far it
66+
beats the runner-up: a confidence in the inference). ``None`` if no scale
67+
matched (template never fit the haystack).
68+
"""
69+
sweep = scale_sweep(template, haystack, region=region, scales=scales,
70+
method=method)
71+
if not sweep:
72+
return None
73+
ranked = sorted(sweep, key=lambda entry: entry["score"], reverse=True)
74+
best = ranked[0]
75+
margin = best["score"] - ranked[1]["score"] if len(ranked) > 1 else best["score"]
76+
return {"scale": best["scale"], "scale_percent": round(best["scale"] * 100),
77+
"score": best["score"], "center": best["center"],
78+
"margin": float(margin), "candidates": sweep}

0 commit comments

Comments
 (0)