Skip to content

Commit 1e4a2db

Browse files
authored
Merge pull request #443 from Integration-Automation/feat/marks-layout-batch
Add marks_layout: non-overlapping Set-of-Marks labels with readable colour
2 parents ebe3fbc + d798e9d commit 1e4a2db

11 files changed

Lines changed: 384 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,12 @@
22

33
## What's new (2026-06-26)
44

5+
### Set-of-Marks Label Layout (No Overlap, Readable Colour)
6+
7+
Number every element without the labels piling up or vanishing into the background. Full reference: [`docs/source/Eng/doc/new_features/v215_features_doc.rst`](docs/source/Eng/doc/new_features/v215_features_doc.rst).
8+
9+
- **`place_labels` / `label_color`** (`AC_place_labels`, `AC_label_color`): Set-of-Marks draws each numbered label at a fixed offset, so on dense UIs the numbers pile on top of each other and a dark label on a dark element vanishes. `place_labels` is greedy non-overlap placement — for each mark it tries a ring of candidate positions around its box (above/below/inside, left/right aligned) and takes the first that stays in bounds and clears every already-placed label; `label_color` picks black or white by whichever has the better WCAG contrast against the element background (reusing `a11y_audit.contrast_ratio`). Pure standard library, deterministic, fully testable without rendering. Second feature of the ROUND-15 perception lane. No `PySide6`.
10+
511
### Colour-Vision-Deficiency Simulation + Collision Check
612

713
Check whether your red/green status colours are distinguishable to colour-blind users. Full reference: [`docs/source/Eng/doc/new_features/v214_features_doc.rst`](docs/source/Eng/doc/new_features/v214_features_doc.rst).
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
Set-of-Marks Label Layout (No Overlap, Readable Colour)
2+
=======================================================
3+
4+
Set-of-Marks overlays a numbered label on every element so a vision model can
5+
say "click 7". ``set_of_marks`` draws each label at a fixed offset, so on dense
6+
UIs the numbers pile on top of each other (unreadable) and a dark label on a
7+
dark element vanishes. ``marks_layout`` fixes both with pure geometry.
8+
9+
* :func:`place_labels` — greedy non-overlap placement: for each mark, try a ring
10+
of candidate positions around its box (above, below, inside; left/right
11+
aligned) and take the first that stays in bounds and clears every
12+
already-placed label.
13+
* :func:`label_color` — pick the label text colour (black or white) with the
14+
better WCAG contrast against the element's background.
15+
16+
Pure standard library; reuses :func:`a11y_audit.contrast_ratio`. Fully testable
17+
without rendering. Imports no ``PySide6``.
18+
19+
Headless API
20+
------------
21+
22+
.. code-block:: python
23+
24+
from je_auto_control import mark_elements, place_labels, label_color
25+
26+
marks = mark_elements(elements) # [{id, bbox, ...}]
27+
layout = place_labels(marks, bounds=(1920, 1080))
28+
# [{'id': 1, 'label': [x, y, 22, 16], 'anchor': [bx, by]}, ...]
29+
30+
label_color((30, 30, 30)) # {'rgb': [255, 255, 255], 'contrast': ...}
31+
32+
Feed the ``label`` boxes from :func:`place_labels` to your renderer instead of a
33+
naive fixed offset, and pick each number's colour with :func:`label_color` so it
34+
stays legible on its background. ``place_labels`` is deterministic and ordered by
35+
the input marks, so the same screen always numbers the same way.
36+
37+
Executor commands
38+
-----------------
39+
40+
``AC_place_labels`` (``marks`` JSON list + ``label_width`` / ``label_height`` /
41+
``bounds`` ``[w, h]`` → ``{labels}``) and ``AC_label_color`` (``background``
42+
``[r, g, b]`` → ``{rgb, contrast}``). They are the matching read-only ``ac_*``
43+
MCP tools and Script Builder commands under **Image**.
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
Set-of-Marks 標籤佈局(不重疊、可讀顏色)
2+
=========================================
3+
4+
Set-of-Marks 在每個元素上疊一個編號標籤,讓視覺模型能說「點 7」。``set_of_marks`` 以固定偏移繪製
5+
每個標籤,故在密集 UI 上數字會互相疊壓(難以辨讀),而深色標籤在深色元素上會消失。``marks_layout``
6+
以純幾何修正兩者。
7+
8+
* :func:`place_labels` ——貪婪式不重疊放置:對每個 mark,在其方框周圍嘗試一圈候選位置
9+
(上、下、內;左/右對齊),取第一個仍在邊界內且不與任何已放置標籤重疊者。
10+
* :func:`label_color` ——挑選標籤文字顏色(黑或白),取對元素背景 WCAG 對比較佳者。
11+
12+
純標準函式庫;重用 :func:`a11y_audit.contrast_ratio`。無需繪製即可完整測試。不匯入 ``PySide6``。
13+
14+
無頭 API
15+
--------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import mark_elements, place_labels, label_color
20+
21+
marks = mark_elements(elements) # [{id, bbox, ...}]
22+
layout = place_labels(marks, bounds=(1920, 1080))
23+
# [{'id': 1, 'label': [x, y, 22, 16], 'anchor': [bx, by]}, ...]
24+
25+
label_color((30, 30, 30)) # {'rgb': [255, 255, 255], 'contrast': ...}
26+
27+
:func:`place_labels` 產生的 ``label`` 方框餵給你的繪製器(取代固定偏移),並用 :func:`label_color`
28+
挑選每個編號的顏色,使其在背景上維持可讀。``place_labels`` 是確定性的且依輸入 marks 排序,
29+
故同一畫面總是以相同方式編號。
30+
31+
執行器指令
32+
----------
33+
34+
``AC_place_labels``(``marks`` JSON 清單加上 ``label_width`` / ``label_height`` /
35+
``bounds`` ``[w, h]`` → ``{labels}``)與 ``AC_label_color``(``background``
36+
``[r, g, b]`` → ``{rgb, contrast}``)。皆以對應的唯讀 ``ac_*`` MCP 工具及 Script Builder 指令
37+
(位於 **Image** 分類下)形式提供。

je_auto_control/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,8 @@
135135
from je_auto_control.utils.cvd_simulate import (
136136
color_distance, colors_collide, simulate_cvd,
137137
)
138+
# Lay out Set-of-Marks labels without overlap + readable colour
139+
from je_auto_control.utils.marks_layout import label_color, place_labels
138140
# Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
139141
from je_auto_control.utils.clipboard_rich_formats import (
140142
build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
@@ -1760,6 +1762,7 @@ def start_autocontrol_gui(*args, **kwargs):
17601762
"ensure_state", "ensure_toggle",
17611763
"wait_until_app_idle", "idle_point",
17621764
"simulate_cvd", "colors_collide", "color_distance",
1765+
"place_labels", "label_color",
17631766
"build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
17641767
"set_clipboard_rtf", "get_clipboard_rtf",
17651768
"set_clipboard_csv", "get_clipboard_csv",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4547,6 +4547,26 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None:
45474547
),
45484548
description="Whether two colours become confusable under a CVD type.",
45494549
))
4550+
specs.append(CommandSpec(
4551+
"AC_place_labels", "Image", "Place Mark Labels",
4552+
fields=(
4553+
FieldSpec("marks", FieldType.STRING,
4554+
placeholder="JSON list of {id, bbox}"),
4555+
FieldSpec("label_width", FieldType.INT, optional=True, default=22),
4556+
FieldSpec("label_height", FieldType.INT, optional=True,
4557+
default=16),
4558+
FieldSpec("bounds", FieldType.STRING, optional=True,
4559+
placeholder="[width, height]"),
4560+
),
4561+
description="Lay out non-overlapping Set-of-Marks label boxes.",
4562+
))
4563+
specs.append(CommandSpec(
4564+
"AC_label_color", "Image", "Label Colour for Background",
4565+
fields=(
4566+
FieldSpec("background", FieldType.STRING, placeholder="[r, g, b]"),
4567+
),
4568+
description="Higher-contrast label colour (black/white) for a background.",
4569+
))
45504570
specs.append(CommandSpec(
45514571
"AC_normalize_ext", "Shell", "Normalize Extension",
45524572
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2863,6 +2863,23 @@ def _colors_collide(left: Any, right: Any, kind: Any = "deuteranopia",
28632863
threshold=float(threshold))
28642864

28652865

2866+
def _place_labels(marks: Any, label_width: Any = 22, label_height: Any = 16,
2867+
bounds: Any = None) -> Dict[str, Any]:
2868+
"""Adapter: lay out non-overlapping Set-of-Marks label boxes (pure)."""
2869+
from je_auto_control.utils.marks_layout import place_labels
2870+
items = _coerce_list(marks) if marks else []
2871+
limit = _coerce_list(bounds) if bounds else None
2872+
labels = place_labels(items, label_width=int(label_width),
2873+
label_height=int(label_height), bounds=limit)
2874+
return {"labels": labels}
2875+
2876+
2877+
def _label_color(background: Any) -> Dict[str, Any]:
2878+
"""Adapter: the higher-contrast label colour for a background (pure)."""
2879+
from je_auto_control.utils.marks_layout import label_color
2880+
return label_color(_coerce_rgb(background))
2881+
2882+
28662883
def _normalize_ext(target: str) -> Dict[str, Any]:
28672884
"""Adapter: the lowercased extension of a path / bare ext (pure)."""
28682885
from je_auto_control.utils.file_assoc import normalize_ext
@@ -6896,6 +6913,8 @@ def __init__(self):
68966913
"AC_idle_point": _idle_point,
68976914
"AC_simulate_cvd": _simulate_cvd,
68986915
"AC_colors_collide": _colors_collide,
6916+
"AC_place_labels": _place_labels,
6917+
"AC_label_color": _label_color,
68996918
"AC_normalize_ext": _normalize_ext,
69006919
"AC_file_association": _file_association,
69016920
"AC_get_control_text": _get_control_text,
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Place Set-of-Marks labels without overlap, with readable label colours."""
2+
from je_auto_control.utils.marks_layout.marks_layout import (
3+
label_color, place_labels,
4+
)
5+
6+
__all__ = ["place_labels", "label_color"]
Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
"""Place Set-of-Marks labels so they don't overlap, with readable label colours.
2+
3+
Set-of-Marks overlays a numbered label on every element so a vision model can
4+
say "click 7". ``set_of_marks`` draws each label at a fixed offset, so on dense
5+
UIs the numbers pile on top of each other (unreadable) and a dark label on a
6+
dark element vanishes. ``marks_layout`` fixes both with pure geometry:
7+
8+
* :func:`place_labels` — greedy non-overlap placement: for each mark, try a ring
9+
of candidate positions around its box and take the first that stays in bounds
10+
and clears every already-placed label.
11+
* :func:`label_color` — pick the label text colour (black or white) with the
12+
better WCAG contrast against the element's background.
13+
14+
Pure standard library; reuses :func:`a11y_audit.contrast_ratio`. Fully testable
15+
without rendering. Imports no ``PySide6``.
16+
"""
17+
from typing import Any, Dict, List, Optional, Sequence, Tuple
18+
19+
Rect = Tuple[int, int, int, int]
20+
21+
_BLACK = (0, 0, 0)
22+
_WHITE = (255, 255, 255)
23+
24+
25+
def _overlap(first: Rect, second: Rect) -> bool:
26+
"""Whether two ``(x, y, w, h)`` rectangles overlap (pure)."""
27+
ax, ay, aw, ah = first
28+
bx, by, bw, bh = second
29+
return not (ax + aw <= bx or bx + bw <= ax
30+
or ay + ah <= by or by + bh <= ay)
31+
32+
33+
def _in_bounds(rect: Rect, bounds: Tuple[int, int]) -> bool:
34+
"""Whether ``rect`` fits inside ``(width, height)`` (pure)."""
35+
x, y, w, h = rect
36+
return x >= 0 and y >= 0 and x + w <= int(bounds[0]) \
37+
and y + h <= int(bounds[1])
38+
39+
40+
def _candidates(bbox: Sequence[int], label_w: int,
41+
label_h: int) -> List[Tuple[int, int]]:
42+
"""Candidate label top-left positions around an anchor box (pure)."""
43+
bx, by, bw, bh = (int(bbox[0]), int(bbox[1]), int(bbox[2]), int(bbox[3]))
44+
right = bx + bw - label_w
45+
below = by + bh
46+
return [
47+
(bx, by - label_h), # above, left-aligned (default SoM spot)
48+
(right, by - label_h), # above, right-aligned
49+
(bx, below), # below, left-aligned
50+
(right, below), # below, right-aligned
51+
(bx, by), # inside, top-left
52+
(right, by), # inside, top-right
53+
]
54+
55+
56+
def _clamp_to_bounds(rect: Rect, bounds: Tuple[int, int]) -> Rect:
57+
"""Shift ``rect`` to fit inside ``(width, height)`` (pure fallback)."""
58+
x, y, w, h = rect
59+
x = max(0, min(int(bounds[0]) - w, x))
60+
y = max(0, min(int(bounds[1]) - h, y))
61+
return (x, y, w, h)
62+
63+
64+
def _pick_position(bbox: Sequence[int], label_w: int, label_h: int,
65+
bounds: Optional[Tuple[int, int]],
66+
placed: List[Rect]) -> Rect:
67+
"""Pick the first candidate that is in bounds and clears placed labels."""
68+
fallback: Optional[Rect] = None
69+
for cx, cy in _candidates(bbox, label_w, label_h):
70+
rect = (cx, cy, label_w, label_h)
71+
if fallback is None:
72+
fallback = rect
73+
if bounds is not None and not _in_bounds(rect, bounds):
74+
continue
75+
if any(_overlap(rect, other) for other in placed):
76+
continue
77+
return rect
78+
if bounds is not None and fallback is not None:
79+
return _clamp_to_bounds(fallback, bounds)
80+
return fallback if fallback is not None else (0, 0, label_w, label_h)
81+
82+
83+
def place_labels(marks: Sequence[Dict[str, Any]], *, label_width: int = 22,
84+
label_height: int = 16,
85+
bounds: Optional[Sequence[int]] = None
86+
) -> List[Dict[str, Any]]:
87+
"""Lay out non-overlapping label boxes for ``marks`` (pure).
88+
89+
``marks`` is the :func:`set_of_marks.mark_elements` output (each has an
90+
``id`` and ``bbox`` ``[x, y, w, h]``). ``bounds`` is the ``(width, height)``
91+
the labels must stay within. Returns ``[{id, label, anchor}]`` where
92+
``label`` is the placed ``[x, y, w, h]`` box.
93+
"""
94+
size = (int(label_width), int(label_height))
95+
limit = (int(bounds[0]), int(bounds[1])) if bounds else None
96+
placed: List[Rect] = []
97+
results: List[Dict[str, Any]] = []
98+
for mark in marks:
99+
bbox = [int(value) for value in mark["bbox"][:4]]
100+
rect = _pick_position(bbox, size[0], size[1], limit, placed)
101+
placed.append(rect)
102+
results.append({"id": mark.get("id"), "label": list(rect),
103+
"anchor": [bbox[0], bbox[1]]})
104+
return results
105+
106+
107+
def label_color(background: Sequence[float]) -> Dict[str, Any]:
108+
"""Pick the higher-contrast label text colour for ``background`` (pure).
109+
110+
Returns ``{rgb, contrast}`` — black or white, whichever has the better WCAG
111+
contrast ratio against the element background colour.
112+
"""
113+
from je_auto_control.utils.a11y_audit import contrast_ratio
114+
black_contrast = contrast_ratio(background, _BLACK)
115+
white_contrast = contrast_ratio(background, _WHITE)
116+
if white_contrast >= black_contrast:
117+
return {"rgb": list(_WHITE), "contrast": round(white_contrast, 3)}
118+
return {"rgb": list(_BLACK), "contrast": round(black_contrast, 3)}

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4030,6 +4030,33 @@ def img_histogram_tools() -> List[MCPTool]:
40304030
handler=h.colors_collide,
40314031
annotations=READ_ONLY,
40324032
),
4033+
MCPTool(
4034+
name="ac_place_labels",
4035+
description=("Lay out non-overlapping Set-of-Marks label boxes for "
4036+
"'marks' (each {id, bbox:[x,y,w,h]}). 'bounds' is "
4037+
"[width, height] to stay within. Pure. Returns "
4038+
"{labels:[{id, label:[x,y,w,h], anchor}]}."),
4039+
input_schema=schema({"marks": {"type": "array",
4040+
"items": {"type": "object"}},
4041+
"label_width": {"type": "integer"},
4042+
"label_height": {"type": "integer"},
4043+
"bounds": {"type": "array",
4044+
"items": {"type": "integer"}}},
4045+
required=["marks"]),
4046+
handler=h.place_labels,
4047+
annotations=READ_ONLY,
4048+
),
4049+
MCPTool(
4050+
name="ac_label_color",
4051+
description=("The higher-contrast label text colour (black or "
4052+
"white) for a 'background' [r,g,b], by WCAG contrast. "
4053+
"Returns {rgb, contrast}."),
4054+
input_schema=schema({"background": {"type": "array",
4055+
"items": {"type": "integer"}}},
4056+
required=["background"]),
4057+
handler=h.label_color,
4058+
annotations=READ_ONLY,
4059+
),
40334060
]
40344061

40354062

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -744,6 +744,16 @@ def colors_collide(left, right, kind="deuteranopia", severity=1.0,
744744
return _colors_collide(left, right, kind, severity, threshold)
745745

746746

747+
def place_labels(marks, label_width=22, label_height=16, bounds=None):
748+
from je_auto_control.utils.executor.action_executor import _place_labels
749+
return _place_labels(marks, label_width, label_height, bounds)
750+
751+
752+
def label_color(background):
753+
from je_auto_control.utils.executor.action_executor import _label_color
754+
return _label_color(background)
755+
756+
747757
def normalize_ext(target):
748758
from je_auto_control.utils.executor.action_executor import _normalize_ext
749759
return _normalize_ext(target)

0 commit comments

Comments
 (0)