Skip to content

Commit bfed3bc

Browse files
JE-ChenJE-Chen
authored
Merge pull request #378 from Integration-Automation/feat/form-fields-batch
Add form_fields: multi-direction label/value association + checkbox state
2 parents a878e12 + 51515b4 commit bfed3bc

15 files changed

Lines changed: 404 additions & 0 deletions

File tree

‎README/WHATS_NEW_zh-CN.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-06-24) — 表单字段关联(多方向)+ 复选框状态
4+
5+
即使值在下方或右对齐也能把标签与值配对,并读取复选框状态。完整参考:[`docs/source/Zh/doc/new_features/v166_features_doc.rst`](../docs/source/Zh/doc/new_features/v166_features_doc.rst)。
6+
7+
- **`associate_fields` / `match_labels_to_widgets` / `checkbox_state`**(`AC_associate_fields`、`AC_match_labels_to_widgets`):`ocr/structure` 只把 `label:` 与*紧接的下一格*配对——无法处理标签在上、双列 key/value、右对齐值或非文字 widget,且无复选框概念。本功能把每个标签与多*方向*(右 / 下)中 `max_gap` 内最近的对齐值配对,把独立 widget(复选框 / 单选钮 / 输入框)配到最近标签,并由框内暗像素填充比例读取复选框状态。关联部分纯标准库;只有 `checkbox_state` 触及像素(隔离在 `visual_match` 灰阶加载器之后)。不导入 `PySide6`。
8+
39
## 本次更新 (2026-06-24) — 留白投影列推断(无框线表格)
410

511
靠留白间隙推断列来读取无框线表格。完整参考:[`docs/source/Zh/doc/new_features/v165_features_doc.rst`](../docs/source/Zh/doc/new_features/v165_features_doc.rst)。

‎README/WHATS_NEW_zh-TW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# 本次更新 — AutoControl
22

3+
## 本次更新 (2026-06-24) — 表單欄位關聯(多方向)+ 核取方塊狀態
4+
5+
即使值在下方或右對齊也能把標籤與值配對,並讀取核取方塊狀態。完整參考:[`docs/source/Zh/doc/new_features/v166_features_doc.rst`](../docs/source/Zh/doc/new_features/v166_features_doc.rst)。
6+
7+
- **`associate_fields` / `match_labels_to_widgets` / `checkbox_state`**(`AC_associate_fields`、`AC_match_labels_to_widgets`):`ocr/structure` 只把 `label:` 與*緊接的下一格*配對——無法處理標籤在上、雙欄 key/value、右對齊值或非文字 widget,且無核取方塊概念。本功能把每個標籤與多*方向*(右 / 下)中 `max_gap` 內最近的對齊值配對,把獨立 widget(核取方塊 / 單選鈕 / 輸入框)配到最近標籤,並由框內暗像素填充比例讀取核取方塊狀態。關聯部分純標準函式庫;只有 `checkbox_state` 觸及像素(隔離在 `visual_match` 灰階載入器之後)。不匯入 `PySide6`。
8+
39
## 本次更新 (2026-06-24) — 留白投影欄位偵測(無框線表格)
410

511
靠留白間隙推導欄位來讀取無框線表格。完整參考:[`docs/source/Zh/doc/new_features/v165_features_doc.rst`](../docs/source/Zh/doc/new_features/v165_features_doc.rst)。

‎WHATS_NEW.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-24) — Form Field Association (Multi-Direction) + Checkbox State
4+
5+
Pair form labels with values even when the value is below or right-aligned, and read checkbox state. Full reference: [`docs/source/Eng/doc/new_features/v166_features_doc.rst`](docs/source/Eng/doc/new_features/v166_features_doc.rst).
6+
7+
- **`associate_fields` / `match_labels_to_widgets` / `checkbox_state`** (`AC_associate_fields`, `AC_match_labels_to_widgets`): `ocr/structure` only pairs a `label:` with the *immediately next* cell — it can't handle label-above-value, two-column key/value, right-aligned values, or non-text widgets, and has no checkbox notion. This pairs each label with the nearest aligned value across *directions* (right / below) within `max_gap`, matches free-standing widgets (checkbox/radio/input) to their nearest label, and reads checkbox state from the box's dark-pixel fill ratio. Association is pure-stdlib; only `checkbox_state` touches pixels (behind the `visual_match` gray loader). No `PySide6`.
8+
39
## What's new (2026-06-24) — Whitespace-Projection Columns (Borderless Tables)
410

511
Read borderless tables by inferring columns from the whitespace gaps. Full reference: [`docs/source/Eng/doc/new_features/v165_features_doc.rst`](docs/source/Eng/doc/new_features/v165_features_doc.rst).

‎docs/source/Eng/doc/new_features/v166_features_doc.rst‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Form Field Association (Multi-Direction) + Checkbox State
2+
=========================================================
3+
4+
``ocr/structure`` recognises a label only if its text ends in ``:`` and pairs it with the
5+
*immediately next* cell — it cannot handle a label sitting *above* its value, a two-column
6+
key/value layout, right-aligned values, or any widget that isn't a text cell, and it has no
7+
notion of checkbox / radio state at all. ``form_fields`` generalises this: it pairs each
8+
label with the nearest aligned value in any of several *directions* (right, below), matches
9+
free-standing widgets (checkboxes, radios, inputs) to their nearest label, and reads a
10+
checkbox's checked state from its fill ratio.
11+
12+
The association is pure-stdlib over plain box dicts (fully unit-testable, no image); only
13+
``checkbox_state`` touches pixels, isolated behind the shared ``visual_match`` gray loader so
14+
tests can pass a synthetic array. Reuses ``table_grid_fill``'s box-bounds reader. Imports no
15+
``PySide6``.
16+
17+
Headless API
18+
------------
19+
20+
.. code-block:: python
21+
22+
from je_auto_control import (associate_fields, match_labels_to_widgets,
23+
checkbox_state)
24+
25+
fields = associate_fields(ocr_boxes, directions=("right", "below"))
26+
# [{"label": "Name", "value": "Ann", "direction": "right", "gap": 20, ...}]
27+
28+
pairs = match_labels_to_widgets(label_boxes, checkbox_boxes)
29+
# [{"widget": {...}, "label": "Accept terms", "distance": 22}]
30+
31+
state = checkbox_state(screenshot, checkbox_box) # "checked" | "unchecked"
32+
33+
``associate_fields`` treats boxes whose text ends in ``:`` as labels and pairs each with the
34+
nearest value box in the requested directions (within ``max_gap``), returning
35+
``{label, value, direction, gap, label_box, value_box}``. ``match_labels_to_widgets`` assigns
36+
each widget to its nearest label by centre distance. ``checkbox_state`` returns
37+
``"checked"`` / ``"unchecked"`` from the dark-pixel fill ratio of the box (image is injectable
38+
— path / ndarray / PIL).
39+
40+
Executor commands
41+
-----------------
42+
43+
``AC_associate_fields`` (``text_boxes`` / ``directions`` / ``max_gap`` → ``{count, fields}``)
44+
and ``AC_match_labels_to_widgets`` (``labels`` / ``widgets`` → ``{count, pairs}``). They are
45+
exposed as the MCP tools ``ac_associate_fields`` / ``ac_match_labels_to_widgets`` (read-only)
46+
and as the Script Builder commands **Associate Form Fields** / **Match Labels To Widgets**
47+
under **OCR**.

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,7 @@ Comprehensive guides for all AutoControl features.
188188
doc/new_features/v163_features_doc
189189
doc/new_features/v164_features_doc
190190
doc/new_features/v165_features_doc
191+
doc/new_features/v166_features_doc
191192
doc/ocr_backends/ocr_backends_doc
192193
doc/observability/observability_doc
193194
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v166_features_doc.rst‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
表單欄位關聯(多方向)+ 核取方塊狀態
2+
======================================
3+
4+
``ocr/structure`` 只有在框文字以 ``:`` 結尾時才視為標籤,並只與*緊接的下一格*配對——無法
5+
處理標籤在值*上方*、雙欄 key/value 版面、右對齊值,或任何非文字格的 widget,且完全沒有
6+
核取方塊 / 單選鈕狀態的概念。``form_fields`` 將其一般化:把每個標籤與多個*方向*(右、下)中
7+
最近的對齊值配對,把獨立的 widget(核取方塊、單選鈕、輸入框)配到最近的標籤,並由填充比例
8+
讀取核取方塊的勾選狀態。
9+
10+
關聯部分為純標準函式庫,作用於純框字典(可完整單元測試、不需影像);只有 ``checkbox_state``
11+
觸及像素,且隔離在共用的 ``visual_match`` 灰階載入器之後,讓測試可傳入合成陣列。重用
12+
``table_grid_fill`` 的框邊界讀取器。不匯入 ``PySide6``。
13+
14+
無頭 API
15+
--------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import (associate_fields, match_labels_to_widgets,
20+
checkbox_state)
21+
22+
fields = associate_fields(ocr_boxes, directions=("right", "below"))
23+
# [{"label": "Name", "value": "Ann", "direction": "right", "gap": 20, ...}]
24+
25+
pairs = match_labels_to_widgets(label_boxes, checkbox_boxes)
26+
# [{"widget": {...}, "label": "Accept terms", "distance": 22}]
27+
28+
state = checkbox_state(screenshot, checkbox_box) # "checked" | "unchecked"
29+
30+
``associate_fields`` 把文字以 ``:`` 結尾的框視為標籤,並在指定方向中(``max_gap`` 內)配到
31+
最近的值框,回傳 ``{label, value, direction, gap, label_box, value_box}``。
32+
``match_labels_to_widgets`` 依中心距離把每個 widget 配到最近標籤。``checkbox_state`` 由框內
33+
暗像素填充比例回傳 ``"checked"`` / ``"unchecked"``(image 可注入——路徑 / ndarray / PIL)。
34+
35+
執行器指令
36+
----------
37+
38+
``AC_associate_fields``(``text_boxes`` / ``directions`` / ``max_gap`` → ``{count, fields}``)
39+
與 ``AC_match_labels_to_widgets``(``labels`` / ``widgets`` → ``{count, pairs}``)。兩者以
40+
MCP 工具 ``ac_associate_fields`` / ``ac_match_labels_to_widgets``(唯讀)及 Script Builder 指令
41+
**Associate Form Fields** / **Match Labels To Widgets**(位於 **OCR** 分類下)形式提供。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,7 @@ AutoControl 所有功能的完整使用指南。
188188
doc/new_features/v163_features_doc
189189
doc/new_features/v164_features_doc
190190
doc/new_features/v165_features_doc
191+
doc/new_features/v166_features_doc
191192
doc/ocr_backends/ocr_backends_doc
192193
doc/observability/observability_doc
193194
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -307,6 +307,10 @@
307307
from je_auto_control.utils.column_layout import (
308308
assign_columns, column_gutters, detect_borderless_table, vertical_projection,
309309
)
310+
# Associate form labels with values (multi-direction) + checkbox state
311+
from je_auto_control.utils.form_fields import (
312+
associate_fields, checkbox_state, match_labels_to_widgets,
313+
)
310314
# Locate on-screen regions by colour (mask + connected components)
311315
from je_auto_control.utils.color_region import (
312316
find_color_region, find_color_regions,
@@ -1238,6 +1242,9 @@ def start_autocontrol_gui(*args, **kwargs):
12381242
"column_gutters",
12391243
"assign_columns",
12401244
"detect_borderless_table",
1245+
"associate_fields",
1246+
"match_labels_to_widgets",
1247+
"checkbox_state",
12411248
"find_color_region",
12421249
"find_color_regions",
12431250
"ssim_compare",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -729,6 +729,29 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None:
729729
),
730730
description="Find borderless-table column separators by whitespace projection.",
731731
))
732+
specs.append(CommandSpec(
733+
"AC_associate_fields", "OCR", "Associate Form Fields",
734+
fields=(
735+
FieldSpec("text_boxes", FieldType.STRING,
736+
placeholder='[{"x":0,"y":0,"width":60,"height":20,'
737+
'"text":"Name:"}]'),
738+
FieldSpec("directions", FieldType.STRING, optional=True,
739+
placeholder='["right", "below"]'),
740+
FieldSpec("max_gap", FieldType.INT, optional=True, default=150),
741+
),
742+
description="Pair 'label:' boxes with the nearest aligned value (right/below).",
743+
))
744+
specs.append(CommandSpec(
745+
"AC_match_labels_to_widgets", "OCR", "Match Labels To Widgets",
746+
fields=(
747+
FieldSpec("labels", FieldType.STRING,
748+
placeholder='[{"x":0,"y":0,"width":60,"height":20,'
749+
'"text":"Accept"}]'),
750+
FieldSpec("widgets", FieldType.STRING,
751+
placeholder='[{"x":120,"y":0,"width":16,"height":16}]'),
752+
),
753+
description="Match each checkbox/radio/input to its nearest label by centre.",
754+
))
732755
specs.append(CommandSpec(
733756
"AC_scroll_to_find", "OCR", "Scroll Until Visible",
734757
fields=(

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3433,6 +3433,33 @@ def _detect_borderless_table(boxes: Any, page_width: Any = None, min_gap: Any =
34333433
return {"found": table is not None, "table": table}
34343434

34353435

3436+
def _associate_fields(text_boxes: Any, directions: Any = None,
3437+
max_gap: Any = 150) -> Dict[str, Any]:
3438+
"""Adapter: pair form labels with their nearest aligned value boxes."""
3439+
import json
3440+
from je_auto_control.utils.form_fields import associate_fields
3441+
if isinstance(text_boxes, str):
3442+
text_boxes = json.loads(text_boxes)
3443+
if isinstance(directions, str):
3444+
directions = json.loads(directions) if directions.strip() else None
3445+
fields = associate_fields(text_boxes,
3446+
directions=tuple(directions) if directions
3447+
else ("right", "below"), max_gap=int(max_gap))
3448+
return {"count": len(fields), "fields": fields}
3449+
3450+
3451+
def _match_labels_to_widgets(labels: Any, widgets: Any) -> Dict[str, Any]:
3452+
"""Adapter: match each widget (checkbox / radio / input) to its nearest label."""
3453+
import json
3454+
from je_auto_control.utils.form_fields import match_labels_to_widgets
3455+
if isinstance(labels, str):
3456+
labels = json.loads(labels)
3457+
if isinstance(widgets, str):
3458+
widgets = json.loads(widgets)
3459+
pairs = match_labels_to_widgets(labels, widgets)
3460+
return {"count": len(pairs), "pairs": pairs}
3461+
3462+
34363463
def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50,
34373464
region: Any = None) -> Dict[str, Any]:
34383465
"""Adapter: locate coloured regions on the screen, largest first."""
@@ -5886,6 +5913,8 @@ def __init__(self):
58865913
"AC_populate_table": _populate_table,
58875914
"AC_column_gutters": _column_gutters,
58885915
"AC_detect_borderless_table": _detect_borderless_table,
5916+
"AC_associate_fields": _associate_fields,
5917+
"AC_match_labels_to_widgets": _match_labels_to_widgets,
58895918
"AC_ssim_compare": _ssim_compare,
58905919
"AC_ssim_changed_regions": _ssim_changed_regions,
58915920
"AC_feature_match": _feature_match,

0 commit comments

Comments
 (0)