Add table_grid_fill: fill a ruling-line grid with OCR text

JE-Chen · JE-Chen · commit 4779bf37a409 · 2026-06-24T02:50:02.000+08:00
edge_lines.find_grid recovers a bordered table's geometry but leaves the cells
empty; OCR gives the text but no structure, and nothing joined them. Drop OCR
boxes into the grid (assigned by cell-centre, gated by an overlap fraction),
concatenate each cell's text in reading order, flag merged-cell spans, and
convert to records / CSV. Pure-stdlib over plain dicts.
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 以 OCR 文字填入框线网格(可定址表格)
+
+把有框线表格的线条 + OCR 文字转成可定址的 `R x C` 表格。完整参考:[`docs/source/Zh/doc/new_features/v162_features_doc.rst`](../docs/source/Zh/doc/new_features/v162_features_doc.rst)。
+
+- **`populate_table` / `assign_text_to_grid` / `table_to_records` / `table_to_csv`**(`AC_populate_table`):`edge_lines.find_grid` 能还原表格的框线几何但返回的单元格是*空的*;OCR 提供文字却无结构——两者从未串接。本功能把 OCR 框放入网格(依单元格中心指派,以重叠比例把关,使横跨细框线的框不被重复计入),将每个单元格的文字依阅读顺序串接,标记合并单元格的 span,并可直接转成 records / CSV。纯标准库,作用于纯字典——不需图像、OCR 引擎或设备。不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 信任评分模板匹配(歧义 / PSR)
 
 在点击前就知道某次模板匹配虽强但*有歧义*。完整参考:[`docs/source/Zh/doc/new_features/v161_features_doc.rst`](../docs/source/Zh/doc/new_features/v161_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 以 OCR 文字填入框線網格(可定址表格)
+
+把有框線表格的線條 + OCR 文字轉成可定址的 `R x C` 表格。完整參考:[`docs/source/Zh/doc/new_features/v162_features_doc.rst`](../docs/source/Zh/doc/new_features/v162_features_doc.rst)。
+
+- **`populate_table` / `assign_text_to_grid` / `table_to_records` / `table_to_csv`**(`AC_populate_table`):`edge_lines.find_grid` 能還原表格的框線幾何但回傳的儲存格是*空的*;OCR 提供文字卻無結構——兩者從未串接。本功能把 OCR 框放入網格(依儲存格中心指派,以重疊比例把關,使橫跨細框線的框不被重複計入),將每個儲存格的文字依閱讀順序串接,標記合併儲存格的 span,並可直接轉成 records / CSV。純標準函式庫,作用於純字典——不需影像、OCR 引擎或裝置。不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 信任評分樣板比對(歧義 / PSR)
 
 在點擊前就知道某次樣板比對雖強但*有歧義*。完整參考:[`docs/source/Zh/doc/new_features/v161_features_doc.rst`](../docs/source/Zh/doc/new_features/v161_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Fill a Ruling-Line Grid With OCR Text (Addressable Tables)
+
+Turn a bordered table's lines + OCR words into an addressable `R x C` table. Full reference: [`docs/source/Eng/doc/new_features/v162_features_doc.rst`](docs/source/Eng/doc/new_features/v162_features_doc.rst).
+
+- **`populate_table` / `assign_text_to_grid` / `table_to_records` / `table_to_csv`** (`AC_populate_table`): `edge_lines.find_grid` recovers a table's ruling-line geometry but the cells come back *empty*; OCR gives the text but no structure — nothing joined them. This drops OCR boxes into the grid (assigned by cell-centre, gated by an overlap fraction so a box straddling a thin rule isn't double-counted), concatenates each cell's text in reading order, flags merged-cell spans, and converts straight to records / CSV. Pure-stdlib over plain dicts — no image, no OCR engine, no device. No `PySide6`.
+
 ## What's new (2026-06-24) — Trust-Scored Template Matching (Ambiguity / PSR)
 
 Know when a template match is strong but *ambiguous* before clicking it. Full reference: [`docs/source/Eng/doc/new_features/v161_features_doc.rst`](docs/source/Eng/doc/new_features/v161_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v162_features_doc.rst b/docs/source/Eng/doc/new_features/v162_features_doc.rst
@@ -0,0 +1,46 @@
+Fill a Ruling-Line Grid With OCR Text (Addressable Tables)
+==========================================================
+
+``edge_lines.find_grid`` recovers a bordered table's geometry — ``{rows: [y…], cols: [x…],
+cells: […]}`` — but the cells come back *empty* (just rectangles between the ruling lines).
+OCR gives the text but no table structure. Nothing joined the two, so reading a bordered
+table meant hand-rolling the box→cell assignment. ``table_grid_fill`` drops OCR text boxes
+into the grid and returns an addressable ``R x C`` table.
+
+Each box is assigned to the cell its centre falls in (gated by an overlap fraction so a box
+straddling a thin rule is not double counted); text within a cell is concatenated in reading
+order; boxes that straddle multiple cells are reported as merged-cell candidates. The result
+converts straight to records or CSV.
+
+Pure-stdlib geometry over plain dicts (the grid + the boxes) — no image, no OCR engine, no
+device. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (find_grid, find_text_lines,  # producers
+                                 populate_table, assign_text_to_grid,
+                                 table_to_records, table_to_csv)
+
+    grid = find_grid(region=[0, 0, 800, 400])      # ruling-line geometry
+    boxes = [{"x": 10, "y": 5, "width": 60, "height": 20, "text": "Name"}, ...]
+
+    table = assign_text_to_grid(grid, boxes)        # [["Name","Age"], ["Ann","30"]]
+    records = table_to_records(table)               # [{"Name": "Ann", "Age": "30"}]
+    csv_text = table_to_csv(table)
+
+    full = populate_table(grid, boxes)              # {n_rows, n_cols, cells, spans}
+
+``assign_text_to_grid`` returns the 2-D text table; ``populate_table`` returns the richer
+``{n_rows, n_cols, cells:[{row, col, text}], spans:[{row, col, row_span, col_span, text}]}``.
+``table_to_records`` uses the first row as headers; ``table_to_csv`` renders CSV. Boxes accept
+either ``{x, y, width, height}`` or ``{left, top, right, bottom}`` plus a ``text`` field.
+
+Executor command
+----------------
+
+``AC_populate_table`` (``grid`` / ``text_boxes`` / ``overlap`` → ``{n_rows, n_cols, cells,
+spans}``) is exposed as the MCP tool ``ac_populate_table`` (read-only) and as the Script
+Builder command **Fill Table From Grid + OCR** under **OCR**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -184,6 +184,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v159_features_doc
    doc/new_features/v160_features_doc
    doc/new_features/v161_features_doc
+   doc/new_features/v162_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v162_features_doc.rst b/docs/source/Zh/doc/new_features/v162_features_doc.rst
@@ -0,0 +1,44 @@
+以 OCR 文字填入框線網格(可定址表格)
+======================================
+
+``edge_lines.find_grid`` 能還原有框線表格的幾何——``{rows: [y…], cols: [x…], cells: […]}``
+——但回傳的儲存格是*空的*(僅是框線之間的矩形)。OCR 提供文字卻無表格結構。兩者從未串接,
+因此讀取有框線表格只能自行撰寫 box→cell 的指派。``table_grid_fill`` 把 OCR 文字框放入網格,
+回傳可定址的 ``R x C`` 表格。
+
+每個框依其中心落在哪個儲存格而被指派(以重疊比例把關,使橫跨細框線的框不被重複計入);
+同一儲存格內的文字依閱讀順序串接;橫跨多個儲存格的框則回報為合併儲存格候選。結果可直接
+轉成 records 或 CSV。
+
+純標準函式庫幾何,作用於純字典(網格 + 框)——不需影像、不需 OCR 引擎、不需裝置。不匯入
+``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (find_grid, find_text_lines,  # 產生來源
+                                 populate_table, assign_text_to_grid,
+                                 table_to_records, table_to_csv)
+
+    grid = find_grid(region=[0, 0, 800, 400])      # 框線幾何
+    boxes = [{"x": 10, "y": 5, "width": 60, "height": 20, "text": "Name"}, ...]
+
+    table = assign_text_to_grid(grid, boxes)        # [["Name","Age"], ["Ann","30"]]
+    records = table_to_records(table)               # [{"Name": "Ann", "Age": "30"}]
+    csv_text = table_to_csv(table)
+
+    full = populate_table(grid, boxes)              # {n_rows, n_cols, cells, spans}
+
+``assign_text_to_grid`` 回傳二維文字表格;``populate_table`` 回傳更豐富的
+``{n_rows, n_cols, cells:[{row, col, text}], spans:[{row, col, row_span, col_span, text}]}``。
+``table_to_records`` 以第一列為標頭;``table_to_csv`` 輸出 CSV。框接受 ``{x, y, width, height}``
+或 ``{left, top, right, bottom}`` 加上 ``text`` 欄位。
+
+執行器指令
+----------
+
+``AC_populate_table``(``grid`` / ``text_boxes`` / ``overlap`` → ``{n_rows, n_cols, cells,
+spans}``)以 MCP 工具 ``ac_populate_table``(唯讀)及 Script Builder 指令
+**Fill Table From Grid + OCR**(位於 **OCR** 分類下)形式提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -184,6 +184,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v159_features_doc
    doc/new_features/v160_features_doc
    doc/new_features/v161_features_doc
+   doc/new_features/v162_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -291,6 +291,10 @@
 from je_auto_control.utils.screen_grid import (
     GridCell, cell_for_point, grid_cells, point_for_cell,
 )
+# Fill a ruling-line grid with OCR text → addressable table
+from je_auto_control.utils.table_grid_fill import (
+    assign_text_to_grid, populate_table, table_to_csv, table_to_records,
+)
 # Locate on-screen regions by colour (mask + connected components)
 from je_auto_control.utils.color_region import (
     find_color_region, find_color_regions,
@@ -1209,6 +1213,10 @@ def start_autocontrol_gui(*args, **kwargs):
     "grid_cells",
     "cell_for_point",
     "point_for_cell",
+    "assign_text_to_grid",
+    "populate_table",
+    "table_to_records",
+    "table_to_csv",
     "find_color_region",
     "find_color_regions",
     "ssim_compare",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -674,6 +674,19 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Decode 1-D barcodes (EAN / UPC) in an image / screen region.",
     ))
+    specs.append(CommandSpec(
+        "AC_populate_table", "OCR", "Fill Table From Grid + OCR",
+        fields=(
+            FieldSpec("grid", FieldType.STRING,
+                      placeholder='{"rows": [0, 30, 60], "cols": [0, 100, 200]}'),
+            FieldSpec("text_boxes", FieldType.STRING,
+                      placeholder='[{"x": 10, "y": 5, "width": 60, "height": 20, '
+                                  '"text": "Name"}]'),
+            FieldSpec("overlap", FieldType.FLOAT, optional=True, default=0.4,
+                      min_value=0.0, max_value=1.0),
+        ),
+        description="Drop OCR text boxes into a ruling-line grid → addressable table.",
+    ))
     specs.append(CommandSpec(
         "AC_scroll_to_find", "OCR", "Scroll Until Visible",
         fields=(
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3373,6 +3373,17 @@ def _point_for_cell(label: str, rows: Any, cols: Any,
     return {"point": point}
 
 
+def _populate_table(grid: Any, text_boxes: Any, overlap: Any = 0.4) -> Dict[str, Any]:
+    """Adapter: fill a ruling-line grid with OCR text boxes → addressable table."""
+    import json
+    from je_auto_control.utils.table_grid_fill import populate_table
+    if isinstance(grid, str):
+        grid = json.loads(grid)
+    if isinstance(text_boxes, str):
+        text_boxes = json.loads(text_boxes)
+    return populate_table(grid, text_boxes, overlap=float(overlap))
+
+
 def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50,
                        region: Any = None) -> Dict[str, Any]:
     """Adapter: locate coloured regions on the screen, largest first."""
@@ -5799,6 +5810,7 @@ def __init__(self):
             "AC_grid_cells": _grid_cells,
             "AC_cell_for_point": _cell_for_point,
             "AC_point_for_cell": _point_for_cell,
+            "AC_populate_table": _populate_table,
             "AC_ssim_compare": _ssim_compare,
             "AC_ssim_changed_regions": _ssim_changed_regions,
             "AC_feature_match": _feature_match,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3665,6 +3665,21 @@ def screen_grid_tools() -> List[MCPTool]:
             handler=h.point_for_cell,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_populate_table",
+            description=("Fill a ruling-line 'grid' (from find_grid: {rows:[y..],"
+                         "cols:[x..]}) with OCR 'text_boxes' ([{x,y,width,height,"
+                         "text}]) into an addressable table. Returns {n_rows, n_cols, "
+                         "cells:[{row,col,text}], spans:[merged-cell candidates]}. "
+                         "'overlap' (default 0.4) gates a box straddling a rule."),
+            input_schema=schema({
+                "grid": {"type": "object"},
+                "text_boxes": {"type": "array", "items": {"type": "object"}},
+                "overlap": {"type": "number"}},
+                required=["grid", "text_boxes"]),
+            handler=h.populate_table,
+            annotations=READ_ONLY,
+        ),
     ]
 
 
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2130,6 +2130,11 @@ def point_for_cell(label, rows, cols, region=None):
     return _point_for_cell(label, rows, cols, region)
 
 
+def populate_table(grid, text_boxes, overlap=0.4):
+    from je_auto_control.utils.executor.action_executor import _populate_table
+    return _populate_table(grid, text_boxes, overlap)
+
+
 def find_color_region(rgb, tolerance=20, min_area=50, region=None):
     from je_auto_control.utils.executor.action_executor import (
         _find_color_region)
diff --git a/je_auto_control/utils/table_grid_fill/__init__.py b/je_auto_control/utils/table_grid_fill/__init__.py
@@ -0,0 +1,9 @@
+"""Fill a ruling-line grid with OCR text to get an addressable table."""
+from je_auto_control.utils.table_grid_fill.table_grid_fill import (
+    assign_text_to_grid, populate_table, table_to_csv, table_to_records,
+)
+
+__all__ = [
+    "assign_text_to_grid", "populate_table",
+    "table_to_records", "table_to_csv",
+]
diff --git a/je_auto_control/utils/table_grid_fill/table_grid_fill.py b/je_auto_control/utils/table_grid_fill/table_grid_fill.py
@@ -0,0 +1,132 @@
+"""Fill a ruling-line grid with OCR text to get an addressable table.
+
+``edge_lines.find_grid`` recovers a bordered table's geometry — ``{rows: [y…],
+cols: [x…], cells: […]}`` — but the cells come back *empty* (pure rectangles from the
+ruling lines). ``ocr`` / OCR word boxes give the text but no table structure. Nothing
+joined the two, so reading a bordered table meant hand-rolling the box→cell assignment.
+
+This drops OCR text boxes into the grid: each box is assigned to the cell its centre
+falls in (gated by an overlap fraction so a box straddling a thin rule is not double
+counted), text within a cell is concatenated in reading order, and boxes that span
+multiple cells are reported separately. The result is an ``R x C`` text table that
+converts straight to records / CSV.
+
+Pure-stdlib geometry over plain dicts (the grid + the boxes); fully unit-testable with
+no image, no OCR engine, no device. Imports no ``PySide6``.
+"""
+import csv
+import io
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+Box = Dict[str, Any]
+Bounds = Tuple[int, int, int, int]
+
+
+def _box_bounds(box: Box) -> Bounds:
+    """Return ``(left, top, right, bottom)`` from an ``x/y/w/h`` or ``l/t/r/b`` box."""
+    if "width" in box and "height" in box:
+        left, top = int(box["x"]), int(box["y"])
+        return left, top, left + int(box["width"]), top + int(box["height"])
+    if {"left", "top", "right", "bottom"} <= box.keys():
+        return int(box["left"]), int(box["top"]), int(box["right"]), int(box["bottom"])
+    raise ValueError("box needs x/y/width/height or left/top/right/bottom")
+
+
+def _intervals(edges: Sequence[int]) -> List[Tuple[int, int]]:
+    """Turn sorted edge coordinates into consecutive ``(start, end)`` spans."""
+    ordered = sorted(int(e) for e in edges)
+    return [(ordered[i], ordered[i + 1]) for i in range(len(ordered) - 1)]
+
+
+def _index_of(value: float, spans: Sequence[Tuple[int, int]]) -> Optional[int]:
+    """Return the index of the span containing ``value``, else ``None``."""
+    for i, (start, end) in enumerate(spans):
+        if start <= value < end:
+            return i
+    return None
+
+
+def _overlap_fraction(bounds: Bounds, cell: Tuple[Tuple[int, int], Tuple[int, int]]) -> float:
+    """Intersection area of a box and a cell, divided by the box area."""
+    (left, top, right, bottom) = bounds
+    (cx0, cx1), (cy0, cy1) = cell
+    inter = max(0, min(right, cx1) - max(left, cx0)) * max(0, min(bottom, cy1) - max(top, cy0))
+    area = max(1, (right - left) * (bottom - top))
+    return inter / area
+
+
+def _grid_spans(grid: Dict[str, Any]) -> Tuple[List[Tuple[int, int]], List[Tuple[int, int]]]:
+    """Return ``(column_spans, row_spans)`` from a grid's ``cols`` / ``rows`` edges."""
+    return _intervals(grid.get("cols", [])), _intervals(grid.get("rows", []))
+
+
+def _placed(box: Box, col_spans, row_spans, overlap: float):
+    """Return ``(row, col)`` for a box, or ``None`` if it misses every cell."""
+    left, top, right, bottom = _box_bounds(box)
+    col = _index_of((left + right) / 2, col_spans)
+    row = _index_of((top + bottom) / 2, row_spans)
+    if row is None or col is None:
+        return None
+    cell = (col_spans[col], row_spans[row])
+    if _overlap_fraction((left, top, right, bottom), cell) < overlap:
+        return None
+    return row, col
+
+
+def assign_text_to_grid(grid: Dict[str, Any], text_boxes: Sequence[Box], *,
+                        overlap: float = 0.4) -> List[List[str]]:
+    """Return an ``R x C`` table of cell text from a grid + OCR boxes (reading order)."""
+    col_spans, row_spans = _grid_spans(grid)
+    buckets: Dict[Tuple[int, int], List[Box]] = {}
+    for box in text_boxes:
+        placed = _placed(box, col_spans, row_spans, float(overlap))
+        if placed is not None:
+            buckets.setdefault(placed, []).append(box)
+    table: List[List[str]] = []
+    for row in range(len(row_spans)):
+        cells = []
+        for col in range(len(col_spans)):
+            ordered = sorted(buckets.get((row, col), []), key=_box_bounds)
+            cells.append(" ".join(str(b.get("text", "")) for b in ordered).strip())
+        table.append(cells)
+    return table
+
+
+def _spans(grid: Dict[str, Any], text_boxes: Sequence[Box]) -> List[Dict[str, Any]]:
+    """Return boxes that straddle more than one cell (merged-cell candidates)."""
+    col_spans, row_spans = _grid_spans(grid)
+    found: List[Dict[str, Any]] = []
+    for box in text_boxes:
+        left, top, right, bottom = _box_bounds(box)
+        c0, c1 = _index_of(left, col_spans), _index_of(right - 1, col_spans)
+        r0, r1 = _index_of(top, row_spans), _index_of(bottom - 1, row_spans)
+        if None in (c0, c1, r0, r1) or (c0 == c1 and r0 == r1):
+            continue
+        found.append({"row": r0, "col": c0, "row_span": r1 - r0 + 1,
+                      "col_span": c1 - c0 + 1, "text": str(box.get("text", ""))})
+    return found
+
+
+def populate_table(grid: Dict[str, Any], text_boxes: Sequence[Box], *,
+                   overlap: float = 0.4) -> Dict[str, Any]:
+    """Fill ``grid`` with ``text_boxes`` → ``{n_rows, n_cols, cells, spans}``."""
+    table = assign_text_to_grid(grid, text_boxes, overlap=overlap)
+    cells = [{"row": r, "col": c, "text": table[r][c]}
+             for r in range(len(table)) for c in range(len(table[r]))]
+    return {"n_rows": len(table), "n_cols": len(table[0]) if table else 0,
+            "cells": cells, "spans": _spans(grid, text_boxes)}
+
+
+def table_to_records(rows: Sequence[Sequence[str]]) -> List[Dict[str, str]]:
+    """Use the first row as headers; return the remaining rows as dicts."""
+    if not rows:
+        return []
+    header = list(rows[0])
+    return [dict(zip(header, row)) for row in rows[1:]]
+
+
+def table_to_csv(rows: Sequence[Sequence[str]]) -> str:
+    """Render a 2-D text table as a CSV string."""
+    buffer = io.StringIO()
+    csv.writer(buffer).writerows(rows)
+    return buffer.getvalue()
diff --git a/test/unit_test/headless/test_table_grid_fill_batch.py b/test/unit_test/headless/test_table_grid_fill_batch.py
