Merge pull request #346 from Integration-Automation/feat/monitor-layout-batch

JE-Chen · web-flow · commit e9285f72adea · 2026-06-23T06:27:04.000+08:00
Add multi-monitor / virtual-desktop geometry
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 多显示器 / 虚拟桌面几何
+
+在多台显示器间正确摆放窗口与坐标。完整参考:[`docs/source/Zh/doc/new_features/v136_features_doc.rst`](../docs/source/Zh/doc/new_features/v136_features_doc.rst)。
+
+- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`**(`AC_enumerate_monitors`、`AC_monitor_at_point`):`snap_window` / `arrange_grid` / 版面规划器都假设单一主屏 `(width, height)`——对多显示器无感,无法在第二台显示器铺排或处理负原点虚拟桌面。本功能补上实体层:并集虚拟边界、某点 / 某窗口属于哪台显示器、虚拟↔显示器区域坐标转换,以及跨分辨率 / DPI 的等效位置重映射。对 `Monitor` dataclass 的纯几何 → 完全无头可测;`enumerate_monitors` 具可注入 provider(默认 `mss`)。
+
 ## 本次更新 (2026-06-23) — 图像预处理(供 OCR / 模板匹配)
 
 在识别或匹配前先清理画面。完整参考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 多螢幕 / 虛擬桌面幾何
+
+在多台顯示器間正確擺放視窗與座標。完整參考:[`docs/source/Zh/doc/new_features/v136_features_doc.rst`](../docs/source/Zh/doc/new_features/v136_features_doc.rst)。
+
+- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`**(`AC_enumerate_monitors`、`AC_monitor_at_point`):`snap_window` / `arrange_grid` / 版面規劃器都假設單一主螢幕 `(width, height)`——對多螢幕無感,無法在第二台顯示器鋪排或處理負原點虛擬桌面。本功能補上實體層:聯集虛擬邊界、某點 / 某視窗屬於哪台螢幕、虛擬↔螢幕區域座標轉換,以及跨解析度 / DPI 的等效位置重映射。對 `Monitor` dataclass 的純幾何 → 完全無頭可測;`enumerate_monitors` 具可注入 provider(預設 `mss`)。
+
 ## 本次更新 (2026-06-23) — 影像前處理(供 OCR / 模板比對)
 
 在辨識或比對前先清理畫面。完整參考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Multi-Monitor / Virtual-Desktop Geometry
+
+Place windows and points correctly across several displays. Full reference: [`docs/source/Eng/doc/new_features/v136_features_doc.rst`](docs/source/Eng/doc/new_features/v136_features_doc.rst).
+
+- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`** (`AC_enumerate_monitors`, `AC_monitor_at_point`): `snap_window` / `arrange_grid` / the layout planner all assumed a single primary `(width, height)` — monitor-blind, unable to tile on a second display or handle a negative-origin virtual desktop. This adds the physical layer: union virtual bounds, which-monitor-owns-this-point/window, virtual↔monitor-local conversion, and equivalent-spot remapping across resolutions/DPI. Pure geometry over `Monitor` dataclasses → fully headless-testable; `enumerate_monitors` has an injectable provider (default `mss`).
+
 ## What's new (2026-06-23) — Image Pre-processing for OCR / Template Matching
 
 Clean up the screen before reading or matching it. Full reference: [`docs/source/Eng/doc/new_features/v135_features_doc.rst`](docs/source/Eng/doc/new_features/v135_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v136_features_doc.rst b/docs/source/Eng/doc/new_features/v136_features_doc.rst
@@ -0,0 +1,46 @@
+Multi-Monitor / Virtual-Desktop Geometry
+========================================
+
+``snap_window``, ``arrange_grid`` and the layout planner all take a single primary
+``(width, height)`` — they are monitor-blind: they cannot tile on the second display
+or cope with a negative-origin virtual desktop, and ``coordinate_space`` only rescales
+a model grid. This adds the missing physical layer: enumerate the monitors, compute
+the union virtual bounds, ask which monitor contains a point or a window, convert
+between virtual and per-monitor-local coordinates, and remap a point to the equivalent
+spot on another display.
+
+The geometry is pure arithmetic over plain ``Monitor`` dataclasses, so it is fully
+unit-testable; only ``enumerate_monitors``' default provider touches the OS (via
+``mss``) and it is injectable. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (enumerate_monitors, monitor_at_point,
+                                 virtual_bounds, to_local, remap_point)
+
+    monitors = enumerate_monitors()
+    print(virtual_bounds(monitors))            # (x, y, w, h) spanning all displays
+
+    here = monitor_at_point(monitors, x, y)    # which monitor owns this point
+    idx, lx, ly = to_local(monitors, x, y)     # virtual -> (monitor, local x, local y)
+
+    # Move a point to the equivalent relative spot on another monitor.
+    second = remap_point(monitors[0], monitors[1], 960, 540)
+
+``Monitor`` carries ``index, x, y, width, height, scale, primary`` and a ``work``
+area (``.bounds`` / ``.contains(x, y)`` / ``.to_dict()``). ``virtual_bounds`` returns
+the union box (origin may be negative); ``primary_monitor`` picks the primary;
+``monitor_for_window(rect, monitors)`` returns the display a window mostly occupies
+(max overlap); ``to_virtual`` is the inverse of ``to_local``; ``remap_point``
+preserves the fractional position so it works across differing resolutions and DPI.
+
+Executor commands
+-----------------
+
+``AC_enumerate_monitors`` → ``{count, monitors, virtual_bounds}`` and
+``AC_monitor_at_point`` (``x`` / ``y``) → ``{found, monitor}``. They are exposed as
+the MCP tools ``ac_enumerate_monitors`` / ``ac_monitor_at_point`` and as Script
+Builder commands under **Window**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -158,6 +158,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v133_features_doc
    doc/new_features/v134_features_doc
    doc/new_features/v135_features_doc
+   doc/new_features/v136_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v136_features_doc.rst b/docs/source/Zh/doc/new_features/v136_features_doc.rst
@@ -0,0 +1,39 @@
+多螢幕 / 虛擬桌面幾何
+======================
+
+``snap_window``、``arrange_grid`` 與版面規劃器都只取單一主螢幕 ``(width, height)``——它們對多螢幕無感:
+無法在第二台顯示器上鋪排、也無法處理負原點的虛擬桌面,而 ``coordinate_space`` 只縮放模型網格。本功能補上缺少
+的實體層:列舉各螢幕、計算聯集虛擬邊界、查詢某點或某視窗位於哪台螢幕、在虛擬座標與各螢幕區域座標間轉換,
+並把某點重映射到另一台螢幕上的等效位置。
+
+幾何運算皆是對純 ``Monitor`` dataclass 的算術,因此完全可單元測試;只有 ``enumerate_monitors`` 的預設 provider
+會碰到 OS(透過 ``mss``),且可注入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (enumerate_monitors, monitor_at_point,
+                                 virtual_bounds, to_local, remap_point)
+
+    monitors = enumerate_monitors()
+    print(virtual_bounds(monitors))            # 涵蓋所有顯示器的 (x, y, w, h)
+
+    here = monitor_at_point(monitors, x, y)    # 此點屬於哪台螢幕
+    idx, lx, ly = to_local(monitors, x, y)     # 虛擬 -> (螢幕, 區域 x, 區域 y)
+
+    # 把某點移到另一台螢幕上的等效相對位置。
+    second = remap_point(monitors[0], monitors[1], 960, 540)
+
+``Monitor`` 帶有 ``index, x, y, width, height, scale, primary`` 與 ``work`` 區域(``.bounds`` /
+``.contains(x, y)`` / ``.to_dict()``)。``virtual_bounds`` 回傳聯集框(原點可能為負);``primary_monitor`` 取主螢幕;
+``monitor_for_window(rect, monitors)`` 回傳視窗主要佔據的顯示器(最大重疊);``to_virtual`` 是 ``to_local`` 的反向;
+``remap_point`` 保留分數位置,因此可跨不同解析度與 DPI 運作。
+
+執行器命令
+----------
+
+``AC_enumerate_monitors`` → ``{count, monitors, virtual_bounds}`` 與 ``AC_monitor_at_point``(``x`` / ``y``)→
+``{found, monitor}``。它們以 MCP 工具 ``ac_enumerate_monitors`` / ``ac_monitor_at_point`` 以及 Script Builder 中
+**Window** 分類下的命令提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -158,6 +158,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v133_features_doc
    doc/new_features/v134_features_doc
    doc/new_features/v135_features_doc
+   doc/new_features/v136_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -303,6 +303,11 @@
     binarize, denoise, deskew, detect_skew_angle, enhance_contrast,
     preprocess_image, to_grayscale, upscale,
 )
+# Multi-monitor / virtual-desktop geometry (which monitor, where, remapping)
+from je_auto_control.utils.monitor_layout import (
+    Monitor, enumerate_monitors, monitor_at_point, monitor_for_window,
+    primary_monitor, remap_point, to_local, to_virtual, virtual_bounds,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1119,6 +1124,15 @@ def start_autocontrol_gui(*args, **kwargs):
     "deskew",
     "detect_skew_angle",
     "enhance_contrast",
+    "Monitor",
+    "enumerate_monitors",
+    "monitor_at_point",
+    "monitor_for_window",
+    "primary_monitor",
+    "remap_point",
+    "to_local",
+    "to_virtual",
+    "virtual_bounds",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -622,6 +622,18 @@ def _add_window_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Compute staggered, overlapping window rectangles.",
     ))
+    specs.append(CommandSpec(
+        "AC_enumerate_monitors", "Window", "Enumerate Monitors",
+        description="List monitors + virtual bounds (multi-display geometry).",
+    ))
+    specs.append(CommandSpec(
+        "AC_monitor_at_point", "Window", "Monitor at Point",
+        fields=(
+            FieldSpec("x", FieldType.INT),
+            FieldSpec("y", FieldType.INT),
+        ),
+        description="Report which monitor contains a virtual point.",
+    ))
 
 
 def _add_flow_specs(specs: List[CommandSpec]) -> None:
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3431,6 +3431,26 @@ def _preprocess_image(output_path: str, source: Any = None, steps: Any = None,
             "height": int(result.shape[0])}
 
 
+def _enumerate_monitors() -> Dict[str, Any]:
+    """Adapter: list connected monitors with virtual-desktop geometry."""
+    from je_auto_control.utils.monitor_layout import (
+        enumerate_monitors, virtual_bounds)
+    monitors = enumerate_monitors()
+    bounds = virtual_bounds(monitors) if monitors else (0, 0, 0, 0)
+    return {"count": len(monitors),
+            "monitors": [monitor.to_dict() for monitor in monitors],
+            "virtual_bounds": list(bounds)}
+
+
+def _monitor_at_point(x: Any, y: Any) -> Dict[str, Any]:
+    """Adapter: report which monitor contains a virtual point."""
+    from je_auto_control.utils.monitor_layout import (
+        enumerate_monitors, monitor_at_point)
+    monitor = monitor_at_point(enumerate_monitors(), int(x), int(y))
+    return {"found": monitor is not None,
+            "monitor": monitor.to_dict() if monitor else None}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5158,6 +5178,8 @@ def __init__(self):
             "AC_find_shapes": _find_shapes,
             "AC_find_rectangles": _find_rectangles,
             "AC_preprocess_image": _preprocess_image,
+            "AC_enumerate_monitors": _enumerate_monitors,
+            "AC_monitor_at_point": _monitor_at_point,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -2816,6 +2816,34 @@ def preprocess_tools() -> List[MCPTool]:
     ]
 
 
+def monitor_layout_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_enumerate_monitors",
+            description=("List connected monitors with virtual-desktop geometry: "
+                         "{count, monitors:[{index,x,y,width,height,scale,primary,"
+                         "work}], virtual_bounds:[x,y,w,h]}. Unlike a single "
+                         "screen_size, this exposes per-monitor origins (which may "
+                         "be negative) for multi-display placement."),
+            input_schema=schema({}, required=[]),
+            handler=h.enumerate_monitors,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_monitor_at_point",
+            description=("Report which monitor contains virtual point (x, y): "
+                         "{found, monitor}. Returns found=false when the point is "
+                         "off every display."),
+            input_schema=schema({
+                "x": {"type": "integer"},
+                "y": {"type": "integer"}},
+                required=["x", "y"]),
+            handler=h.monitor_at_point,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6318,8 +6346,8 @@ def media_assert_tools() -> List[MCPTool]:
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
     modifier_state_tools, grid_locator_tools, visual_match_tools,
     color_region_tools, ssim_tools, feature_match_tools, shape_locator_tools,
-    window_layout_tools, window_arrange_tools, preprocess_tools, plugin_sdk_tools,
-    governance_tools,
+    window_layout_tools, window_arrange_tools, preprocess_tools,
+    monitor_layout_tools, plugin_sdk_tools, governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2162,6 +2162,16 @@ def preprocess_image(output_path, source=None, steps=None, scale=2.0, region=Non
                              block_size, c)
 
 
+def enumerate_monitors():
+    from je_auto_control.utils.executor.action_executor import _enumerate_monitors
+    return _enumerate_monitors()
+
+
+def monitor_at_point(x, y):
+    from je_auto_control.utils.executor.action_executor import _monitor_at_point
+    return _monitor_at_point(x, y)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)
diff --git a/je_auto_control/utils/monitor_layout/__init__.py b/je_auto_control/utils/monitor_layout/__init__.py
@@ -0,0 +1,9 @@
+"""Multi-monitor / virtual-desktop geometry (which monitor, where, remapping)."""
+from je_auto_control.utils.monitor_layout.monitor_layout import (
+    Monitor, enumerate_monitors, monitor_at_point, monitor_for_window,
+    primary_monitor, remap_point, to_local, to_virtual, virtual_bounds,
+)
+
+__all__ = ["Monitor", "enumerate_monitors", "monitor_at_point",
+           "monitor_for_window", "primary_monitor", "remap_point", "to_local",
+           "to_virtual", "virtual_bounds"]
diff --git a/je_auto_control/utils/monitor_layout/monitor_layout.py b/je_auto_control/utils/monitor_layout/monitor_layout.py
diff --git a/test/unit_test/headless/test_monitor_layout_batch.py b/test/unit_test/headless/test_monitor_layout_batch.py
