Skip to content

Commit b25b747

Browse files
authored
Merge pull request #413 from Integration-Automation/feat/step-timeline-batch
Add step_timeline: per-run step waterfall + bottleneck steps
2 parents 59767cf + a108984 commit b25b747

11 files changed

Lines changed: 329 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-25) — Per-Run Step Timeline (waterfall + bottleneck steps)
4+
5+
Read why *this* run was slow — a step waterfall and its bottlenecks. Full reference: [`docs/source/Eng/doc/new_features/v194_features_doc.rst`](docs/source/Eng/doc/new_features/v194_features_doc.rst).
6+
7+
- **`build_timeline` / `critical_steps`** (`AC_build_timeline`, `AC_critical_steps`): the action profiler aggregates timings by step *name* across runs — useless for "why was *this* run slow". This turns one run's ordered steps into a waterfall (each step's offset, duration, and `pct` share of the total) with the `bottleneck` step and a `parallelism` ratio (`> 1` when steps overlap via explicit `start` times); `critical_steps` ranks the dominant steps to optimise. A step is any `{name, duration, start?}` dict. Pure stdlib. No `PySide6`.
8+
39
## What's new (2026-06-25) — Flaky-Test Co-Failure Clustering
410

511
Find the tests that flake *together* — and the shared root cause behind them. Full reference: [`docs/source/Eng/doc/new_features/v193_features_doc.rst`](docs/source/Eng/doc/new_features/v193_features_doc.rst).
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
Per-Run Step Timeline (waterfall + bottleneck steps)
2+
====================================================
3+
4+
The action profiler aggregates timings by step *name* across many runs — great
5+
for "which action is slow on average", useless for "why was *this* run slow". A
6+
single run is an ordered timeline: step A ran, then B, then C, and one of them
7+
dominated. ``step_timeline`` turns one run's steps into a waterfall (each step's
8+
offset from the start, its duration and its share of the total) and ranks the
9+
bottleneck steps, so you can read a single slow run instead of an average.
10+
11+
* :func:`build_timeline` — the waterfall + total / busy / bottleneck /
12+
parallelism,
13+
* :func:`critical_steps` — the steps that dominate the run, longest first.
14+
15+
A step is any dict with a name (default ``"name"``) and a ``duration``; an
16+
optional ``start`` places it on an absolute timeline (overlapping / parallel
17+
steps), else steps are laid out back-to-back. Pure standard library; no device,
18+
no ``PySide6``.
19+
20+
Headless API
21+
------------
22+
23+
.. code-block:: python
24+
25+
from je_auto_control import build_timeline, critical_steps
26+
27+
steps = [{"name": "login", "duration": 1.0},
28+
{"name": "load_dashboard", "duration": 4.0},
29+
{"name": "submit", "duration": 1.0}]
30+
31+
build_timeline(steps)
32+
# {"steps": [{"name": "login", "offset": 0.0, "duration": 1.0, "pct": 16.7},
33+
# {"name": "load_dashboard", "offset": 1.0, ..., "pct": 66.7}, ...],
34+
# "total": 6.0, "busy": 6.0,
35+
# "bottleneck": {"name": "load_dashboard", "duration": 4.0},
36+
# "parallelism": 1.0}
37+
38+
critical_steps(steps, top=2)
39+
# [{"name": "load_dashboard", "duration": 4.0, "pct": 66.7},
40+
# {"name": "login", "duration": 1.0, "pct": 16.7}]
41+
42+
``total`` is the wall-clock span, ``busy`` the summed step time; ``parallelism`` =
43+
busy / total is ``1.0`` for a purely sequential run and ``> 1`` when steps overlap
44+
(supply ``start`` times). ``pct`` is each step's share of the total time.
45+
46+
Executor commands
47+
-----------------
48+
49+
``AC_build_timeline`` (``steps``) and ``AC_critical_steps`` (``steps`` / ``top``).
50+
They are exposed as read-only ``ac_*`` MCP tools and as Script Builder commands
51+
under **Testing**.
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
單次執行的步驟時間軸(瀑布圖 + 瓶頸步驟)
2+
==========================================
3+
4+
動作 profiler 把計時按步驟*名稱*跨多次執行聚合——很適合「哪個動作平均較慢」,卻無助於「為什麼
5+
*這一次*執行很慢」。單次執行是一條有序時間軸:步驟 A 跑完、接著 B、再 C,其中某一步主導了時間。
6+
``step_timeline`` 把一次執行的步驟轉成瀑布圖(每步距起點的偏移、其時長、其占總時間的比例),並
7+
排名瓶頸步驟,讓你能讀懂單一慢執行,而非平均值。
8+
9+
* :func:`build_timeline` ——瀑布圖加上 total / busy / bottleneck / parallelism,
10+
* :func:`critical_steps` ——主導該次執行的步驟,最長者在前。
11+
12+
步驟可為任何帶名稱(預設 ``"name"``)與 ``duration`` 的字典;選填 ``start`` 會把它放到絕對
13+
時間軸上(重疊 / 平行步驟),否則步驟會背靠背排列。純標準庫;不涉及裝置,不匯入 ``PySide6``。
14+
15+
無頭 API
16+
--------
17+
18+
.. code-block:: python
19+
20+
from je_auto_control import build_timeline, critical_steps
21+
22+
steps = [{"name": "login", "duration": 1.0},
23+
{"name": "load_dashboard", "duration": 4.0},
24+
{"name": "submit", "duration": 1.0}]
25+
26+
build_timeline(steps)
27+
# {"steps": [{"name": "login", "offset": 0.0, "duration": 1.0, "pct": 16.7},
28+
# {"name": "load_dashboard", "offset": 1.0, ..., "pct": 66.7}, ...],
29+
# "total": 6.0, "busy": 6.0,
30+
# "bottleneck": {"name": "load_dashboard", "duration": 4.0},
31+
# "parallelism": 1.0}
32+
33+
critical_steps(steps, top=2)
34+
# [{"name": "load_dashboard", "duration": 4.0, "pct": 66.7},
35+
# {"name": "login", "duration": 1.0, "pct": 16.7}]
36+
37+
``total`` 是牆鐘時間跨度,``busy`` 是各步驟時長總和;``parallelism`` = busy / total,純序列執行
38+
為 ``1.0``,步驟重疊時 ``> 1``(需提供 ``start`` 時間)。``pct`` 是每步占總時間的比例。
39+
40+
執行器指令
41+
----------
42+
43+
``AC_build_timeline``(``steps``)與 ``AC_critical_steps``(``steps`` / ``top``)。皆以唯讀
44+
``ac_*`` MCP 工具及 Script Builder 指令(位於 **Testing** 分類下)形式提供。

je_auto_control/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,8 @@
9696
from je_auto_control.utils.run_diff import diff_runs, summarize_run_diff
9797
# Flaky-test co-failure clustering (Jaccard over shared failing runs)
9898
from je_auto_control.utils.flake_cluster import cofailure_pairs, failure_clusters
99+
# Per-run step waterfall + bottleneck (critical) steps
100+
from je_auto_control.utils.step_timeline import build_timeline, critical_steps
99101
# VLM element locator (headless)
100102
from je_auto_control.utils.vision import (
101103
VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1676,6 +1678,7 @@ def start_autocontrol_gui(*args, **kwargs):
16761678
"normalize_error", "failure_signature", "group_failures",
16771679
"diff_runs", "summarize_run_diff",
16781680
"cofailure_pairs", "failure_clusters",
1681+
"build_timeline", "critical_steps",
16791682
# VLM locator
16801683
"VLMNotAvailableError", "locate_by_description", "click_by_description",
16811684
"verify_description",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2755,6 +2755,21 @@ def _add_audit_specs(specs: List[CommandSpec]) -> None:
27552755
),
27562756
description="Test pairs that fail together above a Jaccard threshold.",
27572757
))
2758+
specs.append(CommandSpec(
2759+
"AC_build_timeline", "Testing", "Step Timeline (waterfall)",
2760+
fields=(FieldSpec("steps", FieldType.STRING,
2761+
placeholder='[{"name": "login", "duration": 1.2}]'),),
2762+
description="Per-run step waterfall: offsets, durations, bottleneck.",
2763+
))
2764+
specs.append(CommandSpec(
2765+
"AC_critical_steps", "Testing", "Critical (Bottleneck) Steps",
2766+
fields=(
2767+
FieldSpec("steps", FieldType.STRING,
2768+
placeholder='[{"name": "login", "duration": 1.2}]'),
2769+
FieldSpec("top", FieldType.INT, optional=True, default=3),
2770+
),
2771+
description="The steps that dominate a run's time, longest first.",
2772+
))
27582773
specs.append(CommandSpec(
27592774
"AC_scan_secrets", "Tools", "Scan for Hardcoded Secrets",
27602775
description="Scan 'data' (JSON view) for hardcoded secrets that "

je_auto_control/utils/executor/action_executor.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4402,6 +4402,24 @@ def _cofailure_pairs(runs: Any, threshold: Any = 0.5) -> Dict[str, Any]:
44024402
return {"pairs": pairs, "count": len(pairs)}
44034403

44044404

4405+
def _build_timeline(steps: Any) -> Dict[str, Any]:
4406+
"""Adapter: a per-run step waterfall (offsets / durations / bottleneck)."""
4407+
import json
4408+
from je_auto_control.utils.step_timeline import build_timeline
4409+
if isinstance(steps, str):
4410+
steps = json.loads(steps)
4411+
return build_timeline(steps)
4412+
4413+
4414+
def _critical_steps(steps: Any, top: Any = 3) -> Dict[str, Any]:
4415+
"""Adapter: the steps that dominate a run's time (bottlenecks)."""
4416+
import json
4417+
from je_auto_control.utils.step_timeline import critical_steps
4418+
if isinstance(steps, str):
4419+
steps = json.loads(steps)
4420+
return {"steps": critical_steps(steps, top=int(top))}
4421+
4422+
44054423
def _image_histogram(source: Any = None, bins: Any = 32, space: str = "hsv",
44064424
region: Any = None) -> Dict[str, Any]:
44074425
"""Adapter: per-channel colour histogram of an image / the screen."""
@@ -6635,6 +6653,8 @@ def __init__(self):
66356653
"AC_diff_runs": _diff_runs,
66366654
"AC_failure_clusters": _failure_clusters,
66376655
"AC_cofailure_pairs": _cofailure_pairs,
6656+
"AC_build_timeline": _build_timeline,
6657+
"AC_critical_steps": _critical_steps,
66386658
"AC_image_histogram": _image_histogram,
66396659
"AC_histogram_changed": _histogram_changed,
66406660
"AC_changed_regions": _changed_regions,

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7720,6 +7720,30 @@ def flakiness_tools() -> List[MCPTool]:
77207720
handler=h.cofailure_pairs,
77217721
annotations=READ_ONLY,
77227722
),
7723+
MCPTool(
7724+
name="ac_build_timeline",
7725+
description=("Per-run step waterfall from 'steps' (list of {name,"
7726+
"duration,start?}): {steps:[{name,offset,duration,pct}], "
7727+
"total, busy, bottleneck, parallelism}. Reads ONE slow "
7728+
"run, not a per-name average."),
7729+
input_schema=schema({
7730+
"steps": {"type": "array", "items": {"type": "object"}}},
7731+
required=["steps"]),
7732+
handler=h.build_timeline,
7733+
annotations=READ_ONLY,
7734+
),
7735+
MCPTool(
7736+
name="ac_critical_steps",
7737+
description=("The 'top' steps that dominate a run's time (bottlenecks "
7738+
"to optimise): {steps:[{name,duration,pct}]}, longest "
7739+
"first."),
7740+
input_schema=schema({
7741+
"steps": {"type": "array", "items": {"type": "object"}},
7742+
"top": {"type": "integer"}},
7743+
required=["steps"]),
7744+
handler=h.critical_steps,
7745+
annotations=READ_ONLY,
7746+
),
77237747
]
77247748

77257749

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2568,6 +2568,16 @@ def cofailure_pairs(runs, threshold=0.5):
25682568
return _cofailure_pairs(runs, threshold)
25692569

25702570

2571+
def build_timeline(steps):
2572+
from je_auto_control.utils.executor.action_executor import _build_timeline
2573+
return _build_timeline(steps)
2574+
2575+
2576+
def critical_steps(steps, top=3):
2577+
from je_auto_control.utils.executor.action_executor import _critical_steps
2578+
return _critical_steps(steps, top)
2579+
2580+
25712581
def image_histogram(source=None, bins=32, space="hsv", region=None):
25722582
from je_auto_control.utils.executor.action_executor import _image_histogram
25732583
return _image_histogram(source, bins, space, region)
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Per-run step waterfall timeline + bottleneck (critical) step ranking."""
2+
from je_auto_control.utils.step_timeline.step_timeline import (
3+
build_timeline, critical_steps,
4+
)
5+
6+
__all__ = ["build_timeline", "critical_steps"]
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
"""Build a per-run step waterfall and find the run's bottleneck steps.
2+
3+
The action profiler aggregates timings by step *name* across many runs — great for
4+
"which action is slow on average", useless for "why was *this* run slow". A single
5+
run is an ordered timeline: step A ran, then B, then C, and one of them dominated.
6+
``step_timeline`` turns one run's steps into a waterfall (each step's offset from
7+
the start, duration and share of the total) and ranks the bottleneck steps, so you
8+
can read a single slow run instead of an average.
9+
10+
A step is any dict with a name (default ``"name"``) and a ``duration``; an optional
11+
``start`` places it on an absolute timeline (overlapping / parallel steps), else
12+
steps are laid out back-to-back. Pure standard library; no device, no ``PySide6``.
13+
"""
14+
from typing import Any, Dict, List, Sequence
15+
16+
Step = Dict[str, Any]
17+
18+
19+
def _normalize(steps: Sequence[Step], name_key: str, start_key: str,
20+
duration_key: str) -> List[Dict[str, Any]]:
21+
"""Resolve each step to ``{name, start, end, duration}`` (sequential if no start)."""
22+
resolved, cursor = [], 0.0
23+
for step in steps:
24+
duration = float(step.get(duration_key, 0.0) or 0.0)
25+
raw_start = step.get(start_key)
26+
start = float(raw_start) if raw_start is not None else cursor
27+
end = start + duration
28+
cursor = max(cursor, end)
29+
resolved.append({"name": str(step.get(name_key, "")), "start": start,
30+
"end": end, "duration": duration})
31+
return resolved
32+
33+
34+
def build_timeline(steps: Sequence[Step], *, name_key: str = "name",
35+
start_key: str = "start",
36+
duration_key: str = "duration") -> Dict[str, Any]:
37+
"""Return a waterfall timeline for one run.
38+
39+
``{steps:[{name, offset, duration, pct}], total, busy, bottleneck,
40+
parallelism}`` — ``total`` is the wall-clock span, ``busy`` the summed step
41+
time, ``parallelism`` = busy / total (1.0 for a purely sequential run),
42+
``bottleneck`` the longest single step.
43+
"""
44+
resolved = _normalize(steps, name_key, start_key, duration_key)
45+
if not resolved:
46+
return {"steps": [], "total": 0.0, "busy": 0.0, "bottleneck": None,
47+
"parallelism": 0.0}
48+
base = min(step["start"] for step in resolved)
49+
span = max(step["end"] for step in resolved) - base
50+
busy = sum(step["duration"] for step in resolved)
51+
rows = [{"name": step["name"], "offset": round(step["start"] - base, 6),
52+
"duration": step["duration"],
53+
"pct": round(step["duration"] / span * 100, 1) if span > 0 else 0.0}
54+
for step in resolved]
55+
bottleneck = max(resolved, key=lambda step: step["duration"])
56+
return {"steps": rows, "total": round(span, 6), "busy": round(busy, 6),
57+
"bottleneck": {"name": bottleneck["name"],
58+
"duration": bottleneck["duration"]},
59+
"parallelism": round(busy / span, 3) if span > 0 else 1.0}
60+
61+
62+
def critical_steps(steps: Sequence[Step], *, name_key: str = "name",
63+
start_key: str = "start", duration_key: str = "duration",
64+
top: int = 3) -> List[Dict[str, Any]]:
65+
"""Return the ``top`` steps that dominate the run, longest first.
66+
67+
Each entry is ``{name, duration, pct}`` where ``pct`` is the step's share of
68+
the total step time — the bottlenecks worth optimising.
69+
"""
70+
resolved = _normalize(steps, name_key, start_key, duration_key)
71+
busy = sum(step["duration"] for step in resolved) or 1.0
72+
ranked = sorted(resolved, key=lambda step: step["duration"], reverse=True)
73+
return [{"name": step["name"], "duration": step["duration"],
74+
"pct": round(step["duration"] / busy * 100, 1)}
75+
for step in ranked[:max(1, int(top))]]

0 commit comments

Comments
 (0)