Analysis Framework

Analysis framework for comparing GUI agent and human interaction traces, as described in:

Maria Movin, Claudia Hauff, Aron Henriksson, Panagiotis Papapetrou. Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems. arXiv:2604.07929 (2026). https://arxiv.org/abs/2604.07929

Ships with synthetic demo data so the notebooks run end-to-end out of the box. Swap in your own data to apply the framework to new studies.

Structure

├── README.md
├── requirements.txt                 ← Python dependencies
├── generate_sample_data.py          ← synthetic trace generator
├── 01_success_effort.ipynb          ← Task Outcomes and Effort
├── 02_query_formulation.ipynb       ← Query Formulation Patterns
└── 03_navigation.ipynb              ← Navigation Behavior

Notebooks

01 — Task Outcomes and Effort

Metric	Method
Success rates	Wilson 95% CIs, Pearson χ²
Effort (time, actions)	Mean ± SD, median [IQR], two-sided MWU
Participant subgroups	Expert vs regular, familiar vs unfamiliar
Multiple-comparison correction	Holm–Bonferroni step-down, tiered families
Non-compliant runs	Descriptive comparison

02 — Query Formulation Patterns

Metric	Method
First-query similarity	SequenceMatcher, macro-averaged with bootstrap 95% CIs
Robustness check	TF-IDF cosine (word unigrams + bigrams)
Subgroup similarity	Expert–Agent vs Regular–Agent MWU
Typicality	Centroid proximity percentile shares
Coverage	Weighted query mass covered at each threshold τ
Efficiency	Cumulative coverage at τ = 0.6 vs baselines

03 — Navigation Behavior

Metric	Method
Transition graphs	Macro-averaged directed graphs by cohort
High-frequency overlap	Top-k Jaccard (k = 10, 20)
Aggregation	Pooled (micro) and task-equal (macro)

Implementation notes

• Synthetic data. generate_sample_data.py produces fabricated traces for demonstration. Replace with your own data loader to analyse real traces. The saved cell outputs in the notebooks come from these synthetic traces; each notebook sets a fixed random seed so the outputs are reproducible.
• Query normalization. Lowercase, URL-decode, collapse whitespace, trim.
• First-query definition. First search_execute event per run (submitted query, not partial typing events).
• Navigation self-loops. Raw-level self-loops are dropped before
  abstraction. Abstract-level self-loops (e.g. search → search from
  different search instances) arise naturally and are retained.

Quick start

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Data schema

Column	Type	Description
participant_id	str	Unique participant identifier
task_id	str	Task label (T1–T10)
group	str	"user" or "agent"
timestamp_ms	int	Event timestamp (epoch ms)
action_type	str	Action category
query_text	str	Search query text (nullable)
page_from	str	Source page identifier
page_to	str	Destination page identifier
success	int	1 if task completed successfully
expertise_level	str	User expertise category
task_familiar	bool	Whether user was familiar with the task domain
used_shortcut	bool	Protocol-violation flag for agents

Requirements

• Python ≥ 3.9
• pandas, numpy, scipy, matplotlib, networkx, scikit-learn