AppRecording

A research prototype for capturing replay-oriented Android user interaction traces with AccessibilityService.

AppRecording records user interactions performed in selected Android applications and serializes them as structured JSON. In addition to action metadata, the current research version captures the UI hierarchy before and after each interaction so that downstream tools can replay traces, compare UI states, and use real user behavior to guide automated test generation.

The project was developed by Serkan Seker as the recording component of the master's thesis:

Combining Genetic Algorithms and User Interaction Recordings to Enhance Automated App Testing

Master's Thesis in Computer Science, University of Passau, Software Engineering II, February 2026

Application

  

AppRecording interface and Accessibility Service activation (thesis Figure 3.2).

Why this project exists

Android GUIs are event-driven and strongly state-dependent. Random test generators can repeatedly explore easy-to-reach screens while missing flows that require a specific sequence of clicks, text input, and navigation. AppRecording captures those meaningful sequences so they can be reused as guidance for search-based testing.

This repository contains the Android recorder. The thesis's replay, trace-based seeding, and replay-mutation mechanisms were implemented in the MATE Android testing framework and are outside the scope of this repository.

flowchart LR
    U["User interaction"] --> R["AppRecording<br/>AccessibilityService"]
    R --> J["JSON trace<br/>actions + locators + UI states"]
    J --> P["Trace-to-action conversion<br/>in MATE"]
    P --> S["Trace-based replay<br/>and population seeding"]
    P --> M["State-aware<br/>replay mutation"]
    S --> G["GA-based Android<br/>test generation"]
    M --> G

Loading

Recorder capabilities

• Captures CLICK, LONG_CLICK, TEXT_INPUT, SCROLL, and BACK_BUTTON interactions.
• Stores timestamps, package names, action parameters, and target metadata: visible text, class, resource ID, hierarchy path, and screen coordinates.
• Records adapter_index when Android exposes enough list-item context.
• Serializes the active UI hierarchy before and after every recorded action.
• Recovers some missing clicks from focus events or incomplete accessibility events.
• Uses recently observed long-click metadata to enrich otherwise incomplete toolbar or icon clicks.
• Filters Home and Recents interactions from System UI while retaining Back navigation.
• Displays and clears the recorded JSON from the app UI.

Trace format

Interactions are appended to user_interactions.json in the app's private files directory. A typical target-based event has the following shape:

{
  "timestamp": 1769335893796,
  "package": "de.rampro.activitydiary",
  "type": "CLICK",
  "adapter_index": 2,
  "element": {
    "text": "Open navigation",
    "class": "android.widget.ImageButton",
    "resource_id": "com.example:id/navigation",
    "path": "0/0/1",
    "x": 84,
    "y": 156
  },
  "ui_state_before_xml": "<?xml version='1.0' ... ?>",
  "ui_state_after_xml": "<?xml version='1.0' ... ?>"
}

Field availability depends on what Android exposes for a particular widget. TEXT_INPUT additionally stores value. Vertical SCROLL events store centerX, centerY, direction, and distance.

Architecture

The implementation is intentionally compact:

Component	Responsibility
MainActivity	Opens Accessibility settings and displays or clears saved JSON
UserInteractionService	Observes accessibility events, extracts metadata, creates UI dumps, and persists traces
InteractionType	Defines the normalized interaction vocabulary
accessibility_service_config.xml	Configures event types, flags, and the package allowlist

Build

Requirements

• Android Studio or Android SDK command-line tools
• JDK 17
• Android SDK 34
• An Android device or emulator running API 28 or newer

The project uses Android Gradle Plugin 8.5.1 and Gradle 8.7.

git clone https://github.com/srknskr/apprecording.git
cd apprecording
./gradlew assembleDebug

The debug APK is generated at:

app/build/outputs/apk/debug/app-debug.apk

Install it on a connected device or emulator:

adb install -r app/build/outputs/apk/debug/app-debug.apk

Configure target applications

Recording is restricted by the package allowlist in:

app/src/main/res/xml/accessibility_service_config.xml

Add the package name of each application under test to android:packageNames, then rebuild and reinstall AppRecording.

Record a scenario

1. Launch AppRecording.
2. Select Open Accessibility Settings.
3. Enable the App Recording accessibility service.
4. Open an allowlisted application and perform the scenario.
5. Return to AppRecording and select Show Saved JSON Data.
6. Export the trace before starting a new recording session.

For a debug installation, the trace can be copied with:

adb exec-out run-as com.mate.apprecording \
  cat files/user_interactions.json > user_interactions.json

MainActivity clears the existing JSON file when it is created, so export any trace you want to keep before relaunching the activity.

Research context

AppRecording is the data-capture component of a larger thesis pipeline. It records an ordered action sequence and the UI state surrounding each action. The MATE extension then converts the JSON events into executable PrimitiveAction objects for replay and genetic-algorithm integration.

The thesis studied five research questions. RQ1 is the direct record-and-replay validation of the traces produced by this application. RQ2-RQ5 evaluate how those validated traces guide MIO and MOSA through population seeding and replay mutation.

RQ1 - Record-and-Replay Similarity

Research question: Does replaying a recorded user interaction trace result in a UI state similar to the originally recorded state?

This repository implements the recording side of RQ1. Replay and measurement were executed in MATE with JSON_REPLAY and JSON_REPLAY_CHROMOSOME_FACTORY.

Methodology

1. A user completes a scenario while AppRecording captures the interaction sequence and UI hierarchy.
2. The JSON trace is converted into executable actions in MATE.
3. Every scenario is replayed independently 10 times.
4. The final recorded UI hierarchy is compared with the final hierarchy from each replay.
5. Each UI node contributes a signature derived from its class, resource-id, text, and content-desc attributes.
6. The two UI trees become frequency vectors over those node signatures and are compared with cosine similarity:

cos(v_recorded, v_replayed) =
    (v_recorded . v_replayed) /
    (||v_recorded|| * ||v_replayed||)

For every scenario, the median similarity across its 10 replays is calculated. The application-level analysis then reports the median and IQR of those scenario medians. The minimum scenario median and the global minimum across all replay runs are retained to expose rare failures that a median could hide.

Experimental scale

• 7 open-source Android applications
• 53 recorded scenarios
• 530 replay executions
• 10 independent replays per scenario

Results

Application	Scenarios	Median of scenario medians	IQR	Min. scenario median	Global minimum
Shopping List	9	1.000	0.000	1.000	0.280
TSCH_BYL	8	1.000	0.000	0.999	0.332
Activity Diary	8	1.000	0.009	0.986	0.979
Aksharam	5	1.000	0.005	0.994	0.261
Thorium	7	0.999	0.012	0.912	0.311
PIN Mnemonic	5	1.000	0.000	1.000	1.000
Rental Calc	11	1.000	0.000	0.987	0.230

Distribution of scenario-level median cosine similarity across all 53 scenarios (thesis Figure 6.1).

The typical replay result is very strong: application-level medians are 0.999-1.000, and most distributions are tightly concentrated near 1.0. This supports the feasibility of using AppRecording traces as replayable test material.

The minima tell an equally important story. Dynamic content, asynchronous loading, timing differences, and unexpected UI overlays can occasionally cause a replay to diverge even when its scenario median is excellent. PIN Mnemonic remained perfectly stable in all runs, while Rental Calc, Shopping List, Aksharam, and Thorium contained rare low-similarity executions. The RQ1 conclusion is therefore not that replay is deterministic, but that it is typically reliable and should be paired with robustness checks.

Beyond RQ1 - Guiding Genetic Algorithms

After validating replay feasibility, the thesis evaluated two ways of integrating user traces into genetic algorithms:

1. Trace-based seeding: initialize a configurable share of generated test cases with recorded scenarios.
2. Replay mutation: inject trace segments during search when the current UI hierarchy is sufficiently similar to a recorded state.

Selected GA findings

• The globally selected MIO configuration used a seed fraction of 0.8, a UI similarity threshold of 0.2, and a replay probability of 0.4.
• That trace-guided configuration achieved higher median final branch coverage than the baseline in 6 of 7 subject applications.
• Improvements were statistically significant for Shopping List, TSCH_BYL, and Activity Diary; effects and run-to-run stability remained application-dependent.

The broader thesis finding is deliberately nuanced: user traces are useful genetic material for automated Android testing, but they complement rather than replace evolutionary exploration. The MATE-side algorithms, experiment automation, and complete coverage study belong in a separate companion repository; this repository remains focused on trustworthy trace capture.

Limitations

• Accessibility events do not expose complete or stable locator information for every widget, custom view, dialog, or toolbar item.
• Only vertical scroll directions are retained because horizontal direction inference was not reliable enough in the evaluated setup.
• Coordinate-based metadata is sensitive to screen size, resolution, Android version, and layout changes.
• Dynamic content, asynchronous loading, and system dialogs can make replay nondeterministic.
• UI-state equality is a practical replay signal; it does not prove identical internal application state.
• This repository records traces but does not include a standalone replay engine.

Privacy and responsible use

An accessibility service can observe visible UI text, entered values, and screen structure. Recorded traces may therefore contain personal or sensitive information.

Use AppRecording only on applications, devices, and data for which you have explicit authorization. Review and sanitize traces before sharing them, and do not enable the service during unrelated personal activity.

Branch history

The public repository mirrors the final source snapshots as separate, sanitized GitHub branches. Historical IDE caches and machine-local files are intentionally excluded.

Branch	Purpose
updated-app-recording	Latest thesis recorder with UI-state snapshots; GitHub default
app-recording	Earlier standalone development history
main	Original integrated baseline

Academic attribution

• Author: Serkan Seker
• Supervisor: Prof. Dr. Gordon Fraser
• Advisor: Patric Feldmeier
• External examiner: Prof. Dr. Christian Hammer
• Institution: University of Passau, Chair of Software Engineering II

Suggested citation:

@mastersthesis{seker2026interactionrecordings,
  author = {Serkan Seker},
  title = {Combining Genetic Algorithms and User Interaction Recordings
           to Enhance Automated App Testing},
  school = {University of Passau},
  year = {2026},
  type = {Master's Thesis in Computer Science}
}

License

No license file is currently provided. Please contact the author before redistributing or reusing the source code.