Skip to content
BranchesTags

Repository files navigation

🇨🇳 简体中文

OpenAPA

Open Agentic Process Automation (Open APA) is a practical and extensible framework for building computer use agents that can reliably execute long-horizon, real-world tasks across applications, interfaces, and workflows. Built on top of our first-place performance in the OSWorld Benchmark.

License: MIT Python Platform

Overview

OpenAPA (Open Agentic Process Automation) is a cross-platform Computer-Use Agent. Built upon the architectural principles of Agent-S (Large Multimodal Agents for the Browser and Desktop), it combines the perception and reasoning capabilities of Large Multimodal Models (LMMs) to achieve automated execution of complex GUI tasks. This report elaborates on the technical implementation of OpenAPA across five dimensions: knowledge acquisition, knowledge sharing, advance planning, session pruning, and multi-agent coordination.

Key Features

  • Cross-Platform: Supports Windows, macOS, and Linux out of the box.
  • Pluggable LLM Backend: Easily switch between OpenAI, Anthropic, Gemini, and local models.
  • Robust Grounding: Utilizes a specialized Grounding Agent for pixel-perfect UI interactions, eliminating reliance on brittle HTML/DOM structures.
  • State-of-the-Art Benchmark Performance: Core architecture powers top-tier performance on the rigorous OSWorld benchmark.

System Architecture

graph TD
    %% Main Flow
    User([User Task / Instruction]) --> KlickAgent["KlickAgent (Main Orchestrator)"]
    
    subgraph "OpenAPA Core Architecture"
        subgraph "KlickAgent Loop"
            Reflection["1. Reflection<br>(Analyze previous step & current screenshot)"]
            Decision["2. Decision Making<br>(Determine next action based on reflection & history)"]
            Reflection --> Decision
        end
        KlickAgent -.-> Reflection
        
        %% Executor acts as the central hub for dispatching
        KlickExecutor["KlickExecutor<br>(Action Dispatcher & Executor)"]
        
        %% Agents Layer
        subgraph "Specialized Agents"
            CodeAgent["CodeAgent<br>(Planning & Coding)"]
            GroundingAgent["GroundingAgent<br>(Visual Localization)"]
        end
        
        %% LLM Layer
        subgraph "LMM Engine Layer"
            MainLMM(("Main Model<br>(e.g., Gemini)"))
            GroundingLMM(("Grounding Model<br>(e.g., Doubao-seed)"))
        end
    end

    %% Environment
    Environment["OS / Application GUI"]

    %% Connections
    Decision -- "3. Passes action to executor" --> KlickExecutor
    KlickExecutor -- "4a. If action is Code: Delegates task" --> CodeAgent
    CodeAgent -- "Returns execution result" --> KlickExecutor
    KlickExecutor -- "4b. If action is UI: Queries Coordinates" --> GroundingAgent
    GroundingAgent -- "Returns (x,y)" --> KlickExecutor
    
    %% Model connections
    Reflection -. "Analyze" .-> MainLMM
    Decision -. "Planning" .-> MainLMM
    CodeAgent -. "Reasoning" .-> MainLMM
    GroundingAgent -. "Vision" .-> GroundingLMM
    
    %% Execution Loop
    KlickExecutor -- "5. exec() pyautogui/<br>OS commands" --> Environment
    Environment -- "6. New Screenshot /<br>Observation" --> Reflection
Loading

Core Technology Implementation

  • Knowledge Acquisition and Sharing: To eliminate Large Multimodal Model (LMM) hallucinations, the system transforms reliance on internal model knowledge into active external acquisition. Acquired knowledge is persisted within a global runtime context and shared across different agents and multiple steps through prompt injection, ensuring a closed-loop consistency.
  • Advance Planning: The system introduces a hierarchical planning architecture. During the task initiation phase, a specialized planning engine generates high-level execution plans focusing on "goals" rather than "action details." Throughout execution, a reflection mechanism dynamically updates plan progress based on real-time trajectories and adjusts subsequent steps based on environmental feedback, preventing the agent from deviating during long-sequence tasks.
  • Session Pruning: Addressing the "token explosion" in long sessions, the session management module implements automated context management. A "sliding window" strategy is used to dynamically retain core reasoning trajectories and recent high-resolution screenshots. Simultaneously, a token-budget-based pruning logic prioritizes system instructions and the latest observations, significantly reducing API costs while boosting inference response speed and model instruction-following performance.
  • Agent Coordination: OpenAPA establishes a clear collaborative pipeline emphasizing complementary roles between agents. In scenarios such as Office suites or complex Web interactions—where pure code execution struggle to drive intricate GUI logic—the Code-Agent focuses on high-performance data processing (e.g., cleaning, transformation, and calculation), while the UI-Agent handles high-level visual element localization, complex workflow orchestration, and interactive operations. The two collaborate through a "delegate-execute-verify" loop: any results from code execution must be visually confirmed by the UI agent. This "Data to Code, Operation to UI" division of labor, combined with closed-loop verification, greatly enhances the robustness of complex tasks.

Quick Start

Installation

git clone https://github.com/laiye-ai/open-apa.git
cd open-apa
pip install -r requirements.txt
pip install -e .

Running an Agent locally

To run and test the OpenAPA agent locally on your machine, you can use the built-in klick_runner.py script. This serves as an entry point for standalone tasks.

1. Configuration

IMPORTANT: OpenAPA relies heavily on visual observation of the desktop environment. You MUST use Large Multimodal Models (LMMs) that support vision (e.g., GPT-4o, Claude 3.5 Sonnet, Gemini 1.5/2.0 Pro) for the Main Agent. Standard text-only models will not work.

Before running tasks, configure your API keys and models in conf/conf.yaml:

# Model Configuration (Main reasoning & planning)
model:
  engine_type: "openai"      # e.g., openai, anthropic, gemini
  model: "gemini"            # The model name
  base_url: "https://..."    # API endpoint
  api_key: "your_api_key"    # Your API Key

# Grounding Configuration (Vision & coordinate alignment)
grounding:
  engine_type: "openai"
  model: "doubao-seed-1-8-251228"
  base_url: "https://..."
  api_key: "your_api_key"

2. Customizing and Running Tasks The klick_runner.py script contains a built-in task_list in its if __name__ == "__main__": block. By default, it runs a test task (e.g., opening Notepad, typing text, and saving).

To test your own tasks:

  1. Open scripts/klick_runner.py.
  2. Locate the task_list array at the bottom of the file.
  3. Modify or add a task dictionary with your natural language instruction: 
    {
    'task': "Open Chrome, search for the weather in Tokyo today, and take a screenshot.",
    'output_dir': os.path.join(os.getcwd(), "test_outputs/my-custom-task/"),
    'max_steps': 15
}
  4. Change the test_idx variable to point to your new task in the array.
  5. Run the script: 
    python scripts/klick_runner.py
    The agent's reasoning logs and step-by-step screenshots will be saved to the specified output_dir.

Using the openapa CLI

After pip install -e . the package exposes an openapa command for running tasks without editing source files.

1. First-time setup — interactive config wizard

openapa config init                      # writes ~/.openapa/conf.yaml

The wizard walks through every field (main model, grounding model, screenshot, runner defaults) and validates each entry. Press Enter to accept the default in parens; api_key input is hidden via getpass.

Other config commands:

openapa config show                      # print active config (api_key masked)
openapa config set model.api_key sk-xxx  # update one field, comments preserved
openapa config unset screenshot.height
openapa config check                     # ping main + grounding models

Config search order (first hit wins): -c <path> → $OPENAPA_CONFIG → ~/.openapa/conf.yaml. CWD-relative ./conf/conf.yaml is intentionally not auto-loaded; pass -c conf/conf.yaml if you want a project-local file.

openapa config set/unset and the attachment extractors below need a few extra packages:

pip install -r requirements-cli.txt      # ruamel.yaml, pypdf, python-docx, openpyxl

2. Run a single task

# inline
openapa run "Open Notepad and type Hello"

# from a long text file (multi-line / multilingual ok)
openapa run -f tasks/long_task.txt

# explicit log directory + step cap
openapa run "..." -o test_outputs/foo -n 50

# preview without calling the LLM
openapa run "..." --dry-run

Outputs land in test_outputs/run_<timestamp>/ by default — runner.log, step_*.png screenshots, and llm_call_log.jsonl. Exit code: 0 = task DONE, 1 = FAIL or hit max-steps, 2 = config/usage error, 3 = runtime error.

3. Tasks with attachments (PDF/DOCX/XLSX/images)

Create a YAML manifest:

# tasks/with_attachments.yaml
name: reimbursement
instruction: |
  Submit a reimbursement form on the company portal.
  Use the attached invoice PDFs and payment screenshots.
attachments:
  - path: ./files/invoice_01.pdf            # text-extracted, inlined into instruction
  - path: ./files/payment_01.jpg            # image, sent multimodally to the plan LMM
    label: payment screenshot #1
  - path: ./files/invoice_02.pdf
  - path: ./files/payment_02.jpg
max_steps: 100
openapa run -m tasks/with_attachments.yaml

Images go to the planning LMM as multimodal content; PDF / DOCX / XLSX / text files are extracted to plain text and inlined into the instruction inside <<<attachment ...>>> ... <<<end attachment>>> blocks.

4. Batch runs

# tasks/regression.yaml
tasks:
  - name: notepad-hello
    instruction: Open Notepad, type 'Hello', save as test.txt
    max_steps: 20
  - name: image-rearrange
    instruction: Divide tilearray.png on desktop into three vertical sections...
    max_steps: 20
openapa batch tasks/regression.yaml -o test_outputs/regression --continue-on-error

Each subtask gets its own <output_base>/<name>/ directory; a batch_summary.json is written at the root.

5. Pre-run setup code (prepare_code)

A manifest may include Python snippets that run before the agent starts (e.g. to launch an app). Because they're exec()'d, the CLI requires explicit opt-in:

prepare_code:
  - inline: |
      import pyautogui, time
      pyautogui.hotkey('win'); time.sleep(1)
      pyautogui.write('notepad'); pyautogui.press('enter')
  - path: ./prepare/extra_setup.py
openapa run -m tasks/with_prepare.yaml --allow-prepare

Without --allow-prepare the run exits with code 2 and an explanatory message.

6. Useful flags reference

Flag Where Purpose
-c, --config run / batch / config Override config file path
-o, --log-output-dir run Where logs/screenshots go
-n, --max-steps run Cap agent iterations
-f, --task-file run Read task text from file (or - for stdin)
-m, --manifest run Structured YAML/JSON task spec
-p, --prepare run Extra prepare-code file (repeatable)
--allow-prepare run / batch Permit exec() of prepare code
--keep-output run / batch Don't wipe the output dir
--dry-run run / batch Resolve config + attachments, skip LLM
--only NAMES batch Only run named subtasks (csv)
--continue-on-error batch Keep going after a failed subtask

Directory Structure

open-apa/
├── agent_core/             # Core multi-agent framework
│   ├── agents/             # Agent implementations (Klick, Code, Grounding)
│   ├── cli/                # `openapa` CLI: argparse entry, commands, task loader, attachments
│   ├── common/             # Shared utilities and checkers
│   ├── lmms/               # Language Model API wrappers (OpenAI, Anthropic, Gemini)
│   ├── observer/           # Runtime and logging context
│   └── runner.py           # Library API: run_klick_agent_task end-to-end
├── benchmark_test/         # Benchmark integration scripts
│   └── osworld_setup/      # OSWorld evaluation runners
├── conf/                   # Configuration files
│   └── conf.yaml           # Model and runtime configurations
├── scripts/                # Execution entry points
│   └── klick_runner.py     # Main script to run OpenAPA tasks
├── tools/                  # Auxiliary tools and analyzers
│   └── osworld_replay/     # Scripts for replaying OSWorld trajectories
├── setup.py                # Package installation script
├── requirements.txt        # Core Python dependencies
└── requirements-cli.txt    # Optional CLI deps (attachment extractors, ruamel.yaml)

OSWorld Benchmark Performance

OpenAPA was evaluated on the OSWorld benchmark using a subset of 361 tasks (excluding Google Drive tasks). The core architecture achieved top-tier performance across a wide range of applications.

Models Evaluated:

  • Main Agent & Code Agent: gemini-3.1-pro-preview
  • Grounding Model: doubao-seed-1-8-251228

Overall Performance:

  • Total score: 282.8 / 361
  • Average score: 78.34%

Score Breakdown by Domain:

Domain Score Total Tasks
Chrome 32.96 46
GIMP 22.0 26
LibreOffice Calc 44.0 47
LibreOffice Impress 38.96 47
LibreOffice Writer 17.0 23
Multi Apps 61.14 93
OS 20.0 24
Thunderbird 12.0 15
VLC 15.75 17
VS Code 19.0 23

OSWorld Integration

OpenAPA integrates seamlessly with the OSWorld benchmark to evaluate computer-use agents in a realistic OS environment.

Reproduction Steps

1. Prepare OSWorld Environment First, ensure you have the OSWorld repository cloned and its Docker environment configured according to their official instructions.

git clone https://github.com/xlang-ai/OSWorld.git
cd OSWorld
# Follow OSWorld's setup instructions for Docker environments

2. Install OpenAPA

cd /path/to/open-apa
pip install -r requirements.txt
pip install -e .

3. Synchronize Integration Scripts Copy the customized OpenAPA execution scripts into the OSWorld root directory:

cp benchmark_test/osworld_setup/lib_run_single.py /path/to/OSWorld/
cp benchmark_test/osworld_setup/run.py /path/to/OSWorld/

4. Run the Evaluation Navigate to your OSWorld directory and execute the benchmark. Make sure to replace the API keys and endpoints with your own.

cd /path/to/OSWorld
python run.py \
  --provider_name docker \
  --headless \
  --num_envs 1 \
  --max_steps 100 \
  --domain "all" \
  --test_all_meta_path evaluation_examples/test_nogdrive.json \
  --model_provider "gemini" \
  --model "gemini-3.1-pro-preview" \
  --model_url "https://your-model-api-endpoint/v1" \
  --model_api_key "your_gemini_api_key" \
  --model_temperature 1.0 \
  --ground_provider "openai" \
  --ground_model "doubao-seed-1-8-251228" \
  --ground_url "https://your-grounding-api-endpoint/v3" \
  --ground_api_key "your_doubao_api_key" \
  --grounding_width 1000 \
  --grounding_height 1000 \
  --sleep_after_execution 3 \
  --result_dir "result-OpenAPA"

Note: The --grounding_width and --grounding_height arguments must be configured to match the optimal resolution required by the specific grounding model you are utilizing.

5. View Evaluation Results OpenAPA provides a dedicated web dashboard to analyze the evaluation trajectories, LLM reasoning processes, and token/time consumption.

cd /path/to/open-apa
python tools/osworld_result_viewer/app.py --result-dir /path/to/OSWorld/result-OpenAPA --port 8089

Once the server is running, open your browser and navigate to http://localhost:8089 to explore the detailed task playback and dashboard.

About

Open APA (Open Agentic Process Automation) is built on top of our first-place performance in the OSWorld Benchmark. It provides a practical and extensible framework for building computer use agents that can reliably execute long-horizon, real-world tasks across applications, interfaces, and workflows.

Resources

Readme
MIT license
Activity
Custom properties

Stars

33 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

Packages

Contributors

Languages