_{RPent: Agentic Infrastructure for the Physical World}

RPent (Recursive Physical Agent) is an open framework for building embodied agents that continuously evolve through recursive interaction with the physical world. Rather than prescribing a single foundation model, RPent provides a recursive agent framework that harnesses heterogeneous intelligence, including perception, reasoning, memory, execution, and self-evolution, into a unified physical agent. Through continuous interaction, reflection, and adaptation, RPent enables physical agents to acquire new capabilities and evolve beyond their initial design.

RPent is built upon three core design principles: service-oriented, standardized, and composable. RPent enables capabilities to be deployed as reusable services, connected through unified interfaces, and flexibly composed into diverse physical agents. Together, these principles allow RPent to move beyond traditional robot control frameworks and establish an agentic infrastructure for the physical world, where intelligence is not only deployed, but continuously built, expanded, and evolved.

What's NEW!

• [2026/07] 🔥 Our first RPent publication, Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, is released.

Feature Matrix

Agentic Planner	Action Primitive	Simulator	Real World
Claude Code ✅ Codex ✅ Custom planner ✅	VLA manipulation Pi0.5 ✅ RLDX-1 WAM manipulation DreamZero	LIBERO-PRO ✅ RoboCasa	Franka SO-101

Quick Start

RPent runs on top of a forked branch of RLinf for the simulators and VLA models. Clone them side by side.

1. Clone RLinf and RPent side by side.

mkdir workspace && cd workspace
# RPent depends on a forked branch of RLinf; it will be merged back to main after more iterations.
git clone https://github.com/jx-qiu/RLinf -b feature/physicalagent rlinf
git clone https://github.com/RLinf/RPent rpent

2. In RLinf, create an openpi + LIBERO virtualenv.

cd rlinf
bash requirements/install.sh embodied --env libero --model openpi --use-mirror --venv ../.venv-opi-libero
cd ..
source .venv-opi-libero/bin/activate

3. Install RPent's extra dependencies on top of that venv.

cd rpent
uv sync --active --inexact
bash scripts/install_libero_pro_plus.sh

4. Configure keys and checkpoints, then run.

# LLM API keys (the `api` cerebrum)
export ANTHROPIC_BASE_URL=https://xxx
export ANTHROPIC_API_KEY=sk-xxx
export OPENAI_BASE_URL=https://xxx
export OPENAI_API_KEY=sk-xxx

# VLA checkpoint — download from
# https://huggingface.co/datasets/RLinf/rlinf-pi05-libero-130-fullshot-sft
export PI05_CHECKPOINT_PATH=/path/to/rlinf-pi05-libero-130-fullshot-sft
export LIBERO_TYPE=pro
export CUDA_VISIBLE_DEVICES=0

# Run one task: libero_object_swap, task 2, seed 0, using the `api` cerebrum
# with an Anthropic model and an 8192-token cap.
#   • OpenAI-compatible chat endpoints:  --model openai-chat:glm-5.2
#   • OpenAI responses endpoints:        --model openai:gpt-5.5
#   • claude_code / codex cerebrums:     no provider prefix, e.g. --model claude-opus-4-8
python rpent/cli/main.py --suite libero_object_swap --task 2 --seed 0 \
  --cerebrum api --model anthropic:claude-opus-4-8 --max-tokens 8192

Live Dashboard

Add --dashboard to open a browser monitor for the run. It boots a launcher screen where you pick the config, then streams reasoning, live views, and the action timeline. Use --dashboard-language zh-cn for the Chinese UI.

python rpent/cli/main.py --dashboard --dashboard-language zh-cn \
  --suite libero_goal_task --task 1 --seed 0 --cerebrum claude_code

RoboCasa

RoboCasa uses a separate entrypoint and setup guide.

bash scripts/setup_robocasa.sh                                # one-time setup
bash scripts/run_robocasa.sh PickPlaceCounterToCabinet 0 0    # <task> <gpu> <seed>

See SETUP_ROBOCASA.zh.md for the full RoboCasa365 + RLDX-1 walkthrough.

Key CLI Options

Flag	Default	Description
--suite	— (required)	Task suite, e.g. libero_object_task, libero_spatial_swap
--task	— (required)	Task id within the suite
--seed	0	Random seed
--cerebrum	api	Reasoning brain: api | claude_code | codex
--model	—	Model id; for api, prefix the provider (anthropic:…, openai:…, openai-chat:…)
--max-turns	100	Max agent turns
--max-tokens	8192	Max tokens per LLM reply
--max-episode-steps	10000	Max env steps
--libero-type	LIBERO_TYPE or pro	LIBERO variant: standard | pro | plus
--cuda-device	inherited	GPU device(s) exposed to the env / vla servers
--dashboard	off	Start the local dashboard for this run
--dashboard-language	en	Dashboard UI language: en | zh-cn
--vla-endpoint	—	Reuse an already-running vla_server instead of spawning one
--no-driver	off	Attach to an existing env_server / vla_server

Documentation

• Adding a new environment — plug a new simulator / robot into the runner (中文).
• RoboCasa setup — RoboCasa365 + RLDX-1 install and run guide.
• docs/ — local Sphinx build and preview instructions.

Citation and Acknowledgement

If you find RPent or Harness VLA helpful, please cite the paper:

@article{zhang2026harnessvla,
  title={Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents},
  author={Zhang, Yixian and Zhang, Huanming and Gao, Feng and Li, Xiao and Liu, Zhihao and Zhu, Chunyang and Qiu, Jiaxing and Yan, Yuchen and Liu, Jiyuan and Tang, Wenhao and Fang, Zhengru and Nie, Yi and Wei, Changxu and Wang, Yu and Ding, Wenbo and Yu, Chao},
  journal={arXiv preprint arXiv:2607.08448},
  year={2026},
  url={https://arxiv.org/abs/2607.08448}
}

RPent builds on the simulators, VLA models, and training infrastructure of RLinf, and on the agent SDKs of the broader open-source community — pydantic-ai, the Claude Agent SDK, and the OpenAI Codex SDK. Thanks to the teams behind LIBERO, RoboCasa, robosuite, MuJoCo, and openpi.