SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization

🔥 Overview

We introduce SKILL0, an in-context reinforcement learning framework designed for skill internalization.

SKILL0 achieves substantial improvements over the standard RL baseline on ALFWorld and Search-QA.

🗞️ News

• 2026-8-6: 🔥🔥 We released AgentOPSD, introducing recursive credit update for SDAR.
• 2026-7-29: 🔥🔥 We released SkillRise, introducing cross-task skill evolution via agentic RL.
• 2026-7-17: 🔥🔥 We released SEED, introducing self-evolving opd beyond skill internalization.
• 2026-6-25: 🔥 We released OPID, introducing skill evolving beyond skill internalization.
• 2026-5-15: 🔥 Our new work was released: SDAR, which introduces Self-Distilled Agentic Reinforcement Learning for skill internalization.
• 2026-5-07: 🔥 Our new work was released: SKILL1, which evloves skill-augmented agents in one unified policy.
• 2026-4-03: We release our paper and code.

🛠️ Installation

Python environment

conda create -n skillzero python=3.12 -y
conda activate skillzero

pip install vllm==0.10.0
pip install flash-attn==2.7.4.post1 --no-build-isolation --no-cache-dir
pip install -e .

Log in to Weights & Biases if you use WandB logging (scripts pass trainer.logger=['console','wandb'] in many cases):

export WANDB_API_KEY=your_key_here

Install Supported Environments

1. ALFWorld

Install with pip:

pip3 install gymnasium==0.29.1
pip3 install stable-baselines3==2.6.0
pip3 install alfworld

Download PDDL & Game files and pre-trained MaskRCNN detector (will be stored in ~/.cache/alfworld/):

alfworld-download -f

2. Search

cd ./agent_system/environments/env_package/search/third_party
pip install -e .
pip install gym==0.26.2

Prepare dataset (data will be saved at ~/data/searchR1_processed_direct):

cd repo_root/
python examples/data_preprocess/preprocess_search_r1_dataset.py

Build Retriever environments:

conda create -n retriever python=3.10 -y
conda activate retriever

conda install numpy==1.26.4 
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124

pip install transformers datasets pyserini huggingface_hub
conda install faiss-gpu==1.8.0 -c pytorch -c nvidia -y
pip install uvicorn fastapi

Download the index:

conda activate retriever

local_dir=~/data/searchR1
python examples/search/searchr1_download.py --local_dir $local_dir
cat $local_dir/part_* > $local_dir/e5_Flat.index
gzip -d $local_dir/wiki-18.jsonl.gz

Start the local flat e5 retrieval server:

conda activate retriever

# redirect the output to a file to avoid cluttering the terminal
# we have observed outputting to the terminal causing spikes in server response times
bash examples/search/retriever/retrieval_launch.sh > retrieval_server.log 

Validation parquet for SkillZero Search

python -m examples.data_preprocess.generate_search_r1_val

Training

All scripts live under scripts/ and assume the repo root as working directory (they cd there automatically). You can run either:

bash scripts/train_alfworld_skillzero_3b.sh
bash scripts/train_search_skillzero_3b

### Merge checkpoints

See `scripts/model_merger.py` for FSDP/Megatron merge examples using paths under `./checkpoints/...`.

⭐️ Citation

If you find this project useful, welcome to cite us.

@article{lu2026skill0,
  title={Skill0: In-context agentic reinforcement learning for skill internalization},
  author={Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Han, Chengcheng and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and Shen, Yongliang},
  journal={arXiv preprint arXiv:2604.02268},
  year={2026}
}
@article{lu2026sdar,
  title={Self-distilled agentic reinforcement learning},
  author={Lu, Zhengxi and Yao, Zhiyuan and Han, Zhuowen and Wang, Zi-Han and Wu, Jinyang and Gu, Qi and Cai, Xunliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting and others},
  journal={arXiv preprint arXiv:2605.15155},
  year={2026}
}
@article{shi2026skill1,
  title={Skill1: Unified evolution of skill-augmented agents via reinforcement learning},
  author={Shi, Yaorui and Chen, Yuxin and Lu, Zhengxi and Miao, Yuchun and Liu, Shugui and Gu, Qi and Cai, Xunliang and Wang, Xiang and Zhang, An},
  journal={arXiv preprint arXiv:2605.06130},
  year={2026}
}
@article{wu2026seed,
  title={SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning},
  author={Wu, Jinyang and Yang, Shuo and Lu, Zhengxi and Zhang, Fan and Shen, Yuhao and Feng, Lang and Luo, Haoran and Lian, Zheng and Zhang, Shuai and Wen, Zhengqi and others},
  journal={arXiv preprint arXiv:2607.14777},
  year={2026}
}
@article{yang2026opid,
  title={Opid: On-policy skill distillation for agentic reinforcement learning},
  author={Yang, Shuo and Wu, Jinyang and Lu, Zhengxi and Shen, Yuhao and Zhang, Fan and Feng, Lang and Zhang, Shuai and Luo, Haoran and Lian, Zheng and Wen, Zhengqi and others},
  journal={arXiv preprint arXiv:2606.26790},
  year={2026}
}
@article{yao2026skillrise,
  title={SkillRise: Agentic Reinforcement Learning for Cross-Task Skill Evolution},
  author={Yao, Zhiyuan and Chen, Yuxin and Lu, Zhengxi and Xu, Zishan and Sun, Yueqing and Guo, Yifu and Lu, Yuquan and Cai, Zhengzhou and Zhang, Kangning and Han, Zhuowen and others},
  journal={arXiv preprint arXiv:2607.26784},
  year={2026}
}
@article{wang2026agentopsd,
  title={AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning},
  author={Wang, Zi-Han and Lu, Zhengxi and Yao, Zhiyuan and Wu, Jinyang and Wu, Jie and Cai, Zhengzhou and Sun, Yueqing and Ye, Ziang and Hao, Linji and Gu, Qi and others},
  journal={arXiv preprint arXiv:2608.05987},
  year={2026}
}

🤝 Acknowledgement

This project builds on AgentOCR, verl-agent, veRL, ALFWorld, SkillRL, and Search-R1. We thank the authors of those projects.