Cognitive-Physical Decoupling — A Session-Centered Runtime for Embodied Intelligence

_{English · 中文}

---

📢 Changelog

Version	Date	Update
	2026-05-29	Based on — Minecraft ready: cloud agent connects to user's local server
	2026-05-25	Strict separation of PolicySkillRuntime / BuiltinSkillRuntime; Game Agent & Benchmarking ready
	2026-05-20	Perception plugin system: SensorConfig / PerceptionConfig YAML + EnvironmentWriter auditable writeback
	2026-05-18	Session-Centered Runtime MVP: DummySimTarget + DummyAdapter + DummyClient serial pipeline
	2026-04-29	Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline

---

🤔 Why PhyAgentOS?

Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:

🔌	One Codebase, Any Hardware — Adding a new robot means implementing one Target Adapter (~100 lines); zero changes to the scheduling layer.
🛡️	Three Safety Layers — Critic validation → Strict Preflight → Target-side SafetyGuard; mandatory for real-robot deployment.
📋	Fully Auditable — State, actions, and perception results are written to Markdown + YAML files; every step is traceable and reproducible.
🔄	Zero-Friction Migration — The same Session protocol runs identically across sim, real, and game targets.

_{▲ Session-Centered Runtime Architecture Overview}

---

✨ Core Features

🔄	Session-Centered Runtime	WatchdogSupervisor → SessionRunner → SkillRuntime → TargetSessionHandle execution pipeline, replacing the legacy Driver-Center architecture
🎯	Target-Configured	Four target kinds — game / debug / simulation / real_robot — registered in TARGETS.md, adapters attached on demand
🧩	Adapter + Bridge	TargetAdapter + PolicyAdapter + ActionBridge three-way decoupling; AdapterPlan auto-composed, eliminating target×skill combinatorial explosion
⚡	Dual Skill Runtimes	PolicySkillRuntime maintains policy closed-loop + BuiltinSkillRuntime manages agent interactive loop
🛡️	Strict Preflight	10 validation checks (target / sensor / perception / contract / tool); failures are rejected before execution starts
📝	File Protocol Matrix	TARGETS.md · SKILLS.md · SESSIONS.md · ENVIRONMENT.md · LESSONS.md + external YAML configs
🔐	Multi-Layer Safety	Critic validation → Preflight contract checks → Target-side SafetyGuard → Operator Override
🌐	Fleet Mode	Multi-robot coordination with shared + per-robot workspaces, priority-based serial scheduling

---

🚀 5-Minute Quick Start

1	Install git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS pip install -e . # Python ≥ 3.11 pip install -e ".[dev]" # Dev dependencies
2	Initialize Workspace paos onboard
3	Terminal 1: Start Runtime (Track B) python -m PhyAgentOS.runtime.watchdog
4	Terminal 2: Start Agent (Track A) paos agent

Enter natural language commands in the Agent CLI to drive hardware. No hardware? Run the Smoke Test to verify the full pipeline:

python scripts/init_runtime_workspace.py --workspace /tmp/paos_runtime_smoke
python scripts/run_runtime_watchdog.py --workspace /tmp/paos_runtime_smoke --once
# → session marked succeeded, results written to artifacts/

---

📦 Project Structure

PhyAgentOS/
│
├── PhyAgentOS/agent/          # Track A  ─  Planner / Critic / Memory
│
├── PhyAgentOS/runtime/        # Track B  ─  Execution Plane
│   ├── watchdog/              #   WatchdogSupervisor
│   ├── sessions/              #   SessionRunner / TargetSessionHandle
│   ├── targets/               #   RolloutTarget (game·debug·sim·real)
│   ├── skills/                #   PolicySkillRuntime / BuiltinSkillRuntime
│   ├── adapters/              #   TargetAdapter / PolicyAdapter / Bridge
│   ├── perception/            #   Perception Runtime / EnvironmentWriter
│   ├── preflight/             #   RuntimeCompatibilityPreflight
│   └── schemas/               #   Pydantic Schema
│
├── configs/runtime/           # Sensor / Perception / Contract YAML
├── scripts/                   # Utility scripts
├── workspace/                 # Runtime workspace
├── docs/                      # Documentation
└── tests/                     # Tests

---

🏷️ Supported Targets

	Kind	Location	Examples
🎮	game	Local	Minecraft, Stardew Valley — low-cost validation of long-term decisions & memory
🐛	debug	Local	echo / mock / dry-run — zero-hardware protocol pipeline validation
🧪	simulation	Remote	RoboCasa, LIBERO — benchmark evaluation & batch experience mining
🤖	real_robot	Remote	Franka, Go2, XLeRobot, AgileX PIPER — real-world deployment

All targets are registered in TARGETS.md, identified by target_adapter:// URI. More examples & demos → Project Website

---

📖 Documentation

Document	Audience	Description
🌐 Website	Everyone	Full docs, architecture details, demos
📘 User Manual	Users	Installation, deployment, and operation guide
📙 Dev Guide	Developers	Secondary development, hardware integration, plugin authoring

---

🤝 Contributing

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

---

Built on nanobot

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory

       
_{MIT License · Copyright © 2025-2026 PhyAgentOS}