Legion

Swarm robotics with Cyberbricks, orchestrated by Claude Code.

In this demo (4x speed, full video), I told the agent to steer Bot 1 to kick Baymax and then have Bot 2 shoot at Bot 1. The vision pipeline identified objects, tracked bot positions and orientations, and the agent reasoned over that data to coordinate both robots step by step.

About

Legion lets an AI coding agent control physical robots through natural language. You talk, the agent decides what to do, and sends commands to the bots over WiFi.

An important distinction: the agent doesn't use an LLM's built-in vision, which is too slow for real-time control. Instead, a custom vision pipeline turns a single camera frame into structured JSON (positions, headings, object labels, metric depth) in ~300ms, and the agent reasons over that. Any LLM can drive the bots, even ones with no vision support.

This was a weekend project, built over a couple of weekends. The inspiration was this video about giving Claude Code an RC car body, and I had a few CyberBrick kits lying around from backing their Kickstarter a while ago.

This is a collaboration between me and Claude Code. Most commits are co-authored by Claude. I haven't polished every edge case so there will be bugs. If you run into something please open an issue, I will try to look into it but I can't make any promises.

I know two bots can barely be called a swarm, but the full framework is there and adding more bots should be straightforward.

The architecture is agent-agnostic. The agent interacts with bots entirely through the legion CLI, so you could swap Claude Code for OpenCode or any agent that can run shell commands.

   Browser Mic       USB Webcam
        |                 |
        v                 v
  Speech-to-Text    Computer Vision
        |                 |
        v                 |
      Agent               |
        |                 |
        +--------+--------+
                 |
                 v
            legion CLI
                 |
                 v
               MQTT
                 |
                 v
              Robots

Vision pipeline

                   Camera Frame
                        |
           +------------+------------+
           |            |            |
           v            v            v
     ArUco Detect   YOLOE-26x   Depth Anything 3
     every frame    on demand      on demand
           |            |            |
           v            v            v
     Bot IDs +    4,585 class   Metric depth
     positions    detection     in meters
     headings
           |            |            |
           +------------+------------+
                        |
                        v
                 Scene State JSON

• ArUco markers on each robot, detected every frame (<1ms). Gives bot ID, position, and heading.
• YOLOE-26x object detection on demand (~80ms). 4,585 classes from the RAM++ tag set.
• Depth Anything 3 metric depth on demand (~250ms).

Voice goes through the browser mic to mlx-qwen3-asr (local STT on Apple Silicon, ~40ms).

Prerequisites

Hardware: One or more CyberBrick kits (SoccerBot/Tank tested), ArUco markers (4x4_50) on each bot, a USB webcam overhead, and WiFi.

Software: uv, Mosquitto, Claude Code, Arduino Lab for MicroPython.

Getting started

1. Install and clone

curl -LsSf https://astral.sh/uv/install.sh | sh  # install uv (manages Python for you)
brew install mosquitto claude-code                 # MQTT broker + Claude Code

git clone https://github.com/kessler-frost/legion.git && cd legion
uv sync                          # installs Python 3.12 and all deps
uv tool install --editable .     # makes `legion` available globally

Download the YOLOE-26x ONNX model (~260MB) into the project root:

curl -L -o yoloe-26x-seg-pf.onnx https://github.com/kessler-frost/legion/releases/download/v0.1.0/yoloe-26x-seg-pf.onnx

Pinned to Python 3.12 because open3d (a depth model dependency) doesn't support 3.13 yet.

2. Flash firmware to your bots

Use Arduino Lab for MicroPython:

1. Copy firmware/soccerbot/config.example.json to config.json, fill in your WiFi SSID, password, and computer's IP
2. Connect bot via USB-C, upload boot.py, main.py, and config.json

Test in the REPL:

import machine
machine.reset()  # restarts the bot, runs boot.py then main.py

3. Run

mosquitto -d          # start MQTT broker
legion serve start    # start server + vision + agent

Open localhost:8000/command and start talking.

CLI reference

legion serve start [--bg]       # start server (--bg for background)
legion serve stop               # stop background server

legion forward <id> <duration>  # drive forward
legion backward <id> <duration> # drive backward
legion left <id> <duration>     # spin left
legion right <id> <duration>    # spin right
legion kick <id> <duration>     # kicker servo (Bot 1)
legion shoot <id> <duration>    # shooter servo (Bot 2)
legion stop <id>                # emergency stop

legion scene state              # bot positions + headings (instant)
legion scene state --objects    # + object detection (~80ms)
legion scene state --depth      # + depth estimation (~250ms)
legion scene state --full       # everything (~300ms)

Web UI

URL	What
/	Home
/command	Command center: camera + agent chat + voice
/control	Manual joystick control
/docs	API docs

Adding a new bot

1. Print an ArUco marker (4x4_50) and stick it on top of a CyberBrick kit
2. Copy an existing firmware folder, set a new bot_id in config.example.json
3. Add motor calibration + marker mapping in brain/cli.py
4. Flash via Arduino Lab for MicroPython

Resources and inspirations

• Claude Code RC Car (YouTube), the inspiration for this project
• CyberBrick Official Repo
• CyberBrick API Docs
• mlx-qwen3-asr, the local speech-to-text model used for voice commands

License

Apache License 2.0