Whiteboard AI

A collaborative drawing tool for humans and AI agents. Draw with a browser UI, interact via REST/WebSocket, or describe what to draw in plain English from the command line.

Quick Start

# 1. Install dependencies
pip install -r requirements.txt

# 2. Copy and fill in .env
cp .env.example .env

# 3. Start the server
uvicorn server.main:app --reload

# 4. Open the browser UI
open http://localhost:8000

For the AI drawing CLI:

# OpenAI (default)
python -m cli.draw "draw a red house with a blue door"

# Anthropic
python -m cli.draw --model claude-opus-4-7 "draw a red house with a blue door"

Project Structure

whiteboard_ai/
├── server/                  # FastAPI backend
│   ├── main.py              # App entry point, static file serving
│   ├── config.py            # Env-var settings (loads .env)
│   ├── database.py          # SQLite schema & FastAPI dependency
│   ├── auth.py              # JWT issue / verify
│   ├── models.py            # Pydantic request / response models
│   ├── pubsub.py            # In-memory pub/sub for WebSocket broadcast
│   ├── locks.py             # Per-whiteboard asyncio.Lock for seq writes
│   ├── errors.py            # Typed HTTP error helpers
│   ├── db_helpers.py        # Shared whiteboard fetch + visibility check
│   └── routers/
│       ├── auth.py          # POST /auth/token
│       ├── whiteboards.py   # CRUD: create, list, get, patch, delete
│       ├── events.py        # GET/POST events + GET snapshot
│       └── ws.py            # WebSocket (replay, broadcast, ping/pong)
├── frontend/                # Browser UI (vanilla HTML/CSS/JS, no build step)
│   ├── index.html
│   ├── style.css
│   └── app.js
├── cli/                     # AI drawing CLI
│   └── draw.py
├── SPEC.md                  # Full technical specification
├── requirements.txt
├── .env.example
└── .env                     # your local config (not committed)

Configuration

Copy .env.example to .env:

SECRET_KEY=change-me-to-a-random-secret
DATABASE_PATH=whiteboard.db

# AI drawing CLI — set whichever provider(s) you want to use
OPENAI_API_KEY=
ANTHROPIC_API_KEY=

# Optional overrides
# WB_MODEL=gpt-4o
# WB_URL=http://localhost:8000
# WB_CLIENT_ID=draw-cli

Env var	Default	Description
`SECRET_KEY`	dev secret	JWT signing key — change in prod
`DATABASE_PATH`	`whiteboard.db`	SQLite file path
`OPENAI_API_KEY`	—	Required when using OpenAI models
`ANTHROPIC_API_KEY`	—	Required when using Anthropic models
`WB_MODEL`	`gpt-4o`	Default model for the CLI
`WB_URL`	`http://localhost:8000`	Whiteboard server URL (used by the CLI)
`WB_CLIENT_ID`	`draw-cli`	Client ID for CLI-issued tokens

Browser UI

Open http://localhost:8000 after starting the server.

Tools

Key	Tool
`P`	Pen
`M`	Marker (semi-transparent)
`E`	Eraser
`R`	Rectangle
`O`	Ellipse
`L`	Line
`A`	Arrow
`T`	Text
`S`	Select — click to select, drag to move, `Delete` to remove
`H` / `Space+drag`	Pan

Keyboard shortcuts

Shortcut	Action
`Ctrl+Z` / `Cmd+Z`	Undo
`Ctrl+Y` / `Cmd+Y` / `Ctrl+Shift+Z`	Redo
`Ctrl+A` / `Cmd+A`	Select all elements
`Delete` / `Backspace`	Delete selected elements
`Shift+click`	Add to selection
Scroll wheel	Zoom in / out
`Ctrl+0`	Fit all to screen
`Ctrl+` `+` / `-`	Zoom in / out

Select tool

Click an element to select it; selected elements show a dashed blue outline with corner handles.
Shift+click to add to the current selection.
Drag a selected element to move it; the move is synced to all connected clients.
Delete / Backspace removes all selected elements.
Ctrl+A selects everything on the board.

Undo / Redo

Undo and redo are infinite and track three action types:

Action	Undo	Redo
Draw (pen, shape, text)	Removes the element	Restores it
Delete	Restores deleted elements	Deletes them again
Move	Moves back by the same amount	Moves forward again

All undo/redo operations are synced to the server so other connected clients see the changes immediately.

Board management

Click New to create a board or Open to load one by ID or slug. The URL hash updates to /#<slug> so you can share or bookmark the board directly.

AI Drawing CLI

The CLI accepts a plain-English drawing prompt, sends it to an AI model, which generates Python code that calls the whiteboard REST API, and then executes that code locally. Because it uses the same API as the browser, the canvas updates in real time via WebSocket while the CLI runs.

Provider selection

The provider is inferred automatically from the model name:

Model name	Provider	Key required
`gpt-4o`, `o4-mini`, …	OpenAI	`OPENAI_API_KEY`
`claude-opus-4-7`, …	Anthropic	`ANTHROPIC_API_KEY`

Single-shot

# Creates a new board automatically, uses gpt-4o by default
python -m cli.draw "draw a sunset over the ocean with a sailboat"

# Use an existing board
python -m cli.draw --board my-board "add a lighthouse on the right"

# Use a specific model
python -m cli.draw --model gpt-4o-mini "draw a star"
python -m cli.draw --model claude-opus-4-7 "draw a house"

# Print generated code before running
python -m cli.draw --verbose "draw a mountain"

Interactive REPL

python -m cli.draw --board my-board
draw> draw a mountain range with snow
draw> add a pine forest in the foreground
draw> put a small cabin near the trees
draw> :verbose          # toggle generated code display
draw> :clear            # wipe the board
draw> :snapshot         # show current element list
draw> :url              # print the board URL
draw> :quit

Options

--board  -b   Board ID or slug (creates a new board if omitted)
--url    -u   Server URL (default: $WB_URL or http://localhost:8000)
--model  -m   Model name (default: $WB_MODEL or gpt-4o)
--verbose -v  Print generated code before executing

How it works

Fetches the current board snapshot so the model knows what's already drawn.
Calls the model with a detailed system prompt describing the whiteboard API schemas and coordinate system. The system prompt is prompt-cached (Anthropic) on repeated calls in the same REPL session.
Streams the response; shows a progress indicator while the model generates code.
exec()s the generated code in a sandboxed namespace with requests, math, uuid, json, random, and colorsys pre-imported, plus API, BOARD_ID, TOKEN, and HEADERS pre-defined.
The generated code posts element.create (or other) events to the server. Other clients see the changes instantly via WebSocket.

REST API

Base URL: http://localhost:8000/api/v1

All requests except token issuance require:

Authorization: Bearer <token>

Auth

Method	Path	Description
`POST`	`/auth/token`	Issue a JWT for a `client_id`

curl -X POST http://localhost:8000/api/v1/auth/token \
  -H 'Content-Type: application/json' \
  -d '{"client_id": "my-agent"}'

Whiteboards

Method	Path	Description
`POST`	`/whiteboards`	Create a whiteboard
`GET`	`/whiteboards`	List accessible boards
`GET`	`/whiteboards/:id`	Get board (ID or slug)
`PATCH`	`/whiteboards/:id`	Update title / visibility
`DELETE`	`/whiteboards/:id`	Soft-delete a board

Events & Snapshot

Method	Path	Description
`GET`	`/whiteboards/:id/snapshot`	Current canvas state (all elements)
`GET`	`/whiteboards/:id/events`	Event log with filtering
`POST`	`/whiteboards/:id/events`	Append events (draw / modify canvas)

Event log query params:

Param	Description
`since_seq`	Return events with `seq > N` (AI agent polling)
`since`	Return events with `ts >= ISO timestamp`
`until_seq`	Upper bound (inclusive)
`until`	Upper bound timestamp
`types`	Comma-separated event type filter
`client_id`	Filter by author
`limit`	Max results (default 100, max 1000)
`cursor`	Opaque pagination cursor from previous response

WebSocket

WS /whiteboards/:id/ws?token=<jwt>

Handshake (client → server after connect):

{ "type": "subscribe", "since_seq": 42 }

Server replays missed events, then streams live.

Server → client:

{ "type": "event",       "event": { "seq": 1, "type": "element.create", ... } }
{ "type": "cursor.move", "client_id": "...", "x": 120.5, "y": 88.3 }
{ "type": "ping" }

Client → server:

{ "type": "pong" }
{ "type": "events", "events": [ { "type": "element.create", "payload": {...} } ] }
{ "type": "cursor.move", "x": 120.5, "y": 88.3 }

Element Schemas

Stroke (freehand path)

{
  "element_id": "el-abc123",
  "type": "stroke",
  "z_index": 1,
  "data": {
    "points":  [[0,0],[10,10],[20,5]],
    "color":   "#e03131",
    "width":   3,
    "opacity": 1.0,
    "tool":    "pen"
  }
}

Shape

{
  "element_id": "el-def456",
  "type": "shape",
  "z_index": 2,
  "data": {
    "kind":  "rect",
    "x": 100, "y": 100, "w": 200, "h": 120,
    "color": "#1971c2",
    "style": { "stroke": "#1971c2", "fill": "transparent", "width": 2 }
  }
}

Supported kind values: rect, ellipse, line, arrow.

Text

{
  "element_id": "el-ghi789",
  "type": "text",
  "z_index": 3,
  "data": {
    "x": 50, "y": 50,
    "content": "Hello World",
    "size":    18,
    "color":   "#1e1e1e",
    "font":    "sans-serif",
    "bold":    false,
    "italic":  false
  }
}

Event Types

Type	Description	Persisted
`element.create`	Add a new element	Yes
`element.update`	Partial patch of an existing element	Yes
`element.delete`	Soft-delete one or more elements	Yes
`element.move`	Translate elements by `dx`, `dy`	Yes
`element.reorder`	Change `z_index` of elements	Yes
`canvas.pan`	Viewport hint (excluded from snapshot)	Yes
`canvas.resize`	Viewport size hint (excluded from snapshot)	Yes
`whiteboard.update`	Metadata change (title, visibility)	Yes
`cursor.move`	Real-time cursor broadcast — WebSocket only	No

AI Agent Workflow

Any HTTP client can draw programmatically using the same API:

import requests

BASE  = "http://localhost:8000/api/v1"
TOKEN = requests.post(f"{BASE}/auth/token", json={"client_id": "my-agent"}).json()["token"]
HDR   = {"Authorization": f"Bearer {TOKEN}", "Content-Type": "application/json"}

# 1. Create or load a board
wb = requests.post(f"{BASE}/whiteboards",
                   headers=HDR, json={"title": "Agent Board", "visibility": "link-shared"}).json()
board_id = wb["id"]

# 2. Read current state
snap = requests.get(f"{BASE}/whiteboards/{board_id}/snapshot", headers=HDR).json()
last_seq = snap["seq"]

# 3. Draw
requests.post(f"{BASE}/whiteboards/{board_id}/events", headers=HDR, json={"events": [
    {
        "type": "element.create",
        "payload": {
            "element_id": "my-circle",
            "type": "shape",
            "z_index": 1,
            "data": {
                "kind": "ellipse", "x": 350, "y": 250, "w": 100, "h": 100,
                "color": "#2f9e44",
                "style": {"stroke": "#2f9e44", "fill": "#b2f2bb", "width": 2}
            }
        }
    }
]})

# 4. Poll for new changes
changes = requests.get(f"{BASE}/whiteboards/{board_id}/events",
                       headers=HDR, params={"since_seq": last_seq}).json()

Tech Stack

Layer	Technology
Server	Python 3.10+, FastAPI, uvicorn
Storage	SQLite (WAL mode) via aiosqlite
Auth	JWT (PyJWT, HS256)
Frontend	Vanilla HTML / CSS / JS (no build step)
AI CLI	OpenAI SDK + Anthropic SDK

Name		Name	Last commit message	Last commit date
Latest commit History 9 Commits
cli		cli
frontend		frontend
server		server
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
SPEC.md		SPEC.md
raw_prompt.txt		raw_prompt.txt
requirements.txt		requirements.txt
whiteboard.db		whiteboard.db

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Whiteboard AI

Quick Start

Project Structure

Configuration

Browser UI

Tools

Keyboard shortcuts

Select tool

Undo / Redo

Board management

AI Drawing CLI

Provider selection

Single-shot

Interactive REPL

Options

How it works

REST API

Auth

Whiteboards

Events & Snapshot

WebSocket

Element Schemas

Stroke (freehand path)

Shape

Text

Event Types

AI Agent Workflow

Tech Stack

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages