One prompt → a 3D character standing in front of you in VR, talking.
Describe a character. Claude writes a Blender Python script to build the body, picks a voice from ElevenLabs, generates dialogue, and the result drops into a WebXR scene you can walk into. Procedural 3D and voice, generated by an agent, end-to-end from a single text prompt.
Built for the DesignXR Hackathon.
Prereqs: Blender on PATH (brew install blender), Python 3.11+, an Anthropic API key, an ElevenLabs API key.
cp .env.example .env
# Edit .env — fill in ANTHROPIC_API_KEY, ELEVENLABS_API_KEY,
# and the five VOICE_* IDs from your ElevenLabs library.
pip install fastapi uvicorn anthropic httpx
cd server && uvicorn main:app --reload --port 8000Open http://localhost:8000, type something like "grumpy dwarf blacksmith with a long red beard", hit Generate. First generation takes ~20–30 seconds (Blender + ElevenLabs).
To list ElevenLabs voice IDs for the .env:
curl -H "xi-api-key: $ELEVENLABS_API_KEY" \
https://api.elevenlabs.io/v1/voices | jq '.voices[] | {voice_id, name}'User prompt
│
▼
Frontend (A-Frame, vanilla JS)
│ POST /api/generate
▼
FastAPI server (local, on the dev's M4)
│
▼
Claude agent loop (claude-opus-4-7, adaptive thinking, tool use)
├── build_3d_model(name, bpy_script)
│ └── blender --background --python → /models/{name}.glb
└── synthesize_voice_lines(name, voice_id, lines)
└── ElevenLabs TTS → /models/{name}_{line_id}.mp3
│
▼
A-Frame loads the .glb, attaches positional audio, plays the greeting.
Two tools, one agent turn. Claude orchestrates: receives traits → writes the bpy script → invokes the build → writes dialogue → invokes voice synth → returns. Bpy scripts are constrained to primitives only, ~2m tall, Y-up, exporting to a known path — see the system prompt in server/agent.py for the full constraint sheet.
| Layer | What |
|---|---|
| Frontend | A-Frame (WebXR/VR) + vanilla JS, no build step |
| Backend | FastAPI, runs locally on the dev's M4 |
| Agent | Anthropic SDK with tool use (@beta_tool decorator) |
| 3D | Blender headless (blender --background --python) |
| Voice | ElevenLabs TTS (eleven_turbo_v2_5) |
.
├── index.html # Single-page A-Frame frontend
├── server/
│ ├── main.py # FastAPI routes + static serving
│ └── agent.py # Claude agent loop + tool definitions
├── models/ # Generated .glb / .mp3 (gitignored)
├── assets/media/ # Logo and demo media
└── CLAUDE.md # Detailed project notes for AI collaborators
Hackathon project, scope-disciplined. The agent loop, frontend, and end-to-end fetch path are wired. Stretch features (regeneration, animated idle, multiple NPCs in one scene, conversational replies via Web Speech) are deliberately deferred — see CLAUDE.md for the build order.