A living build log. Last updated: 2026-06-30.
This is the story of how an image from a movie turned into a small system for running and supervising a whole fleet of Claude Code instances, and where we plan to take it next.
The constraint that shaped all of it: I live in tmux. I'm not trying to replace my laptop. I just want the added ability to talk to the tmux panes already running on my server when I'm away from my computer. No Anthropic API key to wire up, no separate agent platform to stand up. I just want to talk to the terminals that are already there.
The idea didn't begin in a terminal. It began watching Body of Lies.
Russell Crowe's character runs a field operation from his cell phone while living an utterly ordinary day: driving the kids around, handling domestic errands, half his attention on suburban life and the other half directing something complex and consequential from his pocket. That image stuck with me: the untethered freedom of steering real work from anywhere, without being chained to a desk.
I wanted that for Claude. I already run Claude Code in terminals on my laptop all day, and increasingly I drive it with voice, which is a natural fit for describing what you want.
The interface came before the architecture. I took a screenshot of the iPhone Voice Memos app and sent it to a friend: this, for managing agents on my phone (each session a "recording" you can tap to play back), would be so much better than hunching over an SSH session in Termius. From there the picture got richer, borrowing from Snapchat: a feed where each conversation is an agent, and the big capture button in the middle (the one that normally takes a photo) is instead push-to-talk to the manager. You'd talk to your fleet the way you'd snap a photo.
So, fittingly, the project started away from the desk. I took my laptop with me
on a McDonald's run and, between ordinary tasks, just dictated the idea to
Claude, talking through the vision out loud and letting it capture the first
DESIGN.md. The medium was the message: the whole point was to operate hands-busy,
on the move, by voice.
The technical entry point was a simple question: what hooks can you write in
Claude Code?
There's a Stop hook. What else? The answer turned out to be a rich set of
lifecycle events: PreToolUse, PostToolUse, Notification, Stop,
SessionStart, SessionEnd, and more. Each one is a place to hang your own
behaviour.
That led to the next thought: if Claude can emit events about what it's doing,
could something watch a Claude session and tell you, in plain language, what's
happening? The first experiment was tiny: a shell script, watch-pane.sh, that
peeks at a tmux pane with capture-pane, pipes the text into a cheap Claude
model, and prints a one-line summary on a loop. Crude, but it worked. You could
point it at a pane and get a running narration of what that session was up to.
Two lessons fell out of that prototype almost immediately:
- Scrape the rendered screen, not the raw stream.
tmux capture-panegives clean text;pipe-panegives a firehose of escape codes. The same distinction would matter later. - The pane is data, not instructions. A summariser pointed at a Claude session is reading a screen full of natural language, including text that looks like commands. The prompt had to be hardened to treat the pane strictly as something to describe, never to obey. (Prompt injection, discovered the fun way.)
Once the watcher worked, the upgrade was easy. The same script that read a pane could also write to one. Turning the watcher into a manager meant it could send keys into a specific pane and answer the approval prompts and questions a session throws up: not just narrate what was happening, but actually drive it.
The first sketch had the manager be a Claude session in a tmux pane that you typed into. A dead end: the phone app will never want to type into a terminal and scrape the reply.
So we made the manager a plain command that wraps a single claude -p call: it
classifies the request, resolves the target session, then dispatches the
instruction. A simple but durable core that runs the whole thing.
Once we fixed all the bugs, the whole back end worked end-to-end. The backend was done; now we had to face the UI.
The app is SwiftUI. I've never written a line of Swift in my life and have no idea how any of it works. So I gave Claude a screenshot of the iOS Voice Memos record screen and told it: build this, where each recording is an agent session and the red button is push-to-talk to the manager. The list is that metaphor made literal: each agent a row, hold the red button to speak. You talk to it by voice, and the manager's answer comes back spoken aloud, not popped up in a dialog. Voice in, voice out.
The first real architectural decision of the app phase was: how does the phone reach the fleet?
The obvious answer, type in a server URL, is the wrong one. It means exposing the daemon on a public port, managing auth, and a setup step nobody wants. The box already runs the user's Claude, already has their workspace, already has them logged in. So instead of a URL, the app asks for what you'd type into Termius: a username, an IP, and a project folder. It SSHes in, and if the fleet isn't installed it uploads the scripts and starts the server itself. The first connection bootstraps the entire back end.
That left one genuinely hard part. The server binds to 127.0.0.1 on the box and
is never exposed, so how does the phone talk to it? We built an in-app
port-forward: the ssh -L you'd run by hand, but inside the app. A local TCP
listener on the phone bridges each connection to an SSH direct-tcpip channel to
the box's localhost, glued together with a hand-written NIO handler. The subtle
constraint: the SSH client and the local listener must live on one shared event
loop, or the glue between them is unsafe. It compiled long before it ran, and
the first time it ran, the phone was talking to a server that was never on the
network. I still don't really know anything about direct-tcpip channels or NIO
handlers, but luckily Claude does.
The early app showed each agent as raw terminal text. It worked, but it read like reading a log. The reframe that made it click: an agent is a chat thread. The agent's summaries are incoming messages; your voice (and your tapped choices) are outgoing ones. iMessage for your fleet, backed by a per-agent message log on the box.
The approval flow got the same treatment. When a worker stops at one of Claude's interactive menus, instead of dumping you into the terminal, the app parses the menu out of the screen and renders the options as tappable buttons: pick one and it drives the menu for you, and records your choice in the thread. The terminal is still one tap away for anything we can't parse.
Two smaller honesties from this stretch:
-
Reconnect on return. iOS suspends a backgrounded app and tears down the SSH socket; coming back, every action failed with "not connected." We reconnect on foreground, and learned that reusing the NIO event-loop group after suspension doesn't work; you have to rebuild on a fresh one.
-
Plumbing you can feel. When you spoke to an agent, your own message took a beat too long to appear. The prompt rewriter (an extra reasoning call) was running before the message was even logged, so the bubble waited on it. Invisible plumbing is only invisible if it doesn't block the thing right in front of you. We log your words first, then rewrite.
Once we started using the app through an ordinary day, a different kind of problem showed up. Not a bug, a quality gap. A fleet you run by voice lives or dies on two translations: what your spoken words become when they reach an agent, and what an agent's output becomes when it comes back to you. Both were one-shot, and both were weak. So we added a small, cheap reasoning pass at each edge.
On the way in, a prompt rewriter. What you say to an agent no longer goes in raw. A quick context pass rewrites your spoken, half-formed instruction into a sharper prompt: resolving "it" and "that" against the recent thread, folding in what the agent is already working on, cleaning up speech-to-text, making implicit constraints explicit. You speak loosely; the agent receives something well-formed. Your own message bubble still shows your original words; the rewrite is invisible plumbing only the agent sees.
On the way out, a smarter summary. A summary used to be just "make it short", and short is not the same as useful. The failure mode was over-compression: a tidy one-liner that had thrown away the actual answer. Now each finished turn produces two summaries from the agent's output and the recent thread: a conversational reply that actually answers your latest message, written to fit the conversation, and a short ten-word line for the spoken read-out. Summarize for signal, not just for length.
Both are the same shape as the manager: a cheap, stateless reasoning step over the fleet's own data. And both are invisible when they work. You just notice that what you say lands better, and what comes back actually answers you.
The next frontier (an idea from a friend) was the manager itself, and the first phase of that is now built.
The manager used to make one decision per request: spawn or send or answer or clarify. But a real spoken instruction is often several at once:
"Update the expo agent to also handle Android, and then spin up a new agent to write the release notes."
That's a send and a spawn in one breath, and the old manager had to pick one.
Now it returns a plan (an ordered list of actions) and executes them in
sequence:
{ "reply": "Sending the Android instruction to expo-deploy and spawning a release-notes agent.",
"actions": [
{ "action": "send", "target": "expo-deploy", "message": "…also handle Android…" },
{ "action": "spawn", "label": "release-notes", "task": "Write the release notes…" }
] }The shape barely changed: it's still a cheap, stateless reasoning step over the live fleet; it just emits a list of the moves it already knows how to make, plus one line to say back to you. The first real test did exactly the thing in the example: messaged the existing expo agent and spawned a new one, from a single sentence.
It's a small schema change with a large payoff: you stop thinking in single commands and start thinking in intentions, and the manager works out how many moves that takes. Which is, after all, what a handler running a field op from his phone actually does.
There's a working version. You open the app, connect to a box by username and IP, and it installs and starts the fleet for you over SSH. You see your agents as a live feed, hold the button to talk to the manager, and watch your request turn into a new agent or a message to an existing one. Open an agent and it's a chat thread: its summaries on the left, your voice on the right; if it needs a decision, you tap the choice. It reconnects when you return to it, keeps the screen awake while work is happening, and stays out of your way when nothing is. And when you're back at your laptop, you SSH into the same box and pick up exactly where the manager left off: the agents are still there, still in their panes.
It is genuinely the image from the film, in miniature: a fleet you can see and steer from your pocket.
This article was written by Claude as we built the project together, with a final human edit once we were done.