Joplin Notes Agent

Ask questions about your Joplin notes, by typing or out loud, and get answers grounded in what you actually wrote.

This is a side project for learning how AI agents and voice interfaces work by building one rather than reading about one. No LangGraph, no CrewAI — the reason → act → observe loop those frameworks wrap is written by hand here, in about a hundred lines, so the mechanics stay visible. The voice layer is the same idea applied to speech: real speech-to-text and text-to-speech APIs, wired up directly, with the failure modes left in view.

Licensed under the MIT License (see LICENSE).

What it does

Answers questions from your archive — it searches first and reads the most promising notes, rather than guessing from memory, and tells you which notes an answer came from.
Three ways to search — keyword, meaning (local embeddings), and recency. The model picks; you can steer it.
Talks and listens — an optional voice front-end. Press Enter, ask out loud, hear the answer.

you> What did I write about the bracket character class issue in PowerShell?
you> Find my notes on the CERN network setup
you> What have I written about lately?
you> Anything about the Celana website that's still open?

Watch the [step N] tool_name(...) lines as it works — that's the loop happening in real time.

How it's wired

agent.py            the ReAct loop: reason -> call a tool -> observe -> repeat
joplin_tools.py     talks to Joplin's local Data API (the only Joplin-aware file)
semantic_search.py  local embedding search — meaning, not keywords
help_text.py        the agent's self-knowledge: help screens + live status
service_status.py   API error messages, token spend, Deepgram credit
build_index.py      builds the semantic index (run occasionally)
joplin.ps1          run-anywhere launcher (voice by default, --text for the REPL)

voice_mvp.py        the voice front-end: mic -> STT -> agent -> TTS -> speakers
voice_check.py      audio diagnostic — run this first if the mic misbehaves
deepgram_check.py   speech API smoke test + a keyterm accuracy experiment

Your notes stay local except for the text of the question and the retrieved excerpts, which go to the model API like any normal request. Embedding and indexing happen entirely on your machine.

Voice interface (default — `joplin`, `python voice_mvp.py`)

flowchart TD
    ENTER["Enter to speak<br/>(or just type)"] --> REC["record_until_enter()<br/>mic capture via sounddevice"]
    REC --> STT["Deepgram Nova-3<br/>speech-to-text + keyterm prompting"]
    STT --> Q["Transcribed question"]
 
    Q --> LOOP["ask_agent() — ReAct loop<br/>reason to tool call to observe, repeat"]
    LOOP -->|tool_use| TOOLS{"list_notebooks / search_notes /<br/>list_recent_notes / semantic_search /<br/>get_note / get_help"}
    TOOLS --> JT["joplin_tools.py"]
    JT --> JD["Joplin Data API<br/>Web Clipper, localhost:41184"]
    JD --> LOOP
    TOOLS --> SS["semantic_search.py<br/>reads note_embeddings.npz"]
    SS --> LOOP
    LOOP -->|messages.create| CL["Anthropic API<br/>AGENT_MODEL"]
    CL --> LOOP
 
    LOOP --> ANS["Final answer text"]
    ANS --> FLAT["to_speakable()<br/>strips Markdown, expands URLs"]
    FLAT --> TTS["Deepgram Aura-2<br/>text-to-speech, streamed"]
    TTS --> SPK["Speakers"]
    ANS --> PRINT["Full answer also printed<br/>(unabridged, with URLs)"]

Text-only REPL (`joplin --text`, `python agent.py`)

flowchart TD
    U["you&gt; typed question"] --> LOOP["agent.py — ReAct loop<br/>reason to tool call to observe, repeat"]
    LOOP -->|tool_use| TOOLS{"list_notebooks / search_notes /<br/>list_recent_notes / semantic_search /<br/>get_note / get_help"}
    TOOLS --> JT["joplin_tools.py"]
    JT --> JD["Joplin Data API<br/>Web Clipper, localhost:41184"]
    JD --> LOOP
    TOOLS --> SS["semantic_search.py<br/>reads note_embeddings.npz<br/>built by build_index.py"]
    SS --> LOOP
    LOOP -->|messages.create| CL["Anthropic API<br/>AGENT_MODEL"]
    CL --> LOOP
    LOOP --> OUT["Answer rendered in terminal<br/>colorama + Markdown to ANSI"]

Quickstart

Needs Python 3.10+, the Joplin desktop app running, and an Anthropic API key (https://platform.claude.com/dashboard).

git clone https://github.com/smyrnakis/joplin-notes-agent.git
cd joplin-notes-agent

python -m venv .venv
.venv\Scripts\Activate.ps1

pip install -r requirements-voice.txt      # to include voice
# pip install -r requirements.txt          # or this for text only

Copy-Item .env.example .env              # then fill in your keys
python build_index.py                    # optional, enables semantic search
python agent.py

On macOS/Linux the only differences are source .venv/bin/activate and your editor of choice for .env; the Python is identical.

Enabling Joplin's API (once per machine): open Joplin, go to Tools/Options > Web Clipper, click Enable Web Clipper Service, and copy the Authorization token into .env as JOPLIN_TOKEN. The API only exists while the desktop app is running.

Two notes on the install. sentence-transformers pulls in PyTorch, so the first run takes a few minutes and a few hundred MB. And build_index.py embeds every note locally — slow once, then instant forever, and re-run whenever you want the index to catch up. The agent works without it; it just won't have semantic_search available.

Verify Joplin on its own before involving the model:

python joplin_tools.py "some keyword"

Voice interface

mic → Nova-3 (speech-to-text) → the same agent loop → Aura-2 (text-to-speech) → speakers

pip install -r requirements-voice.txt
python voice_mvp.py          # or just `joplin`, once set up below

Press Enter to start recording, Enter again to stop. Or just type the question instead — same pipeline, and typed questions stay silent unless you ask for audio ("read it out loud"). Every turn prints a latency breakdown: stt | agent | tts | speech | to-first-sound.

Built with Deepgram: Nova-3 for transcription, Aura-2 for speech, sounddevice for capture and playback. Requires a DEEPGRAM_API_KEY in .env. The voice layer is purely additive — it imports ask_agent() and never changes how the agent reasons, so the text interface behaves identically whether or not voice is installed.

If the microphone misbehaves, run the audio path on its own first — no API key, no network, no model:

python voice_check.py            # list devices, record, play back
python voice_check.py --save     # also write a WAV to test against

Domain vocabulary is the interesting problem

A general-purpose speech model has never seen your project names. The failure mode is worse than it sounds: rather than misspelling an unfamiliar proper noun, Nova-3 tends to delete it silently, leaving a perfectly fluent sentence that means something else. A visible misspelling you'd catch; a clean sentence with a word missing you won't.

Deepgram's keyterm prompting fixes it. deepgram_check.py demonstrates this as a controlled experiment — the same recorded WAV sent twice, changing only the keyterm list, then a word-level diff of the two transcripts:

python voice_check.py --save --seconds 12   # record a hard sentence
python deepgram_check.py                    # transcribe it twice and diff

Set DEEPGRAM_KEYTERMS in .env to your own vocabulary — the committed default is a placeholder, since one person's jargon isn't anyone else's.

Worth knowing: this held up over the batch transcription endpoint, but the same keyterms performed noticeably worse over a live streaming connection. Whether that's inherent to incremental decoding or an artifact of browser audio capture is unresolved — see FUTURE_WORK.md.

Commands

Handled locally, before the question reaches the model — instant, free, and they can't hallucinate a command that doesn't exist. Available in both the text and voice REPLs.

Command	What you get
`help`	What it does, example questions per search mode, the command list
`help search`	Keyword vs. meaning vs. recency, and how to steer between them
`help index`	The semantic index, its state, and when a rebuild is needed
`help config`	Every setting with its currently active value
`help trouble`	The real failure modes and their fixes
`status`	Live check: Joplin reachable, index drift, API spend, keys set
`reindex`	Rebuild the semantic index in place

status is the one worth knowing. It compares the index against Joplin's newest notes and turns "is my index current?" into a number, and reports what each API is costing:

  Semantic index   note_embeddings.npz  found
                   1,475 notes · built 2026-07-04 (28 days ago) · 4.6 MB
                   39 notes changed since — run `reindex`

  Anthropic spend  6 calls · 48,210 in / 2,905 out · ~$0.0627
                   this session only — Anthropic exposes
                   no balance on the standard API
  Deepgram credit  $199.80 usd

The two providers differ in what they'll tell you, and the labels say so rather than implying both are balances. Deepgram publishes remaining credit, but only to a key with the billing:read scope — so DEEPGRAM_ADMIN_KEY is a separate, optional entry in .env, keeping the key sent on every transcription request minimally scoped. Anthropic has no balance endpoint on the standard API, so that figure is this process's own metered spend, priced from the public rate card.

Fuzzily phrased questions work too. "How do you operate?", "why can't you find my note?" won't exact-match a command, so the model reaches for a get_help tool that reads the same strings help prints. One source of truth, two consumers — the agent's self-description can't drift from its behaviour.

No key or token material is ever printed, only whether each is set.

How retrieval works

Four tools, all aimed at the same problem: don't blindly read every match, narrow down first.

search_notes — Joplin full-text search, returning a short preview of each match rather than full bodies. The model reads the previews to decide which one or two notes are worth fetching in full.
list_notebooks — when a question clearly points at one area, scope the search to that notebook instead of the whole archive.
list_recent_notes(days) — for "what have I written lately", where there's no keyword to search for. Pages /notes sorted by updated_time and stops at the first note older than the cutoff.
semantic_search(query) — for when you don't remember the words you used. Keyword search finds "onboarding" only if that word is in the note; semantic search compares meaning, so a vague description can match a note that never uses any of those words.

This is the "skim, then commit" pattern: cheap previews first, full content only for the few that look right. Cheaper in tokens, and more accurate, because the model chooses based on real content rather than title-guessing. See Anthropic's Building effective agents for the broader design philosophy.

How semantic search works end to end. build_index.py fetches every note, turns each into a 384-number vector with sentence-transformers, and saves the vectors plus metadata to note_embeddings.npz. That's the slow part, done once. At question time only your short query is embedded — one vector — and compared against the saved ones with a dot product. No API call, no internet once the model is cached.

Design notes

A failed tool call doesn't kill the session. Every dispatch is wrapped; failures become {"error": ...} fed back to the model as a normal tool result, and print in red so they're visible while you watch. The system prompt tells the model not to retry an identical failed call — a bad id or query isn't transient — but to try something different or say plainly that it couldn't find the answer.
Tokens never reach the model or the logs. Every Joplin request goes through one shared helper that strips the token out of error messages, so a 404 can't leak it into a log file or into the model's context.
Folder ids aren't note ids. list_notebooks returns folder ids and get_note needs note ids; the tool descriptions say so explicitly, because the model will otherwise mix them and Joplin will correctly 404.
API failures explain themselves. An expired key, an exhausted credit balance and a dropped connection have completely different fixes, so each gets its own message instead of a shared traceback.
Markdown is rendered, not printed. Step logs are dimmed, **bold** and `code` become real styling, and the voice layer flattens the same Markdown so it isn't read aloud as literal asterisks.

Tuning knobs

Setting	What it controls
`MAX_ITERATIONS` in `agent.py`	Hard cap on loop turns (default 12) — raise for multi-note synthesis, lower to fail fast
`AGENT_MODEL` env var	Swap models without touching code
`SYSTEM_PROMPT`	The agent's rules — tighten this first if it over- or under-searches
`TOOLS[*]["description"]`	The model only knows what these say; vague descriptions cause wrong tool choices
`EMBEDDING_MODEL` env var	The local embedding model — re-run `build_index.py` after changing it
`SEMANTIC_INDEX_PATH` env var	Where the index lives, if you want more than one
`DEEPGRAM_KEYTERMS` env var	Vocabulary to boost in speech recognition
`DEEPGRAM_TTS_MODEL` env var	Which Aura-2 voice speaks the answers

Run from anywhere

joplin.ps1 activates the venv and runs the agent from any directory:


`joplin`	Voice interface — Enter to speak, Enter to stop
`joplin --text`	Text-only REPL; no microphone or Deepgram key needed

To get a plain joplin command, add a function to your PowerShell profile:

if (-not (Test-Path $PROFILE)) { New-Item -ItemType File -Path $PROFILE -Force }
notepad $PROFILE

function joplin { & "<path-to-clone>\joplin.ps1" @args }

Then . $PROFILE to reload. Adding the repo to PATH directly doesn't work, because the agent must run under the venv's Python — joplin.ps1 points at .venv\Scripts\python.exe explicitly.

Troubleshooting

Almost every problem is one of these:

Symptom	Cause
Connection errors from any tool	Joplin desktop isn't open, or Web Clipper isn't enabled
`ModuleNotFoundError`	The venv isn't activated — your prompt should show `(.venv)`
`semantic_search` unavailable	The index hasn't been built; run `python build_index.py`
Answers cite stale notes	The index has drifted; `status` will say by how much, then `reindex`
Silence, or no transcript, from the mic	Run `python voice_check.py` — it isolates audio from everything else
A proper noun goes missing from a transcript	Add it to `DEEPGRAM_KEYTERMS` in `.env`

help trouble covers the same ground without leaving the session.

FilesExpand file tree

README.md

Latest commit

History