Local AI: on-device transcript cleanup model (eval → baseline → QLoRA → GGUF sidecar)

[ElbertePlinio]

Local AI experiment: on-device transcript cleanup model

Ship a small local model inside PickScribe that post-processes transcripts fully offline: punctuation/formatting, disfluency removal, and user-vocabulary correction ("pick forge" → "Pickforge", "cloud code" → "Claude Code"). Zero API cost, privacy story: your voice never leaves your machine.

Gate: fine-tuning only happens if plain prompting of an off-the-shelf small model fails the eval. Data + eval decide everything; training compute is trivial (rented 3090 on Vast.ai, ~$1–5 per run).

Agent plan

Goal: decide via eval whether a local cleanup model ships, and ship the winning variant as an optional GGUF sidecar.

Checklist:

• Phase 1 — Eval set: ~200 real dictation transcripts with hand-fixed targets (punctuation, disfluencies, vocab errors); define scoring (WER-style diff + vocab-hit rate)
• Phase 2 — Baseline: off-the-shelf small models (Qwen3 1.7B/4B, Gemma 3n class) with prompt + user vocab injected in context; measure on eval
• Phase 3 — Only if baseline fails: distill targets from a frontier model, QLoRA fine-tune 1–3B on rented 3090, re-run eval
• Phase 4 — Inference bench: GGUF via llama.cpp sidecar; latency per utterance on CPU-only and consumer GPU (RX 9070 XT via Vulkan as reference AMD target)
• Phase 5 — Ship decision: optional model download (not in installer), settings toggle, fallback to current path
• PR/review

Validation:

• Eval score vs baseline, latency budget per utterance on CPU
• Manual dictation session on Linux + macOS

Stretch (separate issue if pursued): fine-tune Whisper itself on dev-domain vocab (tech terms, model IDs, CLI commands) — fixes mishearings at the source instead of post-hoc; also fits a single 3090.

Current status: Planned
Next action: collect real dictation transcripts for the eval set