Gist

Pause reality and ask. A non-native speaker consuming English — a YouTube video, a podcast, or the friend across the table — taps their AirPod the moment they get tripped up. The last 30 seconds of real audio go to a local Gemma 4, which explains the slang, the idiom, the tone — in the learner's native language, spoken back into their ear. Nothing ever leaves the device. Every existing tool is a caption parasite, a verbatim translator, or a synthetic tutor; nobody lets you pause any real audio and ask "what did she just mean?" grounded in the actual moment, fully offline. We work on reality, not on Netflix.

Stack (all local, 36 GB M3 Max)

Ollama gemma4:12b (brain) · TranslateGemma-4B (translation) · mlx-whisper large-v3-turbo (hearing) · Kokoro-82M (voice) · llama-server 12B + mmproj (optional native-audio "ears") · FastAPI/WebSocket server · Chrome MV3 extension (content mode) · PWA (live mode, iPhone Safari over hotspot)

Browser Gemma YouTube rail

The extension injects a learning rail into the YouTube recommendations column and runs Gemma 4 locally through Transformers.js and WebGPU. A thin top bar holds Settings, a Captions/Audio ↔ Video evidence toggle, text-size controls, and listening status. It supports:

• Captions mode when timed captions exist: synchronized scrolling captions, drag-select explain, then Ask more questions for follow-up chat
• Audio mode without captions: typed chat grounded in buffered tab audio
• Video mode: typed chat grounded in an on-demand paused-frame capture

Requirements: Chrome 116 or newer with hardware acceleration enabled. The first load downloads several gigabytes of quantized model data from Hugging Face. Chrome caches those files; inference itself is local and requires no API key or local server. The build bundles ONNX Runtime WASM into extension/dist/ort/ so Chrome's MV3 CSP does not need to fetch jsDelivr at runtime.

cd extension
npm install
npm run build

Open chrome://extensions, enable Developer mode, choose Load unpacked, and select extension/dist. Open any YouTube /watch page — the Gist rail appears automatically and replaces the recommendations list. If Chrome blocks automatic tab audio capture, click Enable listening in the rail (or the toolbar icon as a fallback). For iterative development, npm run dev rebuilds when source files change; reload the extension from chrome://extensions after each build.

Enable Read responses aloud in Settings to speak each completed Gemma response using an installed system voice that matches the configured answer language. Gist filters out browser-provided remote voices, so read-aloud remains offline. If a language has no local voice, install one in macOS System Settings → Accessibility → Spoken Content → System Voice.

Tab audio capture buffers only the active YouTube watch tab and keeps normal stereo playback audible. Gist never captures the microphone, other tabs, or system audio. Capture stops when you leave the watch page, switch tabs, or the rail is gone.

Pause the video (or select caption words — that auto-pauses), set your native language, the language you are learning, and whether answers should come back in native or learning language. The Captions/Audio ↔ Video switch selects exactly one evidence source. Captions/Audio uses caption text when available, otherwise up to 30 seconds of locally buffered 16 kHz PCM. Video mode sends only the paused frame. Seeking clears the old audio buffer. Ads and YouTube video changes also reset it. Frame capture uses tabs.captureVisibleTab, which requires the <all_urls> host permission in MV3.

Selecting caption words pauses playback; clear the selection or hit play yourself when ready. After Explain selection, use Ask more questions to continue in chat with that explanation already in history.

Quickstart

# 1. Models FIRST — venue Wi-Fi is the killer (~21 GB)
./scripts/download_models.sh

# 2. Server (Lanes A+B)
cd server && pip install -e . && cd ..
./scripts/run_server.sh          # ws://localhost:8443 (wss after dev_certs.sh)

# 3. Extension: build it, then chrome://extensions -> Developer mode
cd extension && npm install && npm run build && cd ..
#    -> Load unpacked -> ./extension/dist

# 4. PWA (live mode)
python3 -m http.server 8080 -d pwa   # then open http://localhost:8080

# 5. iPhone path (later, spec §7): ./scripts/dev_certs.sh, trust the CA on
#    the phone, load https://mac.local:8443 over the Mac's hotspot

Docs: docs/spec-local-language-companion.md (full build spec — read first) and docs/protocol.md (frozen interface contracts).

Team lanes (spec §10)

The seams between lanes are the two contracts in docs/protocol.md, agreed in hour 0 and then frozen. Hour 5–6 is the one planned joint integration session; outside it, if you're touching another lane's files, stop and message instead.

• Lane A — Hearing — Clayton (server/hearing/): WS ingest → Silero VAD → 30s PCM ring buffer → mlx-whisper with LocalAgreement-2 → timestamped TranscriptStore → trigger state machine (1.5s VAD disambiguation, question clip) → replay-slower. Owns everything between raw audio and clean text/events.
• Lane B — Brain & voice — Chase (server/brain/): append-only prompt assembly (KV-cache reuse) → Ollama gemma4:12b streaming → Kokoro sentence-by-sentence TTS → TranslateGemma endpoints → flashcard stub → --gemma-ears (timeboxed). Owns everything between clean text/audio in and answer text/speech out, plus prompt tuning and the latency budget.
• Lane C — Faces — Cale (extension/, pwa/): Chrome extension (captions fast path, pause trigger, YouTube learning rail, tabCapture fallback) → PWA (mic AudioWorklet, Media Session AirPod tap, TTS playback) → iPhone (mkcert/hotspot/Safari) → demo polish and backup video. Owns everything the judges see.

Cut order if behind: gemma-ears → tabCapture podcast path → continuous translation → flashcards → iPhone scene.

Testing

scripts/test_all.sh is the one-command iOS verdict: audio continuity, real GGUF quality, Swift unit/UI tests, and all eight simulator screenshots. It fails fast and prints a compact duration summary; expect under ten minutes.

scripts/audio_pipeline_test.sh certifies the 48→16 kHz capture path under sequential, real-time, and jittered callback schedules. Every run must remain sample-exact with no gaps.

scripts/model_eval.sh loads the real Gemma 4 E2B GGUF through the vendored Mac llama framework and real iOS prompt templates. It requires at least 13 of 15 grounded-language fixtures; override the file with GIST_MODEL_PATH.

scripts/sim_screenshot_tour.sh builds and installs DEMO_MODE, then captures onboarding, settings, warming, listening, bloom, card, speaking, and history to docs/screenshots/. Those images are both the visual regression net and the morning review artifact.