Skip to content

info.md

Latest commit

 

History

History
119 lines (90 loc) · 10.9 KB

info.md

File metadata and controls

119 lines (90 loc) · 10.9 KB
Raw

MentraOS Camera Example — Deep Dive

This is a live streaming viewer for MentraOS smart glasses that layers real-time transcription, active speaker detection (ASD), and persistent face-based identity onto the video feed. It's a hybrid Bun/TypeScript + Python system bridged by WebRTC via MediaMTX.

Physical data flow

┌─────────┐  WHIP (WebRTC publish)    ┌──────────┐  WHEP (WebRTC pull)  ┌──────────────────┐
│ Glasses │ ─────────────────────────▶│ MediaMTX │ ────────────────────▶│ view_stream.py   │
│  (cam   │           :8889/whip      │  media   │       :8889/whep     │  (processors)    │
│  + mic) │                           │  server  │                      └──────────────────┘
└─────────┘                           └──────────┘
     ▲
     │ control (start/stop stream, photo, TTS, ...)
┌────┴────────────────┐     connects to MentraOS cloud via @mentra/sdk
│ src/server (Bun/TS) │◀── cloud relays events + lets it call `session.camera.startStream(...)`
└─────────────────────┘

Two independent processes, glued by MediaMTX:

  • Bun server controls the glasses through the MentraOS cloud SDK.
  • Python viewer receives the raw A/V stream and does all the perception.

They don't talk to each other directly — the TS side could expose an SSE /transcripts endpoint for cloud-side transcription (CameraApp.ts:24-37), but the viewer has USE_MENTRA_TRANSCRIPTION = False (view_stream.py:44), so transcription is done locally via Soniox for lower latency.

Bun/TypeScript side (src/)

src/index.ts → src/server/CameraApp.ts

Boots an AppServer from @mentra/sdk. When a user opens the app on the glasses, onSession() fires and it calls session.camera.startStream({ rtmpUrl: WHIP_URL, ... }). The rtmpUrl field is misnamed — the firmware actually routes based on URL scheme, so a http://.../whip URL triggers WebRTC publishing to MediaMTX. Auto-reconnects on timeout/disconnected/error (CameraApp.ts:105-114).

Per-user composition (src/server/session/User.ts)

Each connected user gets a User object wrapping five managers:

  • PhotoManager — session.camera.requestPhoto(), stores in memory, broadcasts to SSE clients
  • TranscriptionManager — cloud-side session.events.onTranscription(...), broadcasts
  • AudioManager — TTS via session.audio.speak()
  • InputManager — single tap / button press → take photo
  • StorageManager — theme preference

HTTP API (src/server/routes/routes.ts + api/)

Hono-based REST + SSE: /health, /photo-stream (SSE), /transcription-stream (SSE), /speak, /latest-photo, /photo/:id, /theme-preference.

Frontend (src/frontend/)

React 19 + Tailwind webview served by the Bun server. Authenticates via useMentraAuth() from @mentra/react, handles theme, and uses the SSE endpoints to render photos/transcriptions live. This is the user-facing "app" surface inside MentraOS — separate from the Python perception viewer.

Python side (view_stream.py + stream/)

stream/source.py — WHEP pull

Spawns a background asyncio thread running aiortc. Sends a WebRTC offer to MediaMTX's WHEP endpoint, sets up recvonly transceivers for video + audio, and pumps decoded frames into a thread-safe queue (size 16, drops oldest on backpressure — bounded latency). Reconnects on drop. Yields VideoItem (BGR24 ndarray) and AudioItem (float32 samples @ whatever rate) to the main thread via frames().

stream/pipeline.py — Processor abstraction

A tiny per-processor dispatcher. Each Processor declares:

  • mode: "sync" (runs inline on render thread — fast stuff like face detection) or "async" (runs on a dedicated worker thread with a size-256 inbox that drops oldest)
  • consumes: {"video"}, {"audio"}, or both
  • on_video / on_audio callbacks + draw(frame) for overlay

The main loop in view_stream.py:212-220 is a clean: for item in source.frames(): dispatch → draw → cv2.imshow.

stream/processors/asd.py — Active Speaker Detection

The hardest piece. Uses the LR-ASD model (vendored in third_party/lr_asd/), which needs 10 face crops paired with ~400 ms of MFCC audio and outputs per-frame speaking log-odds.

Pipeline per frame:

  1. Detect (every Nth frame, detect_every_n=2) with OpenCV's YuNet → face bboxes + 5-point landmarks
  2. Track (_tracker.py) — IoU-match detections against active tracks, mint new tracks on unmatched detections. Each track maintains a deque(maxlen=10) of 112×112 gray crops
  3. Frame rate conversion — LR-ASD was trained on 25 fps; the glasses stream at 15 fps. Fix: a pulldown accumulator (phase += 25/15) duplicates crops so 10 crops always represent ~400 ms of wall-clock time, matching the audio side's 4:1 ratio
  4. Audio buffer — the same on_audio callback also feeds a 1s ring buffer, resampled to 16 kHz. When inference fires, grabs the last 6640 samples → MFCC (python_speech_features) → 40×13 matrix
  5. Inference (rate-limited to inference_hz=10) — batch (N_tracks, 10, 112, 112) + broadcast MFCC (N_tracks, 40, 13) handed to a size-1 queue. Worker thread runs the PyTorch forward pass; eviction-on-enqueue guarantees the worker always gets the freshest batch, never stale
  6. Score smoothing — EMA on the raw logits (alpha=0.5), plus a rolling history deque for retroactive lookup
  7. Draw — green box if speaking, red if not, white if waiting for first score. Reads tracks live from self._tracker._tracks so bboxes stay snapped to faces even while inference is still working on the previous batch

Two retroactive lookup methods who_spoke(t_start, t_end) and who_is_speaking_now(window_s=0.3) are the API Soniox uses for attribution. Identity resolution (_identity.py) is invoked from the inference worker only when ASD thinks a track is actively speaking — so gallery embeddings only get captured from real speaking frames, not random listener frames. There's a stability gate (min_frames_for_identity=20) to avoid polluting the gallery with transient camera jitter tracks.

stream/processors/_identity.py — Face gallery

Uses MobileFaceNet (w600k_mbf.onnx, 512-d ArcFace variant from InsightFace's buffalo_s) via onnxruntime CPU. Alignment: YuNet's 5 landmarks → ArcFace 112×112 template via similarity transform.

Matching logic (resolve()):

  • For each gallery identity, compute top-3 mean cosine similarity
  • If best ≥ match_threshold=0.35 AND (best - second) ≥ margin=0.05 → match (append embedding if similarity ≥ 0.45 strong threshold, with 2s cooldown per tid)
  • Otherwise mint new "Person N"

On shutdown, flush() runs a cluster-merge pass: any session-new identity whose centroid matches another gallery entry at ≥ 0.50 gets merged (prefers non-new destinations). This catches cases where the same person was seen across two separate tracks. Then atomic write to .npz + .json sidecar.

stream/processors/soniox.py — Live transcription

Holds a single persistent Soniox WebSocket for the whole session (bypasses MentraOS's VAD-gated transcription, which tears down the socket on silence and pays ~1s cold start). Async processor on its own asyncio thread:

  • _send_audio_loop — drains the queue, sends 16 kHz PCM s16le frames
  • _recv_loop — parses Soniox tokens, distinguishes is_final from interim
  • _keepalive_loop — pings every 15s
  • _dwell_loop — fires an update if the in-progress sentence hasn't grown for 0.5s

Speaker attribution (_classify_speaker): for each finalized segment [start_ms, end_ms], converts stream_ms → monotonic time via a live-updated anchor, calls asd.who_spoke_name(t_start, t_end). If ASD saw a face actively speaking in that window → credit that identity ("Bob" / "Person 3" / "Track N" if unresolved). Otherwise → "User" (wearer, since the glasses mic primarily picks them up).

on_transcript_update event — the real downstream hook. Fires on:

  • Speaker flip (with 200 ms debounce to suppress transient misattributions)
  • Sentence terminator .?! finalized
  • Soniox endpoint <end> token
  • 0.5 s silence dwell

Payload = full cumulative [(speaker, text), ...] transcript including the in-progress sentence as the last entry. Emissions are signature-deduped with a 250 ms min-gap. This is where you'd hook LLM reasoning / tool calls / UI agents — currently a no-op in view_stream.py:71-76.

view_stream.py — Orchestrator

Wires up an ASDProcessor and (conditionally) a SonioxProcessor, runs the pipeline, and on exit:

  1. Collects _transcript_lines
  2. Calls Claude Haiku 4.5 (_extract_name_mappings) with the full transcript + list of unresolved "Person N" labels, asking it to find real names (someone addressed by name, introductions, etc.). Returns a JSON mapping
  3. For each mapping, calls _asd.rename_identity(old, new) → relabels the gallery entry before flush
  4. _asd.close() flushes the gallery with the merge pass

Key design decisions worth noting

  • Latency over completeness everywhere: size-1 inference queue evicts stale batches, size-16 stream queue drops oldest video, size-256 processor inboxes drop oldest. Pipeline-wide: "never let a slow thing hold up a fresh thing."
  • Sync vs async processor split: face detection / tracking / audio buffering is cheap and stays on the render thread so bbox draws don't lag. Only the LR-ASD forward pass and Soniox I/O cross a thread boundary.
  • Soniox instead of MentraOS transcription: the cloud-side path exists (wired in CameraApp.ts:90-103 → /transcripts SSE) but is disabled because it kills/restarts the WS on silence. The Python-side persistent socket wins on latency and speaker attribution lives closer to the video signal.
  • Real-time ASD ↔ transcription synchronization via monotonic clock: both sides update a shared (stream_ms, monotonic) anchor on every audio chunk, so Soniox's stream_ms can be converted back into monotonic time to query ASD's history.
  • Gallery hygiene: only enroll while speaking + stability gate + cooldown + end-of-session merge + Claude-assisted renaming means the gallery grows slowly but cleanly across sessions.

Files you'd want to touch for common changes

  • Hook downstream transcript processing: view_stream.py:71-76 (_on_transcript_update)
  • Tune ASD sensitivity: ASDProcessor.__init__ args in view_stream.py:136 (score_threshold, ema_alpha, inference_hz, display_timeout_sec)
  • Add a new processor: implement stream/pipeline.py:Processor, append to PROCESSORS in view_stream.py
  • Change streaming params (resolution, bitrate, fps): CameraApp.ts:72-82