Lightweight VAD + diarization

[ezrakoreen]

Summary

Build the streaming audio segmentation and speaker-attribution pipeline that turns raw conversation audio into timestamped, speaker-labeled speech segments using lightweight VAD, pyannote diarization, and cosine similarity against the enrolled user voice embedding.

Context

Before transcription can happen reliably, the system needs to determine which parts of the audio contain speech and which speaker is talking. This ticket covers the low-level audio pipeline from rolling audio ingestion through diarization and user-vs-interlocutor labeling. It should avoid unnecessary expensive diarization work by using a lightweight VAD pass first.

Tasks

• Implement audio stream ingestion that accepts a live audio stream from a phone
• Maintain a rolling 5-second audio window for downstream processing
• Run lightweight VAD on each window using Silero to detect speech regions before diarization
• Skip pyannote calls for windows with no detected speech
• When there is detected speech, no longer use the 5-second window. All audio from a given conversation should be stored in a single window capped at ~10 minutes.
• Integrate pyannote/speaker-diarization-community-1 for diarization on these speech-positive windows
• Evaluate whether to use regular diarization output or exclusive_speaker_diarization for cleaner downstream reconciliation
• Compute cosine similarity between diarized speaker embeddings and the enrolled user audio embedding
• Label diarized regions as user or interlocutor based on similarity scoring and confidence thresholds
• Handle low-confidence or ambiguous segments explicitly instead of forcing a label
• Merge or stitch adjacent windows if needed so speaker-labeled segments remain coherent across window boundaries
• Produce a structured segment output contract for downstream ASR, including start_time, end_time, speaker_label (user/interlocutor), and a way to actually obtain the audio data from the file.
  • Use sample index to offset into the audio file (index = sample * audio frequency, probably 16000 b/c 16 kHz). If you run into problems lmk.
• Add tests for window behavior in diarization, VAD gating, diarization output handling, and speaker attribution logic.

Acceptance Criteria

• Feature works as described
• No console errors
• Tests pass
• Audio is processed in rolling 10-second windows
• Silero VAD prevents unnecessary pyannote diarization on non-speech windows
• Diarized speech regions are labeled as user, interlocutor, or explicit fallback state when attribution is uncertain
• Output includes stable timestamps and speaker labels suitable for downstream ASR
• Segment stitching across adjacent windows does not create obvious duplicated or fragmented speaker turns

Notes

• This ticket should stop at timestamped, speaker-labeled audio segments; it should not perform ASR or generate final transcript text
• The pyannote model card indicates community-1 expects mono audio sampled at 16kHz, supports in-memory processing, and exposes exclusive_speaker_diarization, which may simplify downstream alignment
• community-1 also requires pyannote.audio, accepted Hugging Face model terms, and a Hugging Face access token
• Suggested output shape for downstream use:
  • start_time
  • end_time
  • speaker_label
  • audio_reference or waveform slice metadata
• Model to use:
  • https://huggingface.co/pyannote/speaker-diarization-community-1