ASR

[ezrakoreen]

Summary

Build the transcription and dialog-assembly pipeline that consumes speaker-labeled audio segments, runs ASR, and outputs structured dialog text in speaker-attributed turn format.

Context

Once the audio pipeline can produce timestamped segments labeled as user or interlocutor, the next step is to transcribe those segments and assemble them into a usable dialog representation. This ticket covers ASR and transcript construction, not diarization or speaker attribution itself.

Tasks

• Define the input contract from the VAD + diarization ticket
• Integrate ASR for speech segments
  • I think OpenAI's Whisper will be best for this but idk for sure
• Transcribe each speaker-labeled segment while preserving timestamps and speaker labels
• Merge adjacent segments from the same speaker into cleaner conversational turns
• Assemble final dialog output in structured form such as speaker: text
• Preserve timing metadata so transcript lines can still be traced back to the original audio
• Handle ASR failures, empty transcriptions, partial speech, and low-confidence cases cleanly
• Add normalization and cleanup rules for punctuation and whitespace
• Define the final output schema for downstream storage or conversation-ingestion services
• Add tests for ASR integration, turn merging, structured dialog formatting, and duplicate-fragment cleanup

Acceptance Criteria

• Feature works as described
• No console errors
• Tests pass
• The pipeline accepts speaker-labeled segments from diarization and produces transcribed dialog output
• Final output preserves speaker attribution as user vs interlocutor (will be integrated with face recognition later)
• Consecutive segments from the same speaker are merged into readable turns where appropriate
• Final output is available in structured dialog form such as:
• user: text
• interlocutor: text

Notes

• This ticket depends on the VAD ticket being complete, or at least having a template of the output. Reference that ticket for the structure.
• Keep clean boundary between tickets:
  • Prev ticket owns audio segmentation, diarization, and speaker attribution
  • This ticket owns ASR and dialog formatting
• Suggested intermediate schema from Ticket 1:
  • start_time
  • end_time
  • speaker_label
  • audio buffer reference
• Suggested final schema from this ticket:
  • ordered dialog turns
  • per-turn speaker label
  • per-turn text
  • optional per-turn timestamps and ASR confidence
• Source used for the diarization dependency boundary:
  • https://huggingface.co/pyannote/speaker-diarization-community-1