Live speaker labels during the call (phase 4, internal test) - #43
Draft
turantekin wants to merge 4 commits into
Draft
Live speaker labels during the call (phase 4, internal test)#43turantekin wants to merge 4 commits into
turantekin wants to merge 4 commits into
Conversation
Co-Authored-By: Claude Fable 5 <[email protected]>
…ities Every 30s during a call (opt-in, default off), re-diarize the audio so far and relabel non-Me bubbles as Speaker N. A greedy embedding match against the previous sweep keeps identities from flipping; the final post-call pass maps through the same anchors for continuity and stays authoritative. Co-Authored-By: Claude Fable 5 <[email protected]>
Display-only, confirm-first: the stored label stays Speaker N until the user confirms in the post-call naming flow. Needs both experimental live labels and Remember voices enabled. Co-Authored-By: Claude Fable 5 <[email protected]>
The live popover drops the clip player on purpose (playback would bleed into the capture, and the voice is audibly live) and keeps the sounds-like confirm button. Pre-sweep "Them" bubbles stay untappable — naming applies to real detected voices only. Co-Authored-By: Claude Fable 5 <[email protected]>
turantekin
force-pushed
the
claude/live-speakers-phase4
branch
from
August 20, 2026 08:55
97e97f1 to
3856c95
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INTERNAL TEST ONLY. Draft on purpose: do not merge or release until it has survived real calls.
During a recording (opt-in, default off: Settings > Transcription > Speaker Detection > "Live speaker labels"), Parrot re-diarizes the call-so-far every 30 seconds and upgrades non-Me bubbles from "Them" to Speaker 1/2/... while the call is still running. The Copilot's context window picks the labels up automatically since it reads the same segments.
Approach
Periodic re-runs of the already-shipped chunked engine, per the spec's phase-4 direction: at ~380x realtime each sweep costs single-digit seconds, gives near-post-call accuracy, and avoids integrating a second (measurably worse) streaming model family. Trade-off accepted for v1: labels refresh in 30 s steps, and segments that arrive between sweeps sit as "Them" until the next pass.
Keeping identities stable
The clustering relabels by talk time on every run, so Speaker 1 and Speaker 2 could swap mid-call. A pure
stableMappingfunction greedily matches each sweep's cluster embeddings to the previous sweep's (cosine >= 0.7, the calibrated same-voice threshold) and hands unmatched clusters the next free label. The final post-call pass maps through the same anchors, so the identities the user watched live survive into the finished transcript. With no live sweeps the mapping is identity, so existing behavior is untouched.Safety rails
Internal test protocol
Enable the toggle, start a recording, play multi-voice audio (e.g. afplay a slice of a past call's system track), and watch: "Them" should become Speaker N after the first sweep (~60 s in) without identity flips on later sweeps; stopping the call must produce the same final labels and naming flow as before. Watch thermals/battery: one 1-6 s ANE burst per 30 s alongside Whisper.
🤖 Generated with Claude Code