✨ Support for voxtral realtime stt#1277
Conversation
cameledev
changed the title
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
5d4ec36 to
ec7eb87
Compare
|
ec7eb87 to
de37643
Compare
| const existingIndex = prevSegments.findIndex( | ||
| (s: TranscriptionSegmentWithParticipant) => s.id === segment.id | ||
| ) | ||
| if (existingIndex === -1) { | ||
| return [...prevSegments, { participant, ...segment }] | ||
| } | ||
| const next = prevSegments.slice() | ||
| next[existingIndex] = { ...next[existingIndex], ...segment } | ||
| return next |
There was a problem hiding this comment.
allow update of previously received segments
| "default_country": settings.ROOM_TELEPHONY_DEFAULT_COUNTRY, | ||
| }, | ||
| "subtitle": {"enabled": settings.ROOM_SUBTITLE_ENABLED}, | ||
| "subtitle": {"enabled": True}, # settings.ROOM_SUBTITLE_ENABLED}, |
There was a problem hiding this comment.
REVERT BEFORE MERGE
There was a problem hiding this comment.
REVERT BEFORE MERGE (2/2)
| send_t.cancel() | ||
| try: | ||
| await send_t | ||
| except (asyncio.CancelledError, websockets.WebSocketException): |
| authentication_classes=[LiveKitTokenAuthentication], | ||
| ) | ||
| @FeatureFlag.require("subtitle") | ||
| ) # @FeatureFlag.require("subtitle") |
There was a problem hiding this comment.
revert before merge
| build: | ||
| context: ./src/agents | ||
| target: development | ||
| command: ["python", "multi_user_transcriber.py", "dev"] |
There was a problem hiding this comment.
It's not necessary, but I understand if you want to keep more control over the compose service
|
Confidence Score: 3/5The new plugin needs the empty-commit ordering question resolved before this is production-ready; everything else is safe. The entire new STT plugin flows through
Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # Start a fresh generation. Safe to send here: the previous utterance's | ||
| # transcription.done has already been received (we await it below), so | ||
| # the server-side generation_task is done and won't ignore this commit. | ||
| await ws.send(json.dumps({"type": "input_audio_buffer.commit"})) | ||
| send_t = asyncio.create_task(self._send_audio(ws, pending)) | ||
| try: | ||
| await self._receive_one_transcription(ws, request_id) |
There was a problem hiding this comment.
An input_audio_buffer.commit is sent at line 369 before any audio chunks have been appended for this utterance. If vLLM responds to an empty-buffer commit with transcription.done (as the OpenAI Realtime protocol would), _receive_one_transcription returns immediately with final_text = "" (both data.get("text") and current_text are empty), send_t is cancelled in the finally block, and every utterance produces an empty transcript.
The standard Realtime protocol flow is: append audio → commit to trigger generation. The "final": True commit in _send_audio already ends the generation correctly. The preceding bare commit looks unintentional; removing it should be considered unless vLLM explicitly requires it as a "clear buffer" signal (which would need a comment explaining that contract).
| build: | ||
| context: ./src/agents | ||
| target: development | ||
| target: development |
There was a problem hiding this comment.
| target: development | |
| target: development |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
2 participants
Purpose
Add support for voxtral realtime stt (live transcription).
Why we can't use mistralai plugin
The official
livekit-plugins-mistralaiplugin is hardwired to Mistral's hosted cloud API both the URL path and the wire protocol. We self-host Voxtral on vLLM, which speaks a different protocol on a different path, so the plugin can't be pointed at it. Four distinct issues, in the order they occured:1. Hardcoded WebSocket path
The Mistral SDK's
RealtimeTranscription._build_urlhardcodes the path/v1/audio/transcriptions/realtime.server_urlonly controls scheme + host there is no kwarg to change the suffix, and the LiveKit plugin callsconnect()without overriding it either. vLLM serves Voxtral realtime at/v1/realtime. That route doesn't exist on vLLM, so every WebSocket connnection attemps returned HTTP 500. A monkeypatch on_build_url(/v1/audio/transcriptions/realtime→/v1/realtime) fixed this issue.2. Incompatible wire protocol (most important issue)
Even with the path fixed, the two sides speak different languages:
RealtimeTranscriptionSessionCreated,TranscriptionStreamTextDelta,TranscriptionStreamDonesession.created,transcription.delta,transcription.donesend_audio/flush_audio/end_audioinput_audio_buffer.append/.commitThe Mistral SDK cannot parse vLLM's messages. Monkeypatching this would mean reimplementing the SDK's entire receive path.
3. The handshake failures (how the mismatch actually surfaced)
rt.connect()does two sequential handshakes, and the plugin broke at both:websockets.exceptions.InvalidStatus: server rejected WebSocket connection: HTTP 500thrown out ofConnectionPool.prewarm._build_urlmonkeypatch, the upgrade succeeded (
[accepted]/connection openin vLLM logs), but_recv_handshakethen blocked onawait websocket.recv()waiting for Mistral'ssession.createdevent. vLLM never sends that event (its OpenAI-style handshake has a different shape), so the 10s LiveKit connect timeout fired →CancelledError→TimeoutError→APIConnectionError: Connection error., retried, then session closed as unrecoverable.4. Dependency resolution failure (packaging)
Independently,
livekit-plugins-mistralai==1.5.4pulls inmistralai[realtime]>=2.0.0. Underuv's universal resolution acrossrequires-python = ">=3.12"(which now includes 3.14), that extra isunresolvable →
requirements are unsatisfiable