feat(voice): Grok streaming TTS, STT, optional realtime bridge, smoke#6
Merged
Conversation
- WS stream TTS (wss://api.x.ai/v1/tts) with TTFA optimize_streaming_latency - Grok STT provider POST /v1/stt + unit tests (mock) - Experimental realtime bridge flag OPENVOICEUI_GROK_REALTIME default off - scripts/smoke_grok_tts.py writes /tmp only; README white-label docs No secrets. XAI_API_KEY env only. pytest green for new Grok suites.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Streaming ignores voice_id kwarg
- stream_speech_sync/async now resolve voice_id like generate_speech, and the registry adapter pops voice_id before delegating.
- ✅ Fixed: WS stream lacks receive timeout
- Both sync and async WS receive loops now enforce a STREAM_TIMEOUT deadline and raise RuntimeError on expiry.
Or push these changes by commenting:
@cursor push 22595d11c5
Preview (22595d11c5)
diff --git a/providers/tts/grok_provider.py b/providers/tts/grok_provider.py
--- a/providers/tts/grok_provider.py
+++ b/providers/tts/grok_provider.py
@@ -78,7 +78,7 @@
self.validate_text(text)
if not self.api_key:
raise TTSError("grok", "XAI_API_KEY not set")
- voice = kwargs.pop("voice", self.default_voice)
+ voice = kwargs.pop("voice_id", None) or kwargs.pop("voice", self.default_voice)
language = kwargs.pop("language", None) or kwargs.pop("lang", None) or self.default_language
try:
return self._get_impl().stream_speech_sync(
diff --git a/tests/test_grok_tts_stream.py b/tests/test_grok_tts_stream.py
--- a/tests/test_grok_tts_stream.py
+++ b/tests/test_grok_tts_stream.py
@@ -18,7 +18,7 @@
def send(self, data):
self.sent.append(data)
- def recv(self):
+ def recv(self, timeout=None):
if not self._frames:
raise RuntimeError("no more frames")
return self._frames.pop(0)
diff --git a/tts_providers/grok_provider.py b/tts_providers/grok_provider.py
--- a/tts_providers/grok_provider.py
+++ b/tts_providers/grok_provider.py
@@ -175,6 +175,7 @@
speed: Optional[float] = None,
text_normalization: bool = False,
connect_fn=None,
+ **kwargs: Any,
) -> Iterator[bytes]:
"""
Stream TTS via bidirectional WebSocket ``wss://api.x.ai/v1/tts`` (TTFA path).
@@ -187,6 +188,7 @@
if not self.api_key:
raise RuntimeError("XAI_API_KEY not set")
+ voice = kwargs.get("voice_id") or voice or "eve"
self.validate_text(text)
# WS path has no hard total length on the session; still guard unary-like
# single-delta size
@@ -235,8 +237,19 @@
with connector(uri, headers) as ws:
ws.send(json.dumps({"type": "text.delta", "delta": text}))
ws.send(json.dumps({"type": "text.done"}))
+ deadline = t0 + STREAM_TIMEOUT
while True:
- raw = ws.recv()
+ remaining = deadline - time.time()
+ if remaining <= 0:
+ raise RuntimeError(
+ f"[grok:timeout] TTS stream after {STREAM_TIMEOUT}s"
+ )
+ try:
+ raw = ws.recv(timeout=remaining)
+ except TimeoutError as exc:
+ raise RuntimeError(
+ f"[grok:timeout] TTS stream after {STREAM_TIMEOUT}s"
+ ) from exc
if isinstance(raw, bytes):
# Some stacks may deliver binary audio frames — yield as-is
if first:
@@ -296,10 +309,12 @@
speed: Optional[float] = None,
text_normalization: bool = False,
connect_fn=None,
+ **kwargs: Any,
) -> AsyncIterator[bytes]:
"""Async variant of WebSocket streaming TTS (prefer TTFA path)."""
if not self.api_key:
raise RuntimeError("XAI_API_KEY not set")
+ voice = kwargs.get("voice_id") or voice or "eve"
self.validate_text(text)
if len(text) > MAX_CHARACTERS:
raise ValueError(f"Text exceeds max {MAX_CHARACTERS} characters per delta")
@@ -330,7 +345,19 @@
async with connector(uri, headers) as ws:
await ws.send(json.dumps({"type": "text.delta", "delta": text}))
await ws.send(json.dumps({"type": "text.done"}))
- async for raw in ws:
+ deadline = t0 + STREAM_TIMEOUT
+ while True:
+ remaining = deadline - time.time()
+ if remaining <= 0:
+ raise RuntimeError(
+ f"[grok:timeout] TTS stream after {STREAM_TIMEOUT}s"
+ )
+ try:
+ raw = await asyncio.wait_for(ws.recv(), timeout=remaining)
+ except asyncio.TimeoutError as exc:
+ raise RuntimeError(
+ f"[grok:timeout] TTS stream after {STREAM_TIMEOUT}s"
+ ) from exc
if isinstance(raw, bytes):
if first:
first = FalseYou can send follow-ups to the cloud agent here.
…n WS recv - stream_speech_sync/async now accept voice_id alias (parity with unary generate_speech), fixing wrong-default/TypeError for voice_id callers - registry stream adapter forwards voice_id - WS recv now bounded by STREAM_TIMEOUT (was defined but unused); raises TimeoutError instead of blocking forever if audio.done never arrives - tests: voice_id alias + stream-timeout coverage (Bugbot findings)
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Async stream silent early exit
- Replaced the StopAsyncIteration break with a RuntimeError so premature WebSocket close surfaces as a failure instead of silently completing with partial audio.
Or push these changes by commenting:
@cursor push c93b74917e
Preview (c93b74917e)
diff --git a/tts_providers/grok_provider.py b/tts_providers/grok_provider.py
--- a/tts_providers/grok_provider.py
+++ b/tts_providers/grok_provider.py
@@ -360,7 +360,9 @@
try:
raw = await asyncio.wait_for(aiter.__anext__(), timeout=remaining)
except StopAsyncIteration:
- break
+ raise RuntimeError(
+ "[grok:ws] WebSocket closed before audio.done"
+ )
except asyncio.TimeoutError as exc:
raise TimeoutError(
f"[grok:ws] recv timed out after {STREAM_TIMEOUT}s"You can send follow-ups to the cloud agent here.
Owner
Author
Owner
Author
Raise RuntimeError when the async WebSocket iterator ends without an audio.done event, matching sync stream behavior and surfacing truncated audio to callers. Applied via @cursor push command
|
The Bugbot autofix is already on the PR branch and pushed. Push status
Fix applied — in except StopAsyncIteration:
raise RuntimeError(
"[grok:ws] WebSocket closed before audio.done"
)PR #6 — feat(voice): Grok streaming TTS, STT, optional realtime bridge, smoke
No further action needed on your side until Bugbot finishes; you can merge when that check is green.   |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Streaming TTS rejects lang
- stream_speech now resolves the lang alias to language before calling stream_speech_sync, matching unary generate_speech behavior.
- ✅ Fixed: Stream forwards extra kwargs
- stream_speech filters kwargs to stream_speech_sync's accepted parameters and the adapter delegates through that wrapper instead of forwarding unknown keys.
- ✅ Fixed: Stream mutates caller kwargs
- GrokTTSProvider.stream_speech now uses non-destructive .get reads like generate_speech, leaving the caller's options dict unchanged.
Or push these changes by commenting:
@cursor push 6c0d1ed8bf
Preview (6c0d1ed8bf)
diff --git a/providers/tts/grok_provider.py b/providers/tts/grok_provider.py
--- a/providers/tts/grok_provider.py
+++ b/providers/tts/grok_provider.py
@@ -78,11 +78,20 @@
self.validate_text(text)
if not self.api_key:
raise TTSError("grok", "XAI_API_KEY not set")
- voice = kwargs.pop("voice_id", None) or kwargs.pop("voice", None) or self.default_voice
- language = kwargs.pop("language", None) or kwargs.pop("lang", None) or self.default_language
+ voice = kwargs.get("voice_id") or kwargs.get("voice") or self.default_voice
+ language = (
+ kwargs.get("language") or kwargs.get("lang") or self.default_language
+ )
try:
- return self._get_impl().stream_speech_sync(
- text, voice=voice, language=language, **kwargs
+ return self._get_impl().stream_speech(
+ text,
+ voice=voice,
+ language=language,
+ **{
+ k: v
+ for k, v in kwargs.items()
+ if k not in ("voice", "voice_id", "language", "lang")
+ },
)
except TTSError:
raise
diff --git a/tts_providers/grok_provider.py b/tts_providers/grok_provider.py
--- a/tts_providers/grok_provider.py
+++ b/tts_providers/grok_provider.py
@@ -298,7 +298,29 @@
**kwargs: Any,
) -> Iterator[bytes]:
"""Alias for :meth:`stream_speech_sync` (iterator of audio bytes)."""
- return self.stream_speech_sync(text, voice=voice, **kwargs)
+ language = kwargs.get("language") or kwargs.get("lang") or "en"
+ voice_id = kwargs.get("voice_id")
+ stream_kwargs = {
+ k: v
+ for k, v in kwargs.items()
+ if k
+ in (
+ "codec",
+ "sample_rate",
+ "bit_rate",
+ "optimize_streaming_latency",
+ "speed",
+ "text_normalization",
+ "connect_fn",
+ )
+ }
+ return self.stream_speech_sync(
+ text,
+ voice=voice,
+ voice_id=voice_id,
+ language=language,
+ **stream_kwargs,
+ )
async def stream_speech_async(
self,You can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit 1e335a2. Configure here.
…peError Bugbot follow-ups after initial voice_id + timeout + async fixes: - stream_speech strips extra kwargs that unary tolerates - registry adapter uses .get (not .pop) so shared option dicts survive - language/lang alias forwarded into stream_speech_sync
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements OpenVoiceUI upgrade round 2 Priority A–D against
main(public fork hygiene: no secrets,XAI_API_KEYenv only).A — Streaming Grok TTS (TTFA)
wss://api.x.ai/v1/ttsonGrokProvider.stream_speech_sync/stream_speech/stream_speech_asyncoptimize_streaming_latency(default 1 on stream path for lower time-to-first-audio)connect_fnfor mocked unit tests (tests/test_grok_tts_stream.py)providers/tts/grok_provider.pydelegatesstream_speechB — Grok STT
providers/stt/grok_provider.py→POST https://api.x.ai/v1/sttmultipart (file last)grok; missing-key + happy-path mocks (tests/test_grok_stt_provider.py)docs/features/grok-stt.mdC — Experimental realtime bridge (FLAG OFF)
services/grok_realtime_bridge.pyOPENVOICEUI_GROK_REALTIME(default off)wss://api.x.ai/v1/realtime?model=grok-voice-latestsession.update(voice=eve,server_vad)tests/test_grok_realtime_bridge.pyD — Dev smoke
scripts/smoke_grok_tts.pywrites under/tmponly, exit 0/1 (soft-skip without key)Test plan
pytest -qon Grok suites → 29 passed/tmp/openvoiceui-grok-tts-smoke-*.mp3Security
Note
Medium Risk
New outbound xAI integrations (STT multipart, TTS/realtime WebSockets) and env-gated realtime helpers; keys stay server-side but misconfiguration or enabling the bridge increases exposure surface.
Overview
Expands xAI / Grok voice support under a single
XAI_API_KEY: batch STT, WebSocket streaming TTS for lower time-to-first-audio, plus an opt-in server-side realtime bridge.Grok STT adds registry id
grok(GrokSTTProvider) with multipartPOST /v1/stt(file field last), word-level segments, andhttpxas a dependency. Streaming TTS on canonicalGrokProvideruseswss://api.x.ai/v1/ttsviastream_speech_sync/stream_speech/stream_speech_async, withoptimize_streaming_latencyand injectableconnect_fnfor tests; the TTS registry adapter delegatesstream_speech.Experimental realtime lives in
services/grok_realtime_bridge.py, gated byOPENVOICEUI_GROK_REALTIME(default off), targetingwss://api.x.ai/v1/realtimewith minimalsession.updateand a helper for ephemeral client secrets—server-only, no browser raw keys.Docs (
.env.example, README,grok-tts/ newgrok-stt),scripts/smoke_grok_tts.py(/tmpoutput), and mocked pytest suites cover the new paths.Reviewed by Cursor Bugbot for commit b3a38d2. Bugbot is set up for automated code reviews on this repo. Configure here.