feat(vibevoice-cpp): true streaming TTS (TTSStream via vv_capi_tts_stream)#10764
Merged
Conversation
Replaces the synth-to-tempfile TTSStream hack with a real streaming path: binds the new vv_capi_tts_stream callback ABI via a single reusable purego callback (CGO_ENABLED=0-safe, no runtime/cgo), copies each int16 PCM window into the gRPC results channel after the streaming WAV header. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
…measurement Gated behind VIBEVOICE_IT=1, this Ginkgo spec dlopens the engine .so and drives the exact Go->purego->C TTSStream/TTS path against the real vibevoice-realtime-0.5B model. It measures time-to-first-audio for the streaming path versus the batch path and asserts the streaming win: 44-byte WAV header first, >=2 PCM windows, non-silent audio, and TTFA < total_stream. Without the env var the spec skips so CI and normal go test are unaffected. Measured: TTFA 2.38s vs batch deliver-time 39.96s (first audio in 5.9% of the batch time, ~17x faster), 18 stream chunks, non-silent 24kHz PCM. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Bumps VIBEVOICE_CPP_VERSION to the streaming-decoder engine commit that adds vv_capi_tts_stream (localai-org/vibevoice.cpp#8). Re-pin to the merged master commit once that PR lands. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
localai-org/vibevoice.cpp#8 merged to master as 000e372; move the pin off the PR branch commit onto the merged master commit. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-opus-4-8 [Claude Code]
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| if activeStream == nil || samples == nil || n <= 0 { | ||
| return 0 | ||
| } | ||
| pcm := unsafe.Slice(samples, int(n)) |
golangci-lint errcheck flagged the unchecked fmt.Fprintf calls that print the streaming TTFA headline. Build the report once with fmt.Sprintf and write it per destination with an explicitly discarded error, matching the GinkgoWriter reporting idiom used by the other backend tests. Signed-off-by: Ettore Di Giacinto <[email protected]> Assisted-by: Claude:claude-fable-5 [Claude Code]
mudler
approved these changes
Jul 9, 2026
mudler
added a commit
that referenced
this pull request
Jul 10, 2026
#10766) The vibevoice transcription e2e hangs until the go test timeout when the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to fit), and curl's --retry restarts from byte zero, so no attempt ever makes forward progress. This killed the job twice on PR #10764 and previously forced skipping it on release tags (#10567). Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s over --speed-time 120s) and resume from the bytes already on disk with -C -, retrying from Go because curl does not re-evaluate the resume offset on its internal retries. Resume against the HF Xet CDN was verified by killing a transfer mid-flight and confirming the next invocation appended (114 MB -> 235 MB, GGUF magic intact). Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default 30m) and raise it to 120m for the vibevoice transcription wrapper: a 10 GB download plus 25 specs does not fit in 30m even on a good day, and the job-level GHA timeout there is already 150m. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <[email protected]> Co-authored-by: Ettore Di Giacinto <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
True streaming for the vibevoice-cpp TTS backend
The
vibevoice-cppbackend advertised streaming (TTSStream) but faked it: it synthesized the entire clip to a tempfile via the file-onlyvv_capi_tts, then chunked the finished PCM. Time-to-first-audio equaled full-synthesis time, defeating the point of a "Realtime" model.This PR wires the backend to real streaming. The heavy lifting is in the engine (localai-org/vibevoice.cpp#8), which adds a bit-exact streaming acoustic decoder and a
vv_capi_tts_streamcallback ABI. Here we:vv_capi_tts_streamvia a single reusable purego callback created once at init (purego has a process-global callback limit and cannot free them).CGO_ENABLED=0-safe: noruntime/cgo.TTSStream: emit the streaming WAV header, then push each int16 PCM window (copied out of the transient C buffer, little-endian) into the gRPC results channel as the engine produces it. The synth-to-tempfile hack is gone.The non-streaming
TTS()path is unchanged. Scope is the realtime-0.5B model; a 1.5B model routed to streaming returns a clean error.Results (real model, this box is CPU-only)
End-to-end through the real
Load->TTSStreampath withmicrosoft/VibeVoice-Realtime-0.5B(q8_0):CPU-only synthesis is not real-time for the whole clip (~40 s for the paragraph), but streaming cuts perceived latency from 40 s to 2.4 s; on GPU the per-window rate clears real-time. The win is time-to-first-audio, which is exactly what streaming is for.
Validation
vv_capi_tts_streamint16 byte-identical to file PCM.VIBEVOICE_IT=1) asserting valid streaming WAV header, >=2 chunks, and TTFA < total.Note on the pin
VIBEVOICE_CPP_VERSIONcurrently points at the engine PR's branch commit (fba2449). Re-pin to the mergedmastercommit once localai-org/vibevoice.cpp#8 lands (or let thebump_depsworkflow do it).Assisted-by: Claude:claude-opus-4-8 [Claude Code]
🤖 Generated with Claude Code