eager_warm_up_for_inference() causing weird memory allocation

[GNBS711]

Hey!

I found what appears to be a VRAM allocation issue when using OmniVoice together with Whisper validation.

Tested on: Win11, model OmniVoice + Whisper large-v3-turbo (CUDA)

After some debugging, I traced the behavior to:
model_manager.py

if should_stt:
    Stt.eager_warm_up_for_inference()

When eager_warm_up_for_inference() is enabled, total GPU memory usage stays around 9-10 GB during the entire generation process. If I disable the call and leave STT lazy-loaded, memory usage drops to about 5 GB total (OmniVoice + Whisper validation still works normally).

I suspect the warm-up transcription triggers additional CTranslate2/CUDA allocations that remain resident until the application is closed. Could this be related to Faster-Whisper / CTranslate2 keeping inference buffers alive after the warm-up transcription?

Let me know if you'd like additional logs or testing. Thanks again for the project!

[zeropointnine]

If VRAM stayed in the 5 GB range after commenting out Stt.eager_warm_up_for_inference(), then the Whisper model must not been loaded at all during that test run, since OmniVoice takes up about 5 GB by itself.

I'm guessing (hoping?) that you were running "Realtime audiobook playback" rather than "Generate audio segments". In the "realtime" mode, it only runs triggers Whisper once the audio buffer duration exceeds 60 seconds, so if that threshold was not reached, it would not have loaded Whisper (with eager initialization disabled).

[GNBS711]

I encountered this issue when running "Generate audio segments".
Whisper model still loads and runs even without "Stt.eager_warm_up_for_inference()". It just takes a few more seconds to load.

But my guess was wrong. It doesn't seem to be related to "Stt.eager_warm_up_for_inference()". I recreated the entire environment e re-downloaded the project from scratch but now I'm having trouble reproducing the results.

I have no idea why OmniVoice+Whisper is allocating this amount (9~10GB) of VRAM. When I generate from my old fork (GNBS711/tts-audiobook-tool) it uses around 5GB VRAM with Whisper transcription.

If you have some time, could you please check the fork and compare the VRAM usage with OmniVoice+Whisper (faster-whisper-large-v3-turbo)?

In the meantime, I'll keep investigating on my side.

[GNBS711]

Ok, so I managed to reproduce the issue.
I used Omnivoice+Whisper-large-v3-turbo in each case (the transcriptions were generated correctly in all of them).

^This first screenshot shows generation using the old fork, allocating about 5GB VRAM throughout the entire generation process. ^This second screenshot shows generation using the current project setup, allocating about 9-10GB VRAM throughout the entire generation process. "Stt.eager_warm_up_for_inference()" is probably loading both the TTS model+Whisper at the same time (still doesn't explain ~10GB allocation).

So I added "torch.cuda.empty_cache()" between the first generation and the transcription step (generate_util.py):

# Should skip or not
        skip_reason = Stt.should_skip(state, is_skip_reason_buffer)
        if not skip_reason:
            printt(f"{COL_DEFAULT}Transcribing audio...", end="") # gets overwritten

            torch.cuda.empty_cache()

^It still allocates about 9-10GB VRAM during the first sentence generation, but VRAM usage drops to ~4.5GB after "torch.cuda.empty_cache()" is called.

So, to avoid offloading to RAM during the first generation, I disabled "Stt.eager_warm_up_for_inference()" in model_manager.py...

^Finally, VRAM usage jumps to ~8GB (TTS loading) and drops below ~4GB after "torch.cuda.empty_cache()". Whisper is then loaded (~4.5GB). The Whisper transcription was also generated successfully (apparently through lazy loading).

Now, why loading the TTS requires significantly more VRAM compared to the behavior of the old fork, I have no idea.