Is this a new feature request?
Wanted change
Save GPU VRAM when not in use. VRAM is quite valuable resource and should be possible to configure a keep_alive value. For example with Ollama it is configured like this:
keep_alive=-1 keeps model in memory indefinitelykeep_alive=0 unloads model after each usekeep_alive=60 keeps the model in memory for 1 minute after use
This can be a environment variable, default to -1 to not be a breaking change for anyone.
Reason for change
Right now the model is loaded into memory as soon as the container starts, and stays there even when not in use.
Proposed code change
No response
Is this a new feature request?
Wanted change
Save GPU VRAM when not in use. VRAM is quite valuable resource and should be possible to configure a
keep_alivevalue. For example with Ollama it is configured like this:keep_alive=-1keeps model in memory indefinitelykeep_alive=0unloads model after each usekeep_alive=60keeps the model in memory for 1 minute after useThis can be a environment variable, default to
-1to not be a breaking change for anyone.Reason for change
Right now the model is loaded into memory as soon as the container starts, and stays there even when not in use.
Proposed code change
No response