fix: pre-flight disk space + cap GPU offload so small-GPU runs don't hang#95
Open
rjckkkkk wants to merge 2 commits into
Open
fix: pre-flight disk space + cap GPU offload so small-GPU runs don't hang#95rjckkkkk wants to merge 2 commits into
rjckkkkk wants to merge 2 commits into
Conversation
…te VRAM
llama.cpp's default n_gpu_layers=999 ("offload every layer") assumes the
model fits in VRAM. On a discrete GPU with less VRAM than the model needs
(e.g. qwen3.5-9b on a 4GB RTX 2050) it forces an impossible allocation and
CUDA-OOMs at load, surfacing to users as a deployment that never becomes
ready.
Cap the engine-declared offload knob (offload_config_key) to CPU when a
discrete GPU can't hold the model's footprint, and warn with the numbers.
Unified-memory hosts (APU/GB10/Apple) share system RAM with the GPU, so
full offload is correct there and is left untouched. The knob is read from
catalog YAML, so no engine name or config key is hardcoded (INV-1/2).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A model pull streamed straight to disk with no free-space check. On a small disk a multi-GB GGUF could fill the filesystem past k3s's disk-pressure eviction threshold, tainting the node NoSchedule so the pod stayed Pending forever — surfacing only as "Timed out waiting for deployment to be ready". Guard the download: the moment its total size is known (reported by the existing downloader progress, before transfer for HF/ModelScope), abort if it won't fit while preserving a disk-pressure reserve (10% of the volume, 2 GiB floor), returning a clear, fatal, pre-transfer error instead of a disk-filling hang. Adds hal.DiskUsage for cross-platform free/total bytes. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
An open-source user ran
aima run qwen3.5-9b(llamacpp/k3s) on a laptop with a 4 GB RTX 2050 and gotTimed out waiting for deployment to be ready, stuck atscheduling 2%for hours. Two independent product bugs combine here:Qwen3.5-9B-IQ4_NL.gguf(~5 GB) streamed straight to disk with no free-space check, filling the filesystem past k3s's disk-pressure eviction threshold. The node got taintednode.kubernetes.io/disk-pressure:NoSchedule, so the pod never scheduled — the only symptom was the generic timeout. (kubectl describe pod→FailedScheduling ... untolerated taint {disk-pressure}.)llama-server ... --n-gpu-layers 999(offload all layers to GPU).999is a catalog default that assumes the model fits VRAM; on a 4 GB discrete card a 9B model can't, so even after the disk issue it would CUDA-OOM at load and time out again.Fixes (two commits)
1.
fix(resolver)— VRAM-aware offload. When a discrete GPU can't hold the model's footprint (ram_min_mibproxy > per-GPU VRAM), downgrade the engine-declaredoffload_config_keyto CPU and surface a clear warning. It still deploys and works — just on CPU — matching AIMA's graceful-degradation model. Driven entirely by catalog YAML (offload_config_key), so no engine name / config key is hardcoded (INV-1/2). Unified-memory hosts (APU / GB10 / Apple) are untouched — the GPU shares system RAM there, son_gpu_layers=999is correct (this is exactly the verified Strix Halo config). Explicit user overrides are respected.2.
fix(run)— disk pre-flight. The moment a download's total size is known (from the existing downloader progress — reported before transfer for HF/ModelScope), abort if it won't fit while keeping a disk-pressure reserve (10% of the volume, 2 GiB floor), returning a clear fatal error (needs ~X MiB … only Y MiB free … short by ~Z MiB; free up space or choose a smaller model) instead of a disk-filling hang. Adds a small cross-platformhal.DiskUsagewrapper. No new download code (reuses the size the downloader already computes).Tests & verification
internal/knowledge/offload_test.go(downgrade / unified-memory left alone / fitting-GPU left alone / user-override respected / no-offload-key noop / warnings surfaced throughCheckFit),cmd/aima/diskcheck_test.go(fit / disk-pressure-trip / reserve floor / unknown-size no-gate).qwen3.5-9b.yaml: on a simulated 4 GB discrete NVIDIA →engine=llamacpp n_gpu_layers=0+ warning; on unified-memory GB10 →n_gpu_layers=999preserved.go build ./...,go vet, and fullgo test ./...all pass.Not in scope