Skip to content

fix: pre-flight disk space + cap GPU offload so small-GPU runs don't hang#95

Open
rjckkkkk wants to merge 2 commits into
developfrom
fix/preflight-disk-and-vram-offload
Open

fix: pre-flight disk space + cap GPU offload so small-GPU runs don't hang#95
rjckkkkk wants to merge 2 commits into
developfrom
fix/preflight-disk-and-vram-offload

Conversation

@rjckkkkk

@rjckkkkk rjckkkkk commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

An open-source user ran aima run qwen3.5-9b (llamacpp/k3s) on a laptop with a 4 GB RTX 2050 and got Timed out waiting for deployment to be ready, stuck at scheduling 2% for hours. Two independent product bugs combine here:

  1. No disk pre-check before model download. The Qwen3.5-9B-IQ4_NL.gguf (~5 GB) streamed straight to disk with no free-space check, filling the filesystem past k3s's disk-pressure eviction threshold. The node got tainted node.kubernetes.io/disk-pressure:NoSchedule, so the pod never scheduled — the only symptom was the generic timeout. (kubectl describe podFailedScheduling ... untolerated taint {disk-pressure}.)
  2. Full GPU offload forced on a GPU too small. The generated command was llama-server ... --n-gpu-layers 999 (offload all layers to GPU). 999 is a catalog default that assumes the model fits VRAM; on a 4 GB discrete card a 9B model can't, so even after the disk issue it would CUDA-OOM at load and time out again.

Fixes (two commits)

1. fix(resolver) — VRAM-aware offload. When a discrete GPU can't hold the model's footprint (ram_min_mib proxy > per-GPU VRAM), downgrade the engine-declared offload_config_key to CPU and surface a clear warning. It still deploys and works — just on CPU — matching AIMA's graceful-degradation model. Driven entirely by catalog YAML (offload_config_key), so no engine name / config key is hardcoded (INV-1/2). Unified-memory hosts (APU / GB10 / Apple) are untouched — the GPU shares system RAM there, so n_gpu_layers=999 is correct (this is exactly the verified Strix Halo config). Explicit user overrides are respected.

2. fix(run) — disk pre-flight. The moment a download's total size is known (from the existing downloader progress — reported before transfer for HF/ModelScope), abort if it won't fit while keeping a disk-pressure reserve (10% of the volume, 2 GiB floor), returning a clear fatal error (needs ~X MiB … only Y MiB free … short by ~Z MiB; free up space or choose a smaller model) instead of a disk-filling hang. Adds a small cross-platform hal.DiskUsage wrapper. No new download code (reuses the size the downloader already computes).

Tests & verification

  • Unit tests: internal/knowledge/offload_test.go (downgrade / unified-memory left alone / fitting-GPU left alone / user-override respected / no-offload-key noop / warnings surfaced through CheckFit), cmd/aima/diskcheck_test.go (fit / disk-pressure-trip / reserve floor / unknown-size no-gate).
  • Verified end-to-end against the real embedded catalog + actual qwen3.5-9b.yaml: on a simulated 4 GB discrete NVIDIA → engine=llamacpp n_gpu_layers=0 + warning; on unified-memory GB10 → n_gpu_layers=999 preserved.
  • go build ./..., go vet, and full go test ./... all pass.

Not in scope

  • Precise partial GPU offload (e.g. "12 of 40 layers") — the catalog carries no per-model layer count, so any nonzero guess risks OOM; CPU fallback is the safe floor. Adding layer-count metadata could enable this later.
  • Engine-image pull disk checks (this addresses the reported model-download case).

Codex and others added 2 commits July 7, 2026 11:14
…te VRAM

llama.cpp's default n_gpu_layers=999 ("offload every layer") assumes the
model fits in VRAM. On a discrete GPU with less VRAM than the model needs
(e.g. qwen3.5-9b on a 4GB RTX 2050) it forces an impossible allocation and
CUDA-OOMs at load, surfacing to users as a deployment that never becomes
ready.

Cap the engine-declared offload knob (offload_config_key) to CPU when a
discrete GPU can't hold the model's footprint, and warn with the numbers.
Unified-memory hosts (APU/GB10/Apple) share system RAM with the GPU, so
full offload is correct there and is left untouched. The knob is read from
catalog YAML, so no engine name or config key is hardcoded (INV-1/2).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
A model pull streamed straight to disk with no free-space check. On a small
disk a multi-GB GGUF could fill the filesystem past k3s's disk-pressure
eviction threshold, tainting the node NoSchedule so the pod stayed Pending
forever — surfacing only as "Timed out waiting for deployment to be ready".

Guard the download: the moment its total size is known (reported by the
existing downloader progress, before transfer for HF/ModelScope), abort if
it won't fit while preserving a disk-pressure reserve (10% of the volume,
2 GiB floor), returning a clear, fatal, pre-transfer error instead of a
disk-filling hang. Adds hal.DiskUsage for cross-platform free/total bytes.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant