Hardware-aware ctx_size clamp + partner model context windows#92
Closed
rjckkkkk wants to merge 2 commits into
Closed
Hardware-aware ctx_size clamp + partner model context windows#92rjckkkkk wants to merge 2 commits into
rjckkkkk wants to merge 2 commits into
Conversation
…text A high catalog/user ctx_size (e.g. the 128000 default we now want for agent clients) would OOM llama-server at load on memory-constrained machines — AIMA previously passed ctx_size through unchanged (estimateVRAM ignores the KV cache; CheckFit only adjusts vLLM/SGLang gpu_memory_utilization, never llama.cpp ctx). On deploy, for llama.cpp GGUF models, size the context to the hardware: - Read the model's real architecture from the GGUF header (new model.ReadKVArch: block_count / head_count_kv / head_dim / context_length) and compute exact f16 KV bytes/token — not a guess. - Clamp ctx_size down so weights + projector + KV cache fit usable memory (GPU VRAM for discrete GPUs, RAM minus an OS reserve for unified/CPU hosts), and cap it at the model's trained context. Only ever lowers; never raises. - Graceful no-op when the GGUF arch can't be read or memory is unknown. So a high default degrades gracefully: big boxes get the full context, small ones auto-shrink to what fits instead of failing. Bump Qwen2.5-VL-3B default ctx_size to 128000 (its trained max) now that it's memory-safe; the agent floor (~9.3K of MCP tool schemas) always fits where the model fits. clampContextForMemory / usableMemoryMiB are pure and unit-tested; verified on the Strix Halo rig (request 200000 → clamped to trained 128000). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…en3-Embedding-4B Set each model's catalog default ctx_size to its full trained context (verified loading + serving on the Strix Halo iGPU, 2026-06-16): - glm-4.7-flash llamacpp variant: ctx_size 8192 -> 202752. - qwen3.6-35b-a3b: add a universal llamacpp GGUF variant (ctx_size 262144) + gguf source + scan-name aliases (Qwen3.6-35B-A3B-UD-Q4_K_M, qwen3.6-35b-a3b-q4_k_m); it previously had only vLLM/Blackwell variants. - new qwen3-embedding-4b.yaml: type embedding, llamacpp variant with embedding=true (-> llama-server --embedding) + ctx_size 40960; serves /v1/embeddings (2560-dim). Deploy-only — embedding models are not written into OpenClaw config by sync. These large ctx defaults are memory-safe via the deploy-time auto-clamp. Also fix usableMemoryMiB to prefer GPU VRAM over system RAM: an all-offloaded llama.cpp model is bounded by GPU memory, and on unified-memory APUs the OS-visible RAM is under-reported (Strix Halo: ~32GB OS vs ~110GB iGPU). The clamp now uses the iGPU pool, so it won't wrongly shrink contexts on such APUs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Collaborator
Author
|
Superseded by the consolidated #(baiying-integration-2026-06) PR which straightens the whole 2026-06 stack into one mergeable develop PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
model.ReadKVArch: block_count / head_count_kv / head_dim / context_length) and clampctx_sizeto fit detected memory (weights + projector + KV cache) and the model's trained context. Only ever lowers; graceful no-op when arch/memory unknown. So a large catalog default degrades gracefully (big boxes get full context, constrained boxes auto-shrink instead of OOM-ing).usableMemoryMiBprefers GPU VRAM over system RAM — an all-offloaded llama.cpp model is bounded by GPU memory, and on unified-memory APUs Win32 under-reports OS RAM (Strix Halo: ~32 GB OS vs ~110 GB iGPU). The clamp now sizes against the iGPU pool.glm-4.7-flashllamacpp variant ctx_size 8192 → 202752qwen3.6-35b-a3b: new universal llamacpp GGUF variant (ctx 262144) + gguf source + scan-name aliases (Qwen3.6-35B-A3B-UD-Q4_K_M,qwen3.6-35b-a3b-q4_k_m)qwen3-embedding-4b.yaml: type embedding,default_config.embedding: true→ llama-server--embedding, ctx 40960, serves/v1/embeddings(2560-dim; deploy-only)qwen2.5-vl-3b-instructctx_size → 128000 (memory-safe via the clamp)Test
clampContextForMemory/usableMemoryMiBpure + unit-tested (cmd/aima/ctxfit_test.go);go vetclean./v1/embeddings; request 200000 → clamped to trained 128000.🤖 Generated with Claude Code