Deploy Qwen3.6-35B-A3B (Q4_K_XL) + MTP speculative decoding on a single NVIDIA L4 24GB — GCP g2-standard-8 — via the official llama.cpp Docker image. Decode-optimized to ~91–99 tok/s (min ~91 chat, max ~99 math), lossless (full GPU residency + ECC-off).
gcp moe quantization inference-optimization mixture-of-experts large-language-models llm llama-cpp gpu-inference local-llm qwen gguf speculative-decoding qwen3 multi-token-prediction nvidia-l4
-
Updated
Jun 29, 2026 - Shell