Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

An Efficient and Versatile Inference Engine for Distributed LLM Serving

Why is LLM inference slow — and how do you make it fast? A hands-on, first-principles course: roofline → KV cache → quantization → parallelism → vLLM/SGLang, with GPU labs on open models.

A deterministic PyTorch autograd verification trap for catching silent KV-cache routing and block-alignment failures in vLLM and SGLang serving infrastructure.

LLM inference kernels from scratch in Triton: KV cache, FlashAttention, PagedAttention, RMSNorm, RoPE, SwiGLU, and benchmarks.

Deadline-aware KV-cache scheduling for protecting decode-critical request-state under long-context LLM inference pressure.

Empirical benchmarking harness mapping the boundary between compute saturation and PagedAttention KV-cache preemption cascades under streamed test-time compute scaling.

A minimal LLM inference engine implementing PagedAttention-style KV cache management on NanoGPT. Based on the "Efficient Memory Management for Large Language Model Serving with PagedAttention" paper.

What to consider when running AI Inference at scale on Kubernetes

LLM inference serving prototype with continuous batching, paged KV-cache management, benchmark telemetry, and C++/ONNX Runtime gRPC backend scaffold.

From-scratch model of an LLM serving engine's systems core: paged KV-cache, continuous batching, preemption, and prefix caching — GPU-free, with reproducible benchmarks.