CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

Multimodal LLM inference gateway with KV-cache-aware routing and LMCache offload. OpenAI-compatible, benchmarked on GPUs.

KV-cache-aware LLM inference mesh on a single 6 GB GPU. 2× vLLM + LMCache shared KV + prefix-aware router on k3s, observed with Cilium/Hubble eBPF. Every number measured, every breakage documented.

Reproducible LLM KV-cache tiered-storage benchmark on AMD MI308X: LMCache parallel-read patch, load clients, orchestration, measured results

Collection of benchmarks, benchmark drivers, harnesses, plotting, and various scripts for KV cache offloading in vLLM.

Benchmarking LMCache under simulated RTT

Provide fast, memory-efficient caching for language models to improve response times and reduce redundant computations.