int4

SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model compression techniques on PyTorch, TensorFlow, and ONNX Runtime

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

row-major matmul optimization

An innovative library for efficient LLM inference via low-bit quantization

Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR (a DeepSeek-OCR-derived 3B MoE VLM). Five-model zoo, custom int8 kernels, no ML framework, no Python, no GPU.

[ICML 2026] GRACE: deployable INT4 VLM via quantization-aware distillation.

Hand-written NVFP4 W4A16 CUDA kernels for Volta

414 KB WASM runtime for Needle a 14M-parameter tool-calling transformer. Runs in browser, Cloudflare Workers, and Node.js. No backend required.

GLM-5.2, a 744 billion parameter mixture of experts model, in a pure C inference engine: quantized to int4, experts streamed from disk, deployed and benchmarked. Generates in 16 GB of RAM.

Serving 4-bit Qwen3.8-27B on a single DGX Spark (GB10): 75 tok/s single-stream, 246 tok/s aggregate at 8-way concurrency. NVFP4 vs MixedInt4-AutoRound vs the FP8 baseline, measured on one harness — including why the quantization advantage collapses to +0.2% by c16.

Optimized local serving engine for Kimi-Linear-48B: INT4 quantizer, fused decode kernels for a measured 3.18x, and an OpenAI-compatible server. Ships with k3, a bridge that detects the client per request so Claude Code, Codex, Cline, Aider and opencode all work unchanged.

🧬🔍 Vecgo is a pure Go, embeddable, hybrid vector database designed for high-performance production workloads. It combines commit-oriented durability with HNSW + DiskANN indexing for best-in-class performance.

⚡️ The fastest way to run local LLMs on Apple Silicon — sub-second model loads, beats Ollama on throughput, tail latency, and full-response time. OpenAI/Ollama-compatible. No cloud, no API keys.

Run Qwen3.8 Flash Next on dual RTX 3090 (2x24GB) + 128GB RAM: 256K context, 75.6 tok/s at 258K input + 4K output (3 runs), 135 tok/s selected warm short-prompt decode. Pinned vLLM, CPU offload, MTP, P2P guide. Dual RTX 4090 testing notes (unvalidated).

Run Qwen3.6-27B on four Tesla V100s at 366 tok/s using hand-written NVFP4 CUDA kernels and chain-MTP speculation.

Research and training stack for AVA — a tool-using, memory-aware virtual assistant targeting 4 GB VRAM. Spans custom transformers, verifier-RL, external memory, multi-domain benchmarks, and Gemma 4 inference optimization.

Train and serve MoE models that do not fit in VRAM: fused 4-bit experts, QLoRA, CPU/NVMe offload, and fast inference on consumer NVIDIA GPUs.

rust library to write integer types of any bit length into a buffer - from `i1` to `i64`.

Low-bit LLM inference engine in Rust + CUDA. Converts models to a 4-bit .wstone format (3.75x smaller) and runs them on consumer GPUs. Built for memory-bandwidth-bound decode on Turing, where bytes per weight — not TOPS — sets token throughput.

VKNN — Vulkan Neural Network: dependency-free C++17 on-device inference engine that runs ONNX models entirely on Android GPUs. CNNs, YOLO, LLMs, VLMs, 3D Gaussian Splatting; int4 quantization; compile once to .vxm.