A minimal LLM inference engine written from scratch in C++. Loads GGUF models, dequantizes Q4_0/Q6_K weights on-the-fly, and generates text through a full transformer forward pass. No dependencies on llama.cpp, PyTorch, or any ML framework have been used to create the engine.
Built to understand every layer and methods of the inference stack: from parsing bytes off disk to producing English one token at a time.
Note
The engine can still be improved by adding more quantization formats, hardware acceleration, and model architectures. Currently, it supports only Q4_0/Q8_0/Q6_K/F16/F32 quantization, is based for CPU inference (with OpenMP/AVX2 optimizations, no GPU acceleration), and is focused on LlaMA-architecture models.
$ ./turtle_cpp "../data/tinyllama-1.1b-chat-v1.0.Q4_0.gguf" --max-tokens 50
Threads: 4
Sampling: greedy
Prompt: "The capital of France is"
a city in the region of France. It is located in the north of Paris, France, and is a city of Paris, and is the capital of Paris, France.<0x0A>190101, and is a French, France,
Prefill: 6 tokens in 779.0 ms (7.7 tok/s)
Generation: 50 tokens in 6585.6 ms (7.59 tok/s)
Benchmarked on AMD Ryzen 5 2600 Six-Core Processor, TinyLlama-1.1B Q4_0, 50-token generation.
| Optimization | tok/s | Speedup |
|---|---|---|
| Baseline (-O0) | 0.29 | 1.0× |
| Compiler (-O2) | 1.05 | 3.6× |
| + OpenMP (8 threads) | 6.93 | 23.9× |
| + AVX2 SIMD | 7.27 | 25.1× |
| llama.cpp (reference) | ~20 | ~69× |
Everything from raw bytes on disk to generated text — no external ML libraries:
GGUF parser: reads the binary file format: header, metadata key-value pairs, tensor info array, and alignment. Supports version 3. Model configuration (hidden size, number of heads, RoPE theta, etc.) is extracted from metadata keys, making the same code work for any LLaMA-family model.
Quantized tensor access: model weights stay in their compressed format in memory (via mmap). No full-model dequantization at load time. Q4_0 weights occupy 0.56 bytes per parameter (7× compression vs FP32), Q6_K uses 0.82 bytes (4.9× compression). Tensor data is accessed through zero-copy pointers into the memory-mapped file.
Fused dequantization: dot products between FP32 activations and quantized weights are computed in a single pass. Each quantized block is dequantized into CPU registers, multiplied with the input, and accumulated, the dequantized values never touch main memory. This is what makes quantized inference memory-efficient.
Transformer forward pass:
- RMSNorm with double-precision accumulation for numerical stability
- Rotary Position Embeddings (RoPE) with configurable theta (10k for TinyLlama, 500k for Llama 3.2)
- Multi-head attention with Grouped Query Attention (GQA): 32 query heads share 4 KV heads
- KV cache: pre-allocated flat buffer, O(1) per-token append, supports sequences up to context length
- SwiGLU MLP: gated activation with SiLU nonlinearity
- Residual connections after attention and MLP blocks
Text generation: autoregressive loop with prefill (process prompt, populate KV cache) and generation (produce one token per step) phases. Supports greedy decoding (argmax) and temperature + top-p sampling.
turtle.cpp/
├── include/
│ ├── gguf.hpp # GGUF file format types + parser declaration
│ ├── quantize.hpp # block structs, dequant, vec_dot, matmul
│ ├── ops.hpp # rmsnorm, rope, softmax, silu, add
│ ├── model.hpp # Model/KVCache structs, forward pass
│ └── cli.h # argument parsing
├── src/
│ ├── gguf.cpp # GGUF parser: header, metadata, tensor info, mmap
│ ├── quantize.cpp # Q4_0/Q6_K/Q8_0 dequantization + fused dot products
│ ├── ops.cpp # transformer building blocks
│ ├── model.cpp # model loading, attention, MLP, forward, generate
│ ├── cli.cpp # CLI argument parsing
│ └── main.cpp # entry point
├── tests/
│ └── test_ops.cpp # GTest: all ops verified against PyTorch (libtorch)
├── CMakeLists.txt
└── README.md
git clone https://github.com/schwp/turtle.cpp.git
cd turtle.cpp
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)Requirements: CMake
Download a GGUF model and run the inference engine. Here are some examples if you want to try it out :
./turtle_cpp tinyllama.gguf --max-tokens 50
./turtle_cpp tinyllama.gguf --max-tokens 100 --sample --temperature 0.8 --top-p 0.9
./turtle_cpp tinyllama.gguf --threads 8 --max-tokens 50mmap is the entire memory strategy. The model file is memory-mapped and tensor pointers index directly into it. No malloc, no memcpy, no conversion pass. The OS pages in data on demand. This is how a 637 MB model loads in milliseconds, you're not reading the file, you're just creating a virtual address mapping.
Dequantize inside the dot product, not before it. A full FP32 copy of TinyLlama would be 4.4 GB, larger than the GPU memory on many machines. By dequantizing block-by-block during the matmul inner loop, the FP32 values exist only in CPU registers for one clock cycle. Memory usage equals the compressed file size, not the uncompressed model size.
The Q6_K struct layout does not match the logical element ordering. The 256
elements are split into two groups of 128, with low and high nibbles of ql mapping
to four interleaved sub-groups. Getting this layout wrong produces valid-looking
but numerically incorrect logits, the model generates real words but not coherent
text. This was the last bug fixed in this project and the hardest to diagnose.
OpenMP gives 7× for one pragma line. The matmul output loop is now in parallel,
each output element is an independent dot product. Adding #pragma omp parallel for
and -fopenmp gave more speedup than AVX2 SIMD.
The Rule of Five matters for real. The GGUF file handle owns a mmap pointer.
Without a move constructor and deleted copy constructor, returning a GGUFFile
from a function silently copies the pointer and the temporary's destructor frees
the mapping, every tensor pointer becomes dangling. This produced a segfault that
took longer to find than the entire forward pass implementation.
cd build
ctest --output-on-failureAll ops (RMSNorm, RoPE, softmax, SiLU) are tested against PyTorch via libtorch (automatically downloaded using CMake) to verify numerical correctness. RoPE is specifically tested for the rotated-half layout used by LLaMA 2/3, position-0 identity, and magnitude preservation.
This project is a learning exercise, not a production tool. Here are the main differences:
| Feature | turtle.cpp | llama.cpp |
|---|---|---|
| Performance | ~7 tok/s | ~20+ tok/s |
| GPU support | No | CUDA, Metal, Vulkan |
| Batch processing | No | Yes |
| Memory allocator | std::vector | Custom pool allocator |
| SIMD | Basic AVX2 | AVX-512, VNNI, ARM NEON |
| Quantization formats | Q4_0, Q8_0, Q6_K | 20+ formats |
| Tokenizer | Decode only | Full BPE encode/decode |
| Model support | LLaMA family only | LLaMA, Mistral, Phi, Gemma, ... |
The value of this project is not competing with llama.cpp, it's to understand what llama.cpp does under the hood and why each optimization matters, and how it could be helpful for some companies.
- GGUF format specification
- gguf-tools project
- ggml project
- LLaMA 2 paper - Touvron et al.
- RoPE - Rotary Position Embedding - Su et al.
- llama.cpp
- llama2.c - Andrej Karpathy
- Pierre SCHWEITZER (schwp)