Linux was not designed with token generation as a first-class workload.
This project explores what changes when the kernel, runtime, and observability stack become inference-aware.
linux-kernel-inference-fastpath is a systems research and engineering project focused on reducing TTFT — time to first token — improving decode tokens/sec, and lowering inference jitter on Linux systems.
The core idea is simple:
Measure first. Patch second.
Most inference optimization today happens above the operating system:
runtime batching
quantization
CUDA / ROCm / Vulkan kernels
model formats
CPU governor tuning
NUMA pinning
hugepage tuning
thread pinning
Those are useful, but Linux still sees an inference server as a generic process with threads, sockets, memory mappings, GPU driver calls, and page faults.
LLM inference is not generic.
It has distinct phases:
request arrival
tokenization
prefill
first-token generation
steady decode
KV-cache growth
streaming response
session idle
session resume
Each phase has different requirements for:
scheduling
memory placement
power state
network priority
GPU submission
NUMA locality
page-fault behavior
KV-cache residency
This project explores a Linux inference fast path built around observability, userspace policy, runtime hints, and eventually experimental kernel primitives.
Modern LLM serving is sensitive to more than raw GPU FLOPs.
Real-world inference performance can be hurt by:
scheduler jitter
slow CPU frequency ramp
page faults during TTFT
bad NUMA placement
GPU launch-thread wakeup delay
incorrect IRQ placement
memory reclaim of important pages
KV-cache memory pressure
model mmap/page-cache behavior
network-to-first-token latency
The inference runtime often knows what is happening:
this request is in TTFT phase
this thread is token-critical
this memory region is model weights
this memory region is hot KV cache
this memory region is cold KV cache
this GPU is closest to this NUMA node
this socket is first-token critical
But Linux does not receive these semantic hints.
The goal of this project is to define and prototype the minimum OS-level mechanisms needed for inference runtimes to communicate intent to Linux.
This repository is currently in the architecture and design phase.
It contains or is intended to contain:
design documents
subsystem proposals
API sketches
roadmap notes
eBPF observability plans
userspace daemon design
runtime hint design
experimental kernel primitive proposals
It does not yet claim to contain:
production kernel patches
validated benchmark results
upstreamable Linux changes
complete runtime integrations
a replacement for vLLM / TensorRT-LLM / llama.cpp
The first implementation milestone is:
infertrace
infertrace is an eBPF-based tool for decomposing TTFT and token latency into scheduler, memory, NUMA, IRQ, CPU power, and runtime components.
┌──────────────────────────────┐
│ vLLM / llama.cpp / SGLang │
│ TensorRT-LLM / Ollama │
└──────────────┬───────────────┘
│ inference hints
▼
┌──────────────────────────────┐
│ libinferhint.so │
│ runtime phase / memory hints │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ inferd │
│ userspace policy controller │
└──────────────┬───────────────┘
│
┌───────────────┼────────────────────────────┐
▼ ▼ ▼
┌─────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Scheduler │ │ Memory Manager │ │ IRQ / NUMA / IO │
│ hints │ │ KV/model hints │ │ topology │
└─────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────┴──────────────┬─────────────┘
▼
┌─────────────────────────┐
│ Linux Kernel │
│ inference fast path │
└─────────────────────────┘
▲
│ eBPF events
▼
┌─────────────────────────┐
│ infertrace │
│ TTFT / jitter profiler │
└─────────────────────────┘
infertrace is the first planned artifact.
It is an eBPF-based observability tool for measuring where inference latency is actually spent.
Planned signals:
scheduler delay
page faults
NUMA misses
thread migration
CPU frequency state
C-state wakeup behavior
IRQ placement
GPU launch-thread wakeups
network receive-to-first-token timeline
memory reclaim events
Example target output:
Request 42 TTFT: 118.4 ms
Breakdown:
socket receive 0.7 ms
userspace queue 4.1 ms
tokenizer 9.8 ms
model page faults 0.0 ms
KV allocation 2.3 ms
scheduler delay 5.7 ms
GPU launch delay 3.1 ms
prefill GPU time 81.6 ms
first decode GPU time 9.4 ms
socket send 0.3 ms
Kernel warnings:
GPU launch thread migrated across NUMA nodes: yes
CPU governor was powersave during request arrival: yes
17 minor page faults during TTFT
NIC IRQ and GPU IRQ on same core
decode thread preempted 4 times
inferd is a userspace policy daemon for applying inference-aware system policy using existing Linux mechanisms.
Planned controls:
cgroups
cpusets
NUMA placement
CPU governor
IRQ affinity
hugepage policy
page warmup
mlock policy
decode-core reservation
GPU/NUMA topology binding
Example future usage:
inferd run \
--app "vllm serve meta-llama/Llama-3.1-8B-Instruct" \
--latency-mode ttft \
--gpu 0 \
--numa auto \
--pin-model \
--reserve-decode-core \
--irq-steer \
--hugepages madviselibinferhint is a proposed runtime hint library.
It allows inference runtimes to communicate semantic information to inferd and, eventually, to kernel primitives.
Planned hints:
request begin
request end
first token emitted
TTFT phase
prefill phase
decode phase
token-critical thread role
model-weight memory region
hot KV memory region
cold KV memory region
shared-prefix KV region
Example API sketch:
typedef enum {
INFER_PHASE_IDLE,
INFER_PHASE_TTFT,
INFER_PHASE_PREFILL,
INFER_PHASE_DECODE,
INFER_PHASE_COOLDOWN
} infer_phase_t;
typedef enum {
INFER_ROLE_FRONTEND,
INFER_ROLE_TOKENIZER,
INFER_ROLE_GPU_LAUNCH,
INFER_ROLE_DECODE_SCHEDULER,
INFER_ROLE_KV_PREFETCH,
INFER_ROLE_BACKGROUND
} infer_role_t;
int infer_set_phase(infer_phase_t phase);
int infer_set_thread_role(infer_role_t role);
int infer_request_begin(uint64_t request_id);
int infer_first_token(uint64_t request_id);
int infer_request_end(uint64_t request_id);
int infer_mem_model_weights(void *ptr, size_t len);
int infer_mem_kv_hot(void *ptr, size_t len);
int infer_mem_kv_cold(void *ptr, size_t len);
int infer_mem_prefix_shared(void *ptr, size_t len);Kernel patches should come after measurement.
Candidate experimental primitives:
MADV_INFER_MODEL
MADV_KV_HOT
MADV_KV_COLD
MADV_KV_PREFIX_SHARED
inference cgroup controller
sched_ext inference scheduler
TTFT boost hook
inference-aware hugepage policy
model mmap fast path
KV-aware reclaim/demotion
The project should prefer existing Linux mechanisms first:
eBPF
sched_ext
cgroups
madvise
prctl
cpusets
NUMA APIs
io_uring
perf events
tracepoints
Core-kernel changes should be minimal and justified by measured bottlenecks.
Classify threads by inference role:
INFER_FRONTEND
INFER_TOKENIZER
INFER_PREFILL
INFER_DECODE
INFER_GPU_LAUNCH
INFER_KV_PREFETCH
INFER_BACKGROUND
Prioritize token-critical threads over logging, metrics, and background work.
Possible mechanisms:
sched_ext inference scheduler
prctl role hints
cgroup role classification
soft token deadlines
TTFT means time to first token.
During TTFT phase, the system can temporarily:
raise CPU frequency immediately
avoid deep C-states
prioritize tokenizer and GPU launch threads
prioritize request socket
avoid reclaim on model/KV pages
steer IRQs to the inference domain
After first token, the workload transitions to steady decode mode.
KV cache is not ordinary memory.
Possible memory hints:
madvise(ptr, len, MADV_KV_HOT);
madvise(ptr, len, MADV_KV_COLD);
madvise(ptr, len, MADV_KV_PREFIX_SHARED);
madvise(ptr, len, MADV_KV_DECODE_CRITICAL);
madvise(ptr, len, MADV_KV_COMPRESSIBLE);Hot KV should be protected.
Cold KV can be demoted, compressed, or moved to slower tiers.
Model weights are often mmap'd from files.
The runtime can tell the kernel:
this mapping contains model weights
these pages are accessed in layer order
these pages should be prefetched
these pages should avoid reclaim during serving
Possible primitive:
madvise(model_ptr, len, MADV_INFER_MODEL);or:
mmap(..., MAP_INFER_MODEL);Create an inference locality domain:
GPU
nearest CPU cores
nearest NUMA memory
NIC queue
NVMe queue
model pages
KV pages
GPU launch threads
decode scheduler threads
The goal is to prevent hidden latency from cross-socket memory traffic and poor IRQ placement.
Decode often performs a repeated loop:
launch kernels
wait
launch next token kernels
wait
repeat
The kernel/runtime should reduce CPU-side wakeup and launch jitter.
Target behavior:
GPU completion interrupt
→ wake exact launch thread
→ on correct CPU core
→ with boosted priority
→ with CPU at useful frequency
Before serving begins, the runtime can ask the system to:
fault in model pages
fault in KV arena
fault in tokenizer tables
fault in pinned host buffers
validate NUMA placement
lock critical pages if allowed
Goal:
no surprise page faults during TTFT
Generic reclaim can destroy inference latency.
A better reclaim order for inference cgroups:
1. reclaim inactive request buffers
2. reclaim cold KV
3. compress old KV
4. demote cold KV to CXL/remote memory
5. reclaim noncritical app memory
6. only then touch hot KV/model pages
This project is not:
a generic Linux tuning script
a CUDA kernel optimization project
a replacement for vLLM or TensorRT-LLM
a claim that kernel changes alone solve inference performance
an upstream-ready Linux patchset today
a scheduler rewrite without measurement
The goal is to identify the minimum OS-level primitives that help inference runtimes communicate intent to Linux.
Phase 0: Design documentation
Phase 1: infertrace eBPF observability
Phase 2: inferd userspace policy daemon
Phase 3: libinferhint runtime hint API
Phase 4: llama.cpp / vLLM / SGLang integrations
Phase 5: experimental kernel patchset
Phase 6: benchmark suite and whitepaper
Goal:
Measure where TTFT and token latency are actually spent.
Initial features:
scheduler delay tracing
page fault tracing
thread migration tracing
CPU frequency tracing
IRQ/core placement tracing
basic TTFT timeline markers
CLI report output
Success criterion:
Given an inference server, infertrace can explain why TTFT was slow.
Goal:
Apply inference-aware policies using existing Linux mechanisms.
Initial features:
cgroup creation
cpuset management
NUMA binding
CPU governor control
IRQ steering
hugepage policy
mlock/page warmup option
decode-core reservation
Success criterion:
inferd improves TTFT or reduces jitter without kernel patches.
Goal:
Allow runtimes to expose inference semantics.
Initial features:
request markers
phase markers
thread role hints
model memory hints
KV memory hints
runtime-to-inferd communication
Success criterion:
llama.cpp or vLLM can emit inference hints consumed by infertrace/inferd.
Targets:
llama.cpp
vLLM
SGLang
Ollama
TensorRT-LLM launcher
TGI
Success criterion:
real inference workloads produce traceable phase data and respond to policy control.
Candidate patches:
MADV_INFER_MODEL
MADV_KV_HOT
MADV_KV_COLD
inference cgroup controller
sched_ext inference scheduler
TTFT boost hook
inference hugepage policy
Success criterion:
kernel primitives show measurable improvement beyond userspace-only control.
Target metrics:
TTFT
p50/p95/p99 TTFT
decode tokens/sec
p50/p95/p99 per-token latency
GPU utilization
CPU scheduler delay
page faults per request
NUMA remote access rate
thread migrations
IRQ interference
CPU frequency at request arrival
C-state exit time
Target workloads:
single request latency
concurrent requests
batch-1 streaming
batch-8 serving
long-context 32K
long-context 128K
multi-session KV-heavy
cold model start
warm model start
Target runtimes:
llama.cpp
vLLM
SGLang
Ollama
TensorRT-LLM wrapper
linux-kernel-inference-fastpath/
├── README.md
├── docs/
│ ├── VISION.md
│ ├── KERNEL_INFERENCE_FAST_PATH.md
│ ├── SUBSYSTEMS.md
│ ├── API_SKETCH.md
│ ├── ROADMAP.md
│ ├── RISKS.md
│ └── PRIOR_ART.md
├── infertrace/
│ ├── README.md
│ ├── src/
│ ├── bpf/
│ └── examples/
├── inferd/
│ ├── README.md
│ ├── src/
│ └── configs/
├── libinferhint/
│ ├── include/
│ ├── src/
│ └── examples/
├── runtime-integrations/
│ ├── llama.cpp/
│ ├── vllm/
│ ├── sglang/
│ └── ollama/
├── kernel-patches/
│ ├── madvise-infer-model/
│ ├── madvise-kv-hot-cold/
│ ├── infer-cgroup/
│ └── sched-ext-infer/
├── benchmarks/
│ ├── ttft/
│ ├── decode-tps/
│ ├── numa/
│ └── pagefaults/
└── scripts/
Measure before patching.
Do the simple thing first.
Avoid policy in the kernel unless necessary.
Expose generic primitives, not AI hype.
Use cgroups, BPF, madvise, prctl, and sched_ext before core rewrites.
Make fast paths obvious.
Do not break normal workloads.
Avoid magic.
Make every knob observable.
Prefer hints over hard requirements.
Potential invention and research directions include:
inference-aware memory hints
TTFT boost mode
token-critical scheduling
KV-aware reclaim and demotion
inference topology domains
model mmap fast path
eBPF TTFT decomposition
token-loop GPU wakeup path
inference cgroup controller
Linux does not need to understand transformers.
But it can understand enough inference semantics to make better system decisions:
this thread is token-critical
this memory is model weights
this memory is hot KV
this memory is cold KV
this request is in TTFT phase
this token loop has a soft deadline
this GPU/NUMA/NIC locality matters
Those hints can help Linux reduce TTFT, improve tokens/sec, and make long-context inference more predictable.
The practical path is:
infertrace → inferd → libinferhint → runtime integrations → kernel patches
The long-term goal is to make Linux inference-aware.