How Oprel uses your hardware — and how to get the best performance.
Oprel runs large language models and image generation on your own machine by wrapping two battle-tested inference engines: llama.cpp for text generation and stable-diffusion.cpp for image generation. Oprel automatically downloads the right pre-built binary for your platform, detects your GPU, calculates optimal settings, and manages the subprocess lifecycle. You don't need to configure backends or compile anything from source.
| Platform | Text (llama.cpp) | Image (stable-diffusion.cpp) |
|---|---|---|
| Linux x86_64 | CPU, Vulkan | CPU, Vulkan, ROCm |
| Windows x86_64 | CPU, CUDA, Vulkan | CPU, CUDA, Vulkan |
| Windows arm64 | CPU | — |
| macOS arm64 (Apple Silicon) | Metal | Metal |
| macOS x86_64 (Intel) | CPU | — |
On startup, Oprel scans your machine in this order:
- PyTorch CUDA — if
torchis installed andtorch.cuda.is_available(), uses it directly. - nvidia-smi — falls back to the NVIDIA driver utility (works without PyTorch).
- rocm-smi — checks for AMD ROCm on Linux.
- lspci fallback — detects AMD GPUs even without ROCm installed (VRAM defaults conservatively).
- Metal — on Apple Silicon, probes
system_profilerfor GPU info and calculates usable unified memory. - CPU — if no GPU is found, runs on CPU with automatic thread detection.
The binary that gets downloaded is chosen to match: if you have an NVIDIA GPU on Windows, you get the CUDA binary. If you're on Linux with an AMD GPU but no ROCm installed, you get the CPU binary (or Vulkan, if your GPU supports it).
llama.cpp: CUDA binaries are available on Windows only. The binary registry does not include a Linux CUDA variant for llama.cpp. On Linux with an NVIDIA GPU, Oprel falls back to CPU or Vulkan.
stable-diffusion.cpp: CUDA binaries are available on Windows.
Vulkan support is available for both backends on Linux x86_64 and Windows x86_64. Vulkan provides GPU acceleration via the Vulkan API and is typically faster than pure CPU. If you have an NVIDIA GPU on Linux, Vulkan is your best available option for text generation.
Metal binaries are available for both backends on macOS arm64 (M1/M2/M3/M4). Because Apple Silicon uses unified memory, the GPU and CPU share the same pool. Oprel estimates how much of your total RAM can safely be used for model inference — typically 50–75% depending on your chip.
ROCm binaries exist only for stable-diffusion.cpp on Linux. There is no ROCm variant for llama.cpp in the binary registry. On Linux with an AMD GPU, text generation runs on CPU or Vulkan; image generation can use ROCm if rocm-smi is detected.
Oprel does not implement its own inference engine, memory manager, or GPU scheduler. It passes models to llama.cpp and stable-diffusion.cpp with the best settings it can calculate from your hardware. The real memory limits are set by those engines.
Model weights live in RAM. Even with GPU offloading, the full model file must fit in system memory (or be memory-mapped). As a rule of thumb, you need at least the GGUF file size plus 1–2 GB for context and overhead.
Only the layers you offload to GPU (n_gpu_layers) consume VRAM. The remaining layers stay in system RAM and run on CPU. Oprel automatically calculates how many layers fit:
- It reads total VRAM from
nvidia-smi. - It estimates total model layers from file size (e.g., ~32 layers for a 7B-class model, ~80 for a 70B-class).
- It reserves headroom for KV cache and CUDA overhead (typically 0.5–1 GB depending on GPU type).
- It outputs a recommended
n_gpu_layersvalue.
You can override this manually with --n-gpu-layers <N> if you want to force more or fewer layers onto GPU.
On Apple Silicon, there is no separate VRAM pool. llama.cpp uses Metal for GPU acceleration and draws from unified memory. Oprel's heuristics estimate usable memory based on total RAM (50% for 8 GB base models, up to 75% for 64 GB Max chips). This is conservative to leave room for macOS and other applications.
Oprel downloads models from the HuggingFace Hub in GGUF format. File size depends on the original model parameters and the quantization level.
| Model class | Q4_K_M (typical) | Q8_0 (high quality) |
|---|---|---|
| 1–3B params | 1–2 GB | 2–4 GB |
| 7–8B params | 4–5 GB | 7–9 GB |
| 13–14B params | 7–9 GB | 13–16 GB |
| 33–34B params | 18–22 GB | 33–38 GB |
| 70–72B params | 38–44 GB | 70–77 GB |
These are approximate. Exact sizes vary by model architecture and vocabulary.
Oprel handles:
- Hardware detection (GPU type, VRAM, RAM, CPU features)
- Binary selection and download (platform + GPU variant)
n_gpu_layerscalculation (how many layers to offload)- Thread count recommendation (physical cores minus 1–2 for system)
- KV cache type selection (
f16,q8_0,q4_0) - Server lifecycle (spawn, health-check, restart on crash)
- VRAM monitoring during model loading
llama.cpp handles:
- Actual inference (token generation, attention, matrix math)
- Memory allocation for model weights
- GPU kernel execution (CUDA, Vulkan, Metal)
- Context window management
- Flash attention and mmap
Oprel stores content under ~/.cache/oprel/:
| Directory | Contents |
|---|---|
~/.cache/oprel/models/ |
Downloaded GGUF model files |
~/.cache/oprel/bin/ |
llama-server and sd-cli binaries |
~/.cache/oprel/ocr/ |
PaddleOCR models (downloaded on first OCR use) |
~/.cache/oprel/models/chat_history.db |
SQLite database for conversations, provider configs, and settings |
The OPREL_HOME environment variable controls the knowledge directory only (OPREL_HOME / "knowledge"). It does not affect the model cache, binary cache, OCR cache, or database paths — those are configured independently via Config.cache_dir and Config.binary_dir in oprel/core/config.py.
By default, the Oprel server binds to localhost and picks a port from the range 54321–54420. This means:
- Only local applications can reach the server.
- Remote access is blocked by default.
Do not expose the server to a network without an authentication layer, reverse proxy, and firewall. The Oprel API has no built-in auth. If you need remote access, put it behind a reverse proxy (nginx, Caddy) with TLS and authentication.
These are guidelines based on how much memory a model needs. Actual performance depends on your specific hardware, quantization, context length, and backend.
- Good fit: 1–3B models (Q4_K_M). Expect slower generation than GPU-backed systems, with speed depending heavily on CPU, quantization, context length, and background load.
- Possible with patience: 7B models at Q4_K_M, but system may swap.
- Not practical: Models above 7B.
- Comfortable: 7–8B models with most layers on GPU. Responsive chat speeds.
- Partial offload: 13B models — some layers on GPU, rest on CPU.
- Not enough for: 33B+ models entirely on GPU.
- Comfortable: 7–13B models fully on GPU.
- Partial offload: 33B models with most layers on GPU.
- Stretched: 70B models at Q4_K_M — fits if you accept CPU fallback for some layers.
- Comfortable: 7–13B models, plus room for large context windows.
- Mostly GPU: 33B models with high GPU layer count.
- Partial offload: 70B models at Q4_K_M.
- Comfortable: 7–34B models fully on GPU with generous context.
- Mostly GPU: 70B models at Q4_K_M fit with most layers on GPU.
- Watch context: Very long contexts (>32K) consume significant VRAM for the KV cache.
- 8 GB (base M1/M2): 1–3B models comfortably. 7B models at Q4_K_M with reduced context.
- 16 GB (M1/M2 with upgrade, M1 Pro base): 7B models comfortably. 13B at Q4_K_M.
- 32–36 GB (M1/M2 Pro with upgrade): 13–34B models.
- 64–96 GB (M1/M2 Max): 34–70B models.
On Apple Silicon, KV cache quantization (q8_0 or q4_0) can significantly reduce memory pressure on larger models.
| Variable | Effect |
|---|---|
OPREL_HOME |
Override knowledge directory base path (default: ~/.cache/oprel). Does not control model, binary, OCR, or database paths |
OPREL_SSL_NO_VERIFY |
Set to 1 to disable SSL verification for binary downloads (useful behind corporate proxies) |
OPREL_SKIP_RUNTIME_DOWNLOAD |
Set to 1, true, or yes to skip runtime binary downloads during pip install |
- NVIDIA: Run
nvidia-smiin a terminal. If it fails, your drivers aren't installed or aren't in PATH. On Linux, also check thatnvidia-smiis available. - AMD on Linux: Install ROCm if you want ROCm-backed image generation. Without
rocm-smi, Oprel may still detect your GPU vialspci, but backend availability depends on the binary selected for your platform. - Apple Silicon: Make sure you're on macOS arm64 (not Intel). Run
uname -m— it should sayarm64.
- Check available RAM:
free -h(Linux) or Activity Monitor (macOS). - Compare model file size to free RAM — the full file must fit.
- Reduce context size (
--ctx-size 4096or--ctx-size 2048). - Use a lower quantization (
Q4_K_Minstead ofQ8_0). - If on Apple Silicon, close memory-heavy applications first.
- Oprel auto-downloads binaries on first use. If downloads fail behind a corporate proxy, set
OPREL_SSL_NO_VERIFY=1or configuressl_verifyandssl_cert_filein your Oprel config. - If you switch GPUs (e.g., from AMD to NVIDIA), delete the old binary from
~/.cache/oprel/bin/— Oprel will re-download the correct variant on next run.
- Check your CPU supports AVX2:
grep avx2 /proc/cpuinfo(Linux). Pre-built binaries target modern CPUs; very old CPUs may fall back to slower code paths. - Reduce thread count if the system is unresponsive during generation — Oprel uses physical cores minus 2 by default, which is usually right, but you can tune it with
--n-threads.
- On Linux with NVIDIA, recall that Oprel's llama.cpp registry currently has no Linux CUDA binary — you're running on CPU or Vulkan. Vulkan provides GPU acceleration through the Vulkan API but may not match native CUDA performance.
- Check
nvidia-smiduring generation — if GPU utilization is low, most layers may be running on CPU. Increasen_gpu_layers. - On AMD with llama.cpp, Vulkan is your only GPU option on Linux — ROCm binaries don't exist for text generation.
- Text generation with llama.cpp
- Image generation with stable-diffusion.cpp
- Model downloads, quantization, and aliases
- Hardware telemetry and GPU detection internals