⚔️ LORE
Local Orchestration & Runtime Engine

---

LORE orchestrates multiple specialized small language models on edge devices with 16 GB RAM. Instead of picking one big model and hoping for the best, LORE loads a primary model for reasoning and a specialist model for lightweight tasks — simultaneously, within budget.

The key insight: With TurboQuant KV cache compression (4.57×) and SSM/hybrid specialist models (near-zero KV cache), you no longer have to choose between model quality, context length, and memory headroom. You get all three.

Architecture

                         USER REQUEST
                              │
                    ┌─────────▼──────────┐
                    │    TOOL ATTENTION    │  Lazy schema loading (93.5% fewer tokens)
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │   CONTEXT MANAGER   │  Budget, compression, memory, health
                    └─────────┬──────────┘
                              │
                    ┌─────────▼──────────┐
                    │       ROUTER        │  TF-IDF + LogReg (<1ms, no LLM cost)
                    └───┬─────────┬───┘
                        │         │
            ┌───────────▼──┐  ┌──▼───────────┐  ┌──────────────────┐
            │  PRIMARY 9B  │  │ SPECIALIST 1.5B│  │   TOOL-ONLY      │
            │  Ornith-1.0  │  │  Falcon-H1     │  │  (no LLM needed) │
            │  5.63 GB     │  │  1.00 GB       │  │                  │
            └──────┬───────┘  └──────┬────────┘  └────────┬─────────┘
                   └────────────────┴─────────────────────┘
                              │
                    ┌─────────▼──────────┐
                    │    ORCHESTRATOR     │  Decompose → schedule → execute → aggregate
                    └────────────────────┘  (only for complex multi-step tasks)

Memory Budget (measured)

Component	Size
Ornith-1.0-9B Q4_K_M (primary)	5.50 GB
Falcon-H1-1.5B Q4_K_M (specialist)	1.09 GB
KV cache (both models, turbo4, 16K context)	included above
Total	6.59 GB
Headroom	7.41 GB

All features enabled (compression, memory, health): 6.99 GB. Never exceeds 14 GB.

What's Inside

Module	Purpose
orchestrator.py	Task decomposition, wave-based scheduling, parallel execution, aggregation
router.py	TF-IDF + LogReg classifier (<1ms, >85% accuracy)
context.py	Token budget management, compression gating, prefix cache
memory.py	Hierarchical memory — episodic (embeddings) + semantic (facts)
tool_attention.py	Lazy tool schema selection via embeddings (NTILC pattern)
verifier.py	JSON/code validation and auto-repair
leaderboard.py	Live HuggingFace benchmark scanning for model upgrades
registry.py	Auto-select best local model per task type from benchmarks
classifier.py	Model-based task complexity estimation (replaces regex heuristics)
session.py	KV cache disk persistence for instant session resume
health.py	Context utilization monitoring, staleness detection
models.py	llama-server lifecycle, model swapping via llama-swap

Optimizations (measured, not theoretical)

Technique	Result	Default
TurboQuant KV compression	0% PPL degradation on hybrid SSM models	Always on
LLMLingua-2 compression	56.5% token reduction	Off (opt-in, activates at 10+ turns)
Tool Attention	93.5% fewer tool tokens (3200→207 at 50 tools)	On (gated at 15+ tools)
Parallel wave execution	Cross-model subtasks run simultaneously	On
Speculative decoding	Skipped — vocab mismatch between models	N/A
TIDE early exit	Skipped — SSM-incompatible	N/A
MiniCache	Skipped — TurboQuant conflict	N/A

Every optimization was tested against real inference. Techniques that don't work on hybrid SSM architectures were skipped with evidence, not hoped away.

Quick Start

# Clone
git clone https://github.com/oniwakaa/lore.git
cd lore

# Install
pip install -e ".[dev]"

# Run tests
python -m pytest tests/ -v

# Setup (on Apple Silicon M4 with 16 GB)
bash scripts/setup.sh          # builds llama.cpp, downloads models, generates imatrix

# Launch
lore                           # starts interactive REPL
lore --api                     # starts OpenAI-compatible API on port 8000
lore --api --port 9000         # custom port
lore "what is 2+2?"            # single-shot mode

OpenAI-Compatible API

LORE exposes an OpenAI-compatible API so it works with Continue.dev, Cline, Open WebUI, and any OpenAI-compatible client:

# Start the API server
lore --api --port 8000

# Use with curl
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "write a fibonacci function in python"}]}'

# Use with Python (openai package)
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="lore",
    messages=[{"role": "user", "content": "what is 2+2?"}]
)
print(response.choices[0].message.content)

# List running models
curl http://127.0.0.1:8000/v1/models

# Health check
curl http://127.0.0.1:8000/health

Endpoints:

Method	Path	Description
POST	/v1/chat/completions	Chat completion (OpenAI-compatible)
GET	/v1/models	List available models
GET	/health	Health check

The response includes a lore extension field with routing info, orchestration status, and latency metrics.

Configuration

All config lives in configs/:

• models.yaml — Model paths, quantization types, engine settings, server path
• router.yaml — Router training parameters
• memory.yaml — Memory system settings (episodic/semantic)
• compression.yaml — LLMLingua-2 settings
• llama-swap.yaml — Model hot-swap configuration

REPL Commands

Command	Action
<message>	Process a task (auto-routes or orchestrates)
/switch <session>	Switch to a different session
/save	Save current session (KV cache + state)
/resume <id>	Resume a saved session
/sessions	List all saved sessions
/models	Show loaded models and memory usage
/upgrades	Check HuggingFace for better model options

Project Status

Phase	Status	Key Result
0: Foundation	✅	Dual model @ 6.59 GB, TurboQuant validated
1: Core Stack	✅	Router, context manager, GBNF structured output
2: Optimizations	✅	6 techniques measured, 3 shipped, 3 skipped with evidence
3: Agentic	✅	Memory, health, sessions, multi-session management
3.5: Wire + Verify	✅	End-to-end integration, verifier, dynamic sizing
4: Orchestration	✅	Decomposer, workers, wave scheduling, aggregation
4.1: Hardening	✅	Public APIs, parallel execution, deduped memory
4.2: Live Benchmarks	✅	HF leaderboard scanning, registry, model-based classifier
5: Benchmark & Harden	🔜	Full A/B: orchestrated vs single model

Design Philosophy

1. Measure before stacking. Every optimization must prove its value at the scale it's designed for. No feature runs unconditionally.
2. Skip with evidence. If a technique doesn't work on hybrid SSM architectures, it gets skipped with a table showing why — not a TODO.
3. Conditional gating. Optimizations activate only when their crossover point is reached (compression at 10+ turns, tool attention at 15+ tools).
4. Honest benchmarks. Toy-scale tests that don't reach crossover points don't count. Real inference, real hardware, real numbers.
5. The orchestration question. Orchestration adds complexity. It must measurably beat single-model dispatch to justify itself. Phase 5 answers this.

Hardware Requirements

• Minimum: Apple Silicon M4 (or equivalent), 16 GB unified memory
• Storage: ~15 GB for model files
• OS: macOS (primary), Linux (supported)
• Backend: Metal (macOS), CPU (fallback)

References

• TurboQuant — KV cache compression (ICLR 2026)
• Ornith-1.0 — Primary model
• Falcon-H1 — Specialist model
• Tool Attention — Lazy schema loading
• Sakana Fugu — Orchestration inspiration

---

Built for the edge. Measured on real hardware. No hand-waving.