Skip to content

CLAUDE.md

Latest commit

 

History

History
60 lines (46 loc) · 11.2 KB

CLAUDE.md

File metadata and controls

60 lines (46 loc) · 11.2 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What this repo is

This repo is a single Docker Compose stack (docker-compose.yml) for a self-hosted, GPU-accelerated AI chatbot with retrieval-augmented generation (RAG). There is no application source code, build system, or test suite — all behavior comes from the compose file wiring together off-the-shelf container images. "Working in this codebase" means editing docker-compose.yml (service config, env vars, ports, volumes) and operating the stack.

Common commands

cp .env.example .env                 # REQUIRED before first up: set WEBUI_SECRET_KEY (openssl rand -hex 32)
docker compose up -d                 # start the whole stack (detached)
docker compose down                  # stop and remove containers (data persists in bind mounts)
docker compose pull && docker compose up -d   # update images and recreate changed services
docker compose ps                    # list running services
docker compose logs -f open-webui    # tail logs for one service
docker compose restart open-webui    # restart after changing its env

docker compose exec ollama ollama pull <model>   # download a model (e.g. gemma3:12b)
docker compose exec ollama ollama list           # list installed models

# Optional bulk PDF import (profile-gated; drop PDFs/zip in ./import first):
docker compose --profile import run --rm pdf-import
# Optional: create/update the grounded model bound to the imported collection:
docker compose --profile setup run --rm setup-grounded
# Optional: provision API access for external consumers (prints an API key):
docker compose --profile setup run --rm setup-api

There is no lint/test/build step — validate changes with docker compose config (renders and checks the compose file) and by bringing the stack up.

Architecture

Four always-on services on one Compose network, plus optional profile-gated extras (one-shot import/setup jobs and a consumer demo). Only ollama and open-webui are published to the host (plus the demo's :8090 when enabled); the rest are reachable only by service name from inside the network. The Open WebUI image tag is shared via the x-owui-image YAML anchor (&owui_image) so open-webui and the helper jobs pull a single image.

  • ollama (host 11500 → 11434) — LLM inference engine. Requires an NVIDIA GPU (reserved via deploy.resources with count: all); the stack will not run models without it. Default context window is raised via OLLAMA_CONTEXT_LENGTH=16384 (Ollama's 4096 default silently truncates RAG prompts; the old OLLAMA_NUM_CTX on open-webui was a no-op). The docs-grounded model also pins num_ctx: 16384 in its params via setup_grounded.py. Models persist in ~/ollama_data.
  • open-webui (host 3000 → 8080) — the chat frontend and the only user-facing UI. Uses the :v0.10.2-cuda image and reserves the GPU (deploy.resources) so embeddings + the reranker run on GPU, not CPU (query latency is otherwise dominated by CPU reranking, independent of corpus size). Talks to Ollama via OLLAMA_BASE_URL=http://ollama:11434. Owns the RAG configuration (see below). Requires WEBUI_SECRET_KEY (from .env). App state (users, chats, vector DB) persists in ~/openwebui_data.
  • docling-serve (internal :5001) — document extraction backend. open-webui offloads file parsing to it via CONTENT_EXTRACTION_ENGINE=docling + DOCLING_SERVER_URL.
  • pipelines — Open WebUI Pipelines runtime (OpenAI-compatible plugin server, default :9099) for custom filters/functions. Config persists in ~/pipelines_data.
  • setup-api (optional, profiles: ["setup"]) — one-shot; runs scripts/setup_api_consumer.py. Provisions external API access: enables API keys endpoint-restricted to /api/chat/completions + /api/models, grants the features.api_keys default-user permission (required for non-admins to hold a key; harmless here since signup is off), creates the non-admin [email protected] user, mints/fetches its API key, smoke-tests it against docs-grounded, and prints the key. Idempotent. Field-name gotcha: this build's admin auth config uses plural ENABLE_API_KEYS / API_KEYS_ALLOWED_ENDPOINTS (unknown fields are silently ignored). Model visibility for non-admins requires an access grant — setup_grounded.py sets a public-read grant (principal_id: "*"); without it non-admin API calls get 400 "Model not found". The knowledge collection needs its own public-read grant (set by import_pdfs.py via /api/v1/knowledge/{id}/access/update) — model access alone is not enough: retrieval is access-filtered per requesting user, so a non-readable collection makes the grounded model refuse every question with "no info in the documents" (empty context), with no error anywhere. Consumer docs: docs/API.md.
  • pdf-import (optional, profiles: ["import"] — not started by up) — one-shot bulk PDF importer. Reuses the open-webui image; runs scripts/import_pdfs.py (pure stdlib). Reads PDFs/zip from ./import, mounts ~/openwebui_data read-only to read the admin id, signs a JWT with the shared WEBUI_SECRET_KEY, and loads files into a knowledge collection (IMPORT_COLLECTION, default "Imported PDFs") via the API. Idempotent. Run: docker compose --profile import run --rm pdf-import.
  • setup-grounded (optional, profiles: ["setup"]) — one-shot; runs scripts/setup_grounded.py. Creates/updates the docs-grounded workspace model (base gemma3:12b) bound to the imported collection, with a system prompt that answers only from the docs (no guessing), replies in the user's language, and cites sources. Idempotent (create-or-update). The strict RAG template, RAG_RELEVANCE_THRESHOLD, and DEFAULT_MODELS=docs-grounded are seeded from open-webui env on first boot; this job only owns the model (it references a runtime collection, so it cannot be an env var). Run after importing: docker compose --profile setup run --rm setup-grounded.

Data flow for a RAG query: open-webui extracts uploaded docs via docling → chunks (CHUNK_SIZE=3000, CHUNK_OVERLAP=500) and embeds them → on a question, runs hybrid retrieval (ENABLE_RAG_HYBRID_SEARCH, BM25 weight 0.5) pulling RAG_TOP_K=20, then reranks with BAAI/bge-reranker-v2-m3 down to RAG_TOP_K_RERANKER=10 → passes context to the Ollama model for generation.

Important gotchas

  • WEBUI_SECRET_KEY is required (referenced as ${WEBUI_SECRET_KEY:?...} in docker-compose.yml). Copy .env.example → .env and set it (openssl rand -hex 32) or docker compose up fails. Pinning it keeps logins alive across restarts and lets the pdf-import service authenticate. .env is gitignored; the dropped import/ files are too.
  • Open WebUI config is write-once (PersistentConfig). Its env vars (web search, tool calling, docling, RAG, etc.) seed the SQLite DB (~/openwebui_data/webui.db, config table) only on the first boot with an empty data dir. After that the DB wins and env edits are ignored. To change a setting on an existing install: use the UI, or patch the config table directly and docker compose restart open-webui (it caches config in memory at boot), or wipe ~/openwebui_data to re-seed. This bit us: added web-search/tool env didn't apply until patched in the DB.
  • Web search is auto-wired (via env, on first boot): ENABLE_WEB_SEARCH=true, WEB_SEARCH_ENGINE=duckduckgo (no key), WEB_LOADER_ENGINE=safe_web (open-webui's built-in page fetcher — no JS rendering; SPA-heavy sites extract poorly). It runs only when a UI user toggles the globe icon; the grounded model and API consumers never trigger it. A crawl4ai + proxy scraping pair was removed (the unmaintained proxy 502'd on real multi-URL batches — see git history if resurrecting). pipelines is standalone — register it manually as an OpenAI connection to http://pipelines:9099 if you want it (optional; the chatbot works without it).
  • function_calling must be legacy, not default, for tool-less models (gemma3). Set via DEFAULT_MODEL_PARAMS={"function_calling":"legacy"}. In open-webui v0.10.2 middleware.py treats ONLY "legacy" as prompt-based; every other value (including "default") is native → it attaches a tools array → gemma3 errors "does not support tools" with web search on. The UI label "Default" maps to native here; "Legacy" is the safe one.
  • Never touch webui.db from the host. It runs in WAL mode; a host (Windows) SQLite connection creates -wal/-shm sidecar files the container's Linux SQLite then cannot open → OperationalError: unable to open database file on every query, and any uncommitted WAL writes can be lost. Edit the DB only via docker compose exec -T open-webui python3 - <<'PY' … PY (container-side), the HTTP API, or with the container stopped.
  • Grounded model + registry refresh. The docs-grounded model answers only from the docs. A model created/updated via the API is invisible to /api/chat/completions (400 "Model not found") until /api/models is fetched once or open-webui restarts. Bound-collection knowledge is injected only when function_calling is legacy. To go grounded end-to-end on a fresh box: up -d (seeds strict RAG_TEMPLATE, RAG_RELEVANCE_THRESHOLD=0.25, DEFAULT_MODELS=docs-grounded) → import PDFs → run setup-grounded.
  • Citation format must be spelled out in RAG_TEMPLATE. Chunks reach the model as <source id="N"> tags; a vague "cite as [id]" instruction makes gemma3 copy the attribute syntax and emit literal [id="1"] in answers (breaks the UI's citation chips, reads as noise over the API). The template now dictates the exact format (bracketed number only). RAG_TEMPLATE is PersistentConfig — changing it on an existing install means patching the rag.template key in the config table (container-side) + restart, not just editing the env.
  • Citation semantics for API consumers. /api/chat/completions responses include a top-level sources array (retrieved chunks + metadata). Marker [n] = nth chunk across all sources, flattened, 1-based. Source names are intentionally NOT exposed as download links: the API key is blocked from /api/v1/files, and public widget visitors shouldn't get the raw (possibly restricted) PDFs. See docs/API.md.
  • Scanned-PDF OCR. DOCLING_PARAMS forces OCR (force_ocr) with ocr_preset=tesseract — ocr_engine is deprecated in docling-serve and silently ignored, and the default RapidOCR mis-reads Latin/Malay (loads a Chinese model). Re-OCRs digital PDFs too (slower). Changing it needs a re-import to take effect.
  • Bind mounts, not named volumes. All persistent data lives under the invoking user's home dir (~/ollama_data, ~/openwebui_data, ~/pipelines_data). Deleting these wipes models/chats. Do not switch to named volumes without migrating this data.
  • All images are pinned (see docker-compose.yml). pipelines publishes no semver tags, so it is pinned by git-<sha>. docker compose pull will not move versions; bump deliberately by editing the tags.
  • ollama was on a stale release candidate. It is now 0.31.1 (stable). A version jump can change the on-disk model store format — after bumping, verify existing models in ~/ollama_data still load.