feat(monitoring): Grafana + Prometheus + GPU metrics embedded in dashboard#71
Closed
AlienWalker1995 wants to merge 3 commits into
Closed
feat(monitoring): Grafana + Prometheus + GPU metrics embedded in dashboard#71AlienWalker1995 wants to merge 3 commits into
AlienWalker1995 wants to merge 3 commits into
Conversation
…hboard
Opt-in (--profile monitoring) real-time llama.cpp + GPU performance monitoring,
embedded in the dashboard's new Grafana tab and served via Caddy + SSO at /grafana/.
- llama-server now runs with --metrics (native Prometheus endpoint: token rates,
KV-cache usage, request queue). Internal-only; harmless when unscraped.
- prometheus: scrapes llamacpp:8080/metrics + the GPU exporter (15s, 15d retention).
- gpu-exporter: nvidia_gpu_exporter (wraps nvidia-smi — the right tool for this host's
consumer GPUs on WSL2, where DCGM-exporter does not work). Pinned to an explicit
--query-field-names set: the default AUTO discovery panic-crashes on driver 581.80
(nvidia-smi emits `...sw_thermal_slowdown [us]`, an invalid Prometheus metric name).
Verified live standalone on the 5090 + 1070.
- grafana: anonymous read-only (SSO is the gate), serves from /grafana/ subpath, embedding
enabled; datasource + a llama.cpp/GPU dashboard auto-provisioned. GPU panels use the
live-verified metric names + `group_left(name)` joins for per-GPU legends.
- Caddy /grafana/ route (protected block); dashboard Grafana tab lazy-loads the iframe.
- All images pinned by digest; no host ports; .env.example + CHANGELOG.
Validated: compose config, promtool check, caddyfile ("Valid configuration"), dashboard
JSON, shell syntax; GPU exporter + metric names exercised live standalone. llama.cpp
panel metric names are the documented set (prompt/predicted_tokens_seconds,
kv_cache_usage_ratio, requests_processing/deferred) — to be confirmed on first live
scrape once llamacpp restarts with --metrics.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Live validation against the running stack caught that this llama.cpp build does NOT expose `llamacpp:kv_cache_usage_ratio` (the two KV-cache panels would have been blank). Replaced them with metrics that exist and are confirmed to carry data: a "Decode rate (tok/s, 5m avg)" stat and a "Smoothed throughput (5m rate)" timeseries, both from `rate(llamacpp:tokens_predicted_total)` / `rate(llamacpp:prompt_tokens_total)`. Now fully live-validated: llamacpp recreated with --metrics (Prometheus target up=1); all 6 llama.cpp queries + all 4 GPU queries resolve with data; Grafana provisioned + serving the dashboard; Caddy /grafana/ → 302 SSO; both GPUs (5090 + 1070) reporting util/VRAM/temp/power. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
AlienWalker1995
pushed a commit
that referenced
this pull request
Jul 9, 2026
Extend the plugin manifest schema so kind=service plugins declare their compose services as data (name/image/gpu/gpu_pin/env/command/volumes/ healthcheck/depends_on); compose.py builds them from the resolved manifests instead of hardcoded if-blocks. Migrate comfyui/song-gen/voice to the schema. Bundle V1 into the substrate as plugins (sha-pinned images kept verbatim): - monitoring (PR #71 parity): grafana + prometheus + nvidia_gpu_exporter, CPU-ok, keeps the driver-581.80 --query-field-names crash-fix; render now emits --metrics on llama-server; monitoring/ config assets copied verbatim (live-verified 6fe4586); named compose volumes declared at top level. - voice (PR #45 parity): faster-whisper stt + Kokoro tts. New gpu_pin:secondary concept — hardware.detect() captures GPU uuids; render pins these no-Blackwell images to the non-primary (Pascal 1070) card via CUDA_VISIBLE_DEVICES + device_ids (the only pin WSL2 honors). No secondary GPU -> gated OFF with a warning (never fall back to the 5090 and ship a crash). Validated in a throwaway container: ruff clean, 103 tests (was 99); rendered dual-GPU compose (media+voice+monitoring) passes real docker compose config; voice pins to GPU-20fac13a-… . Live ordo-ai-stack untouched. Co-Authored-By: Claude Opus 4.8 <[email protected]>
AlienWalker1995
added a commit
that referenced
this pull request
Jul 16, 2026
The tab (button + lazy kiosk iframe of /grafana/d/ordo-llm-gpu) only ever lived on feat/grafana-monitoring; the deployed image had been built from that branch, so rebuilds from main silently dropped it. The branch's V1 compose/monitoring pieces are superseded by the v2 monitoring plugin — this ports the one hunk that wasn't. Co-authored-by: Hermes Bot <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]>
Owner
Author
|
Superseded: the Prometheus/Grafana/exporter stack shipped as the v2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Opt-in (
--profile monitoring) real-time llama.cpp + GPU performance monitoring, embedded in the dashboard's new 📊 Grafana tab and served through the existing Caddy + Google SSO front door at/grafana/.Architecture
--metrics— native Prometheus endpoint (token rates, KV-cache usage, request queue). Internal-only; harmless when unscraped.nvidia_gpu_exporter(wrapsnvidia-smi; the right tool for this host's consumer GPUs on WSL2, where DCGM-exporter doesn't work)./grafana/route; dashboard tab lazy-loads the iframe (?kiosk)..gitattributespinsmonitoring/**to LF.Real bug caught during validation
The GPU exporter's default field auto-discovery panic-crashes on driver 581.80:
nvidia-smiemitsclocks_event_reasons_counters.sw_thermal_slowdown [us], whose derived metric name has a space + brackets → invalid Prometheus name → crash-loop. Fixed by pinning--query-field-namesto an explicit safe set. Verified live standalone on the 5090 + 1070.Validation
promtool check config(SUCCESS),caddy validate("Valid configuration"), dashboard JSON, shell syntax.nvidia_smi_utilization_gpu_ratio,..._memory_used_bytes,..._temperature_gpu,..._power_draw_watts) withgroup_left(name)joins (GPU name is only onnvidia_smi_gpu_info, not the value metrics).llamacpp:predicted_tokens_seconds,prompt_tokens_seconds,kv_cache_usage_ratio,requests_processing,requests_deferred) — not yet live-verified because that needs llamacpp to restart with--metrics. Confirm on first live scrape.Enable (operator — needs SOPS secrets, so runs on the host)
Then open the dashboard → Grafana tab, or
https://<host>/grafana/.🤖 Generated with Claude Code