GPU monitoring tools show utilization percentages, but this can be misleading. A GPU reporting 100% utilization may actually be computing useful work, or wastefully stalled waiting on memory transfers, thermal throttling, or power limits. NVSonar analyzes real-time patterns from NVML metrics to identify what's actually limiting your GPU performance.
- Diagnostics: bottleneck classification (compute, memory, power, thermal, data-starved), temporal pattern detection (clock oscillation, temperature trends, utilization dips, memory leaks)
- Multi-GPU: outlier detection via Z-scores, flags the GPU slowing down distributed training
- Health scoring: 0-100 per GPU with A-F grades, actionable recommendations with specific commands
- Benchmarks: memory bandwidth, compute throughput, PCIe speed vs theoretical specs
- History: tracks GPU health over time, detects degradation trends
- Python API: session monitoring during training (
nvsonar.start(),nvsonar.stop()) - Output: terminal report, JSON, CSV
- Prometheus exporter: scrape bottleneck classification + health score from Grafana (
nvsonar exporter)
- Python 3.10+
- NVIDIA GPU with driver installed
- Linux
- CUDA toolkit (only for
nvsonar benchmark, not required for other commands)
pip install nvsonarnvsonar # interactive TUI
nvsonar report # one-shot diagnostic
nvsonar report --plain # plain text without colors
nvsonar report --json # structured output for scripts/LLMs
nvsonar report --csv # CSV output for spreadsheets
nvsonar report --gpu 0 # single GPU
nvsonar report --gpu 0,1,2 # subset of GPUs
nvsonar benchmark # GPU performance benchmarks
nvsonar history # health trends over time
nvsonar exporter # Prometheus exporter on :9100/metricsnvsonar exporter exposes Prometheus metrics including bottleneck classification, throttle reason, and the NVSonar health score — the things DCGM's exporter doesn't surface. Add it to your prometheus.yml:
scrape_configs:
- job_name: nvsonar
static_configs:
- targets: ['gpu-host:9100']Useful PromQL:
sum by (type) (nvsonar_gpu_bottleneck) # bottleneck distribution across the fleet
avg_over_time(nvsonar_gpu_health_score[1h]) # rolling health average
nvsonar_gpu_throttle_active{severity="critical"} # active critical throttle reasons
A ready-made Grafana dashboard is shipped at dashboards/nvsonar.json — import it in Grafana (+ → Import → Upload JSON file) and pick your Prometheus datasource. Ten panels: health score, bottleneck distribution, temperature with thermal thresholds, power draw vs limit, compute utilization, VRAM usage, active throttle reasons, ECC error rate, and exporter self-monitoring.
- T4 (Turing)
- A30 (Ampere)
- GB10 Spark (Grace + Blackwell)
Apache License 2.0