NVIDIA datacenter telemetry: GPM compute breakdown, health counters, NVLink/C2C interconnect - #1
Open
rocker-zhang wants to merge 9 commits into
Open
NVIDIA datacenter telemetry: GPM compute breakdown, health counters, NVLink/C2C interconnect#1rocker-zhang wants to merge 9 commits into
rocker-zhang wants to merge 9 commits into
Conversation
- NVLink aggregate throughput (Rx/Tx + %-of-peak) via the NVML field-value API, as an opt-in plot series and an absolute-rate panel column. - SM and Tensor-core activity via NVML GPM, opt-in and gated so it only samples the shared perfmon counters while actually displayed. - GPU reliability signals: clock-throttle reasons, ECC SBE/DBE, row-remap pending/failure, NVLink errors, PCIe replay; summarised as a per-GPU status glyph plus detail fields. - Merge the interconnect/health/compute panels into one width-adaptive per-GPU "GPU Detail" table; categories with no data collapse out. - Size the process region to its content so an empty list no longer leaves a large blank gap above the panel. - Experimental host-global GDR/GDS summary behind -DNVTOP_GDR (off by default).
NVML stopped exposing NVLINK_SPEED_MBPS_COMMON (field 90) on Blackwell, so the per-link peak used to compute NVLink utilisation could not be read and the utilisation percentage disappeared on those GPUs. When the field is unavailable, fall back to mapping the NVLink version (nvmlDeviceGetNvLinkVersion) to the per-link speed: version 7 (NVLink 5) is 53.125 GB/s/link, matching nvidia-smi. Unknown versions leave the peak unset so only absolute Rx/Tx is shown, never a wrong percentage. Verified on GB200 and B300: utilisation matches nvidia-smi throughput within measurement error (20% vs 20.2% ground truth under peer-copy traffic); the existing field-90 path on A100/Hopper is unchanged.
With only a few GPUs the per-GPU charts stretched to fill the whole screen, making each chart much taller than it is in an 8-GPU layout. When there are processes to display, cap the plot area to roughly half the height an 8-GPU layout would use (about 13 rows per stacked row of charts) and give the freed rows to the process list. An idle box keeps the full-height charts so it never shows an empty band below them. 8/16-GPU layouts already fit under the budget and are unaffected. Verified on GB200/B300 (4 GPUs): charts shrink from full to about half when processes appear and restore when they exit; A100 (8 GPUs) chart heights are unchanged.
Sample the per-pipe NVML GPM metrics alongside the existing aggregate SM/tensor/occupancy/DRAM-BW, in the same nvmlGpmMetricsGet diff: the tensor MMA classes (HMMA float / IMMA int / DFMA fp64) and the CUDA-core pipes (FP16/FP32/FP64/INT). Tag the GPU Detail tensor field with the dominant tensor class so an INT8-quantised inference workload reads distinctly from FP training/inference. GPM exposes no FP8/FP4 metric and no QMMA/OMMA counter, so float width is not separable: on B300 a real FP8 GEMM (nvjet qqtst kernel, QMMA-class at the SASS level) still lands in the GPM HMMA/ANY_TENSOR bucket at full util, so it is observable but indistinguishable from FP16/BF16. HMMA is therefore labelled "FP" (never a specific width that would mislead). Verified on a B300 SXM6 node: BF16/FP8 -> "FP", INT8 -> "INT8".
Add the HBM/memory junction temperature (NVML field NVML_FI_DEV_MEMORY_TEMP) to the GPU Detail health cluster, coloured by absolute risk band — it is the HBM3 thermal limiter, distinct from the die temperature already shown and a leading indicator of the thermal throttle it precedes. Also plumb, for the upcoming detail drill-down, three health counters read in the same field-value batch: correctable/uncorrectable row-remap counts (fields 142/143, RMA proximity beyond the pending/failure flags) and the perf-policy violation times (power/thermal/total seconds held below application clocks — the non-deprecated replacement for the CUDA-13 deprecated nvmlDeviceGetViolationStatus). Note total-app-clocks also counts benign idle downclocking, so only the power/thermal limiters are surfaced. Field semantics verified on a B300 SXM6 node: MEMORY_TEMP returns degrees C distinct from the die sensor; perf-policy fields return nanoseconds.
Enumerate each device's NVLinks via nvmlDeviceGetNvLinkState and roll up how many are still active. A single link dropping out is invisible in the summed Rx/Tx throughput already shown, yet it halves that GPU's share of collective bandwidth — so the panel now prints "NVLink active/total" next to the aggregate, in red when active < total, to localise a degraded fabric before a collective run stalls on it. The roll-up is two scalar counts (no per-link array), so it fits the existing single-row layout; a full per-link throughput/error drill-down is left for a later expandable detail view. Verified on a B300 SXM6 node (18/18 links active, NVLink 5), matching nvmlDeviceGetNvLinkState enumeration.
Two interconnect-visibility additions to the GPU Detail panel:
- C2C: on Grace systems (GH200/GB200) the coherent CPU<->GPU NVLink-C2C
fabric carries several times the host bandwidth of the PCIe link nvtop
already shows, yet was invisible. Read the C2C link count/status/max-BW
fields and show the aggregate capacity ("C2C 224GB/s"), red when a link
is inactive. NOT_SUPPORTED on non-Grace boards, where the block collapses.
- NVSwitch: probe the NVLink remote device type and tag the link roll-up
with "sw" when the links terminate on an NVSwitch, distinguishing a
switched DGX/NVL72 fabric from direct GPU-to-GPU wiring.
Verified across two real nodes: a B300 SXM6 x86 node ("NVLink 18/18 sw",
no C2C) and a GB200 Grace node ("NVLink 18/18 sw" + "C2C 224GB/s",
5 links x 44712 MB/s).
Address a pre-push review of the GPU Detail additions: drop fields that were extracted but rendered nowhere rather than carry speculative scope. - Remove the FP16/FP32/FP64/INTEGER CUDA-core GPM pipes (and the four GPM metric slots they sampled): the tensor MMA-class label only needs HMMA/IMMA/DFMA, so numMetrics drops 11 -> 7. - Remove the perf-policy throttle-second counters and the correctable row-remap count: these were boot-cumulative and need rate-differencing to display meaningfully, deferred rather than stored unused. - Keep the uncorrectable row-remap count and now render it: a nonzero value (spent remap budget, RMA proximity) appends "<n>u" in yellow to the Rmp field, shown only when nonzero so a healthy row stays unchanged. - Label C2C as "<n>GB/s cap" to mark it max capacity, not live throughput, distinguishing it from the adjacent live NVLink Rx/Tx. Re-verified on B300 (compute "FP", NVLink 18/18 sw, HBM 46c) and GB200 (C2C 224GB/s cap). Build clean; struct/valid-enum still 1:1.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds a GPU Detail bottom panel with NVLink, C2C, SM/Tensor, and health metrics, plus optional host RDMA/GDS readout. The NVIDIA backend now collects the required telemetry, layouts reserve space for the panel, config persists the option, and plots/snapshots expose the new values. ChangesGPU Detail Panel and Telemetry
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/interface_layout_selection.c`:
- Around line 251-252: The bottom panel visibility check in the interface layout
selection logic is too strict and prevents showing the panel on exact-fit
terminals. Update the conditional in the layout-sizing flow around
rows_for_bottom_panel handling so that the bottom panel is kept when rows
exactly equals min_rows_for_header + rows_for_bottom_panel + 1, while still
preserving the existing fallback behavior for smaller terminals. Use the rows,
min_rows_for_header, and rows_for_bottom_panel comparison in this branch to
adjust the threshold accordingly.
- Around line 292-295: The plot-to-process height reallocation in the layout
logic is using proc_count alone, which can still reserve space for an empty
process pane when no process fields are displayed. Update the condition around
the plot_budget adjustment in interface_layout_selection.c to also require
process_field_displayed_count(process_displayed) > 0, so rows are only shifted
from plots when the process pane can actually render content.
In `@src/interface_options.c`:
- Line 132: The default initialization of show_gpu_detail_panel in
interface_options.c is unintentionally enabling downstream
gpuinfo_collect_compute_activity for fresh/default configs. Update the interface
options setup so the merged GPU detail panel is not enabled by default, or
separate the panel-visibility flag from SM/Tensor sampling in the initialization
path. Keep the change localized to the options initialization logic that sets
show_gpu_detail_panel and any related defaults used by
gpuinfo_collect_compute_activity.
In `@src/interface.c`:
- Around line 2234-2322: The health cluster in interface.c can still overflow
the right border because the counter fields printed in the GPU health block are
unbounded. Update the health rendering logic around the health cluster block to
check available width before each numeric `wprintw`, especially for ECC,
remapped counts, NVLink errors, PCIe replay, and HBM temperature, or
pre-truncate those values to fit. Use the existing `getcurx(win)`/`maxx` guard
pattern in the same rendering section so the output stays within the reserved
health area.
- Around line 2354-2362: The GPM sampling gate in interface.c is too broad
because gpuinfo_collect_compute_activity is tied only to show_gpu_detail_panel,
which can stay true even when the bottom panel is hidden. Update this logic so
the actual detail window state is checked before enabling sampling, and keep the
existing plot scan using gpu_specific_opts and
plot_isset_draw_info(plot_sm_util/plot_tensor_util) to allow sampling only when
SM/Tensor data is really being shown.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: c2c8a4e7-cb9a-472b-86b1-c88e2fe8620a
📒 Files selected for processing (13)
CMakeLists.txtinclude/nvtop/extract_gpuinfo_common.hinclude/nvtop/interface_common.hinclude/nvtop/interface_internal_common.hinclude/nvtop/interface_layout_selection.hinclude/nvtop/interface_options.hsrc/CMakeLists.txtsrc/extract_gpuinfo.csrc/extract_gpuinfo_nvidia.csrc/interface.csrc/interface_layout_selection.csrc/interface_options.csrc/interface_setup_win.c
- width-guard each health counter (ECC/remap/NVLink/PCIe/HBM) against the right border so an unexpectedly large value can't wrap past the panel edge; the existing entry check only reserves the common-case width - only arm GPM sampling when the detail-panel window is actually rendered (gpu_detail_win != NULL), not merely when the option is set, so a layout-suppressed panel no longer perturbs the shared perfmon counters - skip the plot-shrink rebalance when no process columns are displayed, so rows aren't shifted into a blank process pane
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a richer NVIDIA datacenter telemetry layer to the GPU Detail panel and a few
layout refinements. Validated on real hardware (A100 8-GPU, B300, GB200/Grace) across
both x86-64 and aarch64 builds. Default full-backend
cmakebuild is green.This stack is grouped into themes for an eventual upstream split; opening it here on the
fork first so it can be reviewed end-to-end before anything is proposed upstream.
What's in here (8 commits, on top of
master@ 3d4a953)Foundation
c6f83a8NVLink/GPM/health metrics + a merged GPU Detail panel5758990derive NVLink peak speed from link version on Blackwell382cc60cap plot height for few-GPU layouts when processes are shownCompute breakdown (GPM)
ab99c43per-pipe GPM compute breakdown with a tensor MMA-class label(SM / occupancy / DRAM-BW + HMMA/IMMA/DFMA pipe activity). GPM capability is decided
by the actual
nvmlGpmSampleGetreturn rather thanisSupportedDevice, which isunreliable on some Blackwell SKUs. The tensor label names the MMA class only (no
precision-width claim) because GPM exposes no FP8/FP4 bucket.
Health
e793331HBM / memory temperature and health-counter plumbing(ECC SBE/DBE, remap pending/failure, NVLink error counters, PCIe replay, mem temp)
Interconnect
7c75ac7NVLink active-link roll-up in the GPU Detail panel (active/total, NVSwitch detect)f59c8f8NVLink-C2C fabric surfacing (Grace coherent fabric capacity)Cleanup
10c6ea9trim speculative fields, render uncorrectable-remap, clarify C2CTesting
nvidia-smi, peak resolves correctly;health counters read real values (incl. a non-zero NVLink error count on one GPU).
dcgmito three decimals; FP8 GEMM observed in the HMMA bucket(kernel verified via profiler), confirming the MMA-class label is honest.
Notes
to stay fork-only; the compute / interconnect pieces are the upstream-candidate parts.
Summary by CodeRabbit
New Features
Bug Fixes