GPU tier: real-hardware smoke test on an NVIDIA node (ship-blocker for GA)

[KeiaiLab-PHIL]

Context

The GPU tier (M2b) is implemented and merged, but its verification ceiling on the dev machine is a compile-check only — there is no NVIDIA GPU available locally. Every layer below the NVML syscall boundary is tested with zero hardware:

• classifyXid / decodeThrottle — pure-Go unit tests
• GPU collector Collect + XID Events() — tested against a hand-written gpuReader fake (no go-nvml import)
• agent async-drain — tested with a fake EventSource
• gpu_nvml.go (the real go-nvml binding, //go:build gpu) — compile-checked in golang:1.26-bookworm via CGO_ENABLED=1 go build -tags gpu ./... (no CUDA/driver needed — NVML is dlopen'd at runtime)

What is NOT yet verified: the actual NVML runtime path on a real GPU node — nvml.Init(), device enumeration, the per-metric Get* calls returning sane values, and the XID EventSet subscription (set.Wait) actually delivering an event.

What this issue tracks (ship-blocker for a GA/v1 release)

Run the :v-gpu image on a real NVIDIA node and confirm:

• nodevitals-gpu DaemonSet schedules on a GPU node (--set tiers.gpu.enabled=true) and the pod reaches Ready
• curl http://<pod-ip>:9847/metrics | grep nodevitals_hw_gpu_ shows non-zero gpu_utilization_pct, gpu_mem_used_bytes, gpu_temperature_celsius, gpu_power_watts for each GPU (device label gpu0, gpu1, …)
• gpu_mem_total_bytes matches the card's real VRAM
• gpu_ecc_uncorrected_total present (0 on healthy hardware), typed as a counter, and sourced from aggregate (lifetime) ECC so it survives a driver reload without resetting
• gpu_throttle_reasons decodes to a sane mask (0 when idle/unthrottled)
• XID path: induce or wait for an XID event and confirm a webhook fires with condition=gpu_xid_error, correct severity (per classifyXid), and detail.xid set
• Clean shutdown: pod termination doesn't hang (the Close()↔watchXid channel lifecycle drains within the 1s set.Wait window) and no "send on closed channel" panic in logs
• Unprivileged securityContext holds on the real node (the NVIDIA container toolkit injects the driver; no caps/privileged needed) with NVIDIA_VISIBLE_DEVICES=all + NVIDIA_DRIVER_CAPABILITIES=utility

How

Either a self-hosted GPU runner in CI (preferred — makes it a permanent gate) or a manual run on a GPU node with the results pasted here. Until this is checked, the GPU tier is documented as compile-checked / unverified-on-hardware.

Refs

• design: docs/superpowers/specs/2026-07-18-nodevitals-m2b-gpu-design.md
• compile-check: make gpu-check
• merged in: a6f06b9 (Merge M2b: GPU tier)