Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

A command-line tool for node provisioning (CUDA toolkit, Nvidia recommended drivers, libraries) and GPU bandwidth benchmarking via `nvbandwidth`. See https://github.com/NVIDIA/nvbandwidth.

Supports all major NVIDIA GPU architectures including the **CoreSpan 5090 Inference System** (Blackwell / RTX 5090, GB202). The provisioning script automatically detects the RTX 5090, selects CUDA 12.8, and installs driver 570+.

## Building from Source

```bash
Expand Down Expand Up @@ -72,6 +74,28 @@ ai-studio-cli nvbandwidth test host_to_device_memcpy_sm

---

## NVLink Benchmarking (CoreSpan 5090 Inference System)

The CoreSpan 5090 Inference System connects multiple RTX 5090 GPUs via NVLink. Use the dedicated `nvlink` subcommand to validate peer-to-peer interconnect health and bandwidth:

```bash
ai-studio-cli nvbandwidth nvlink
```

This runs six peer-to-peer testcases covering unidirectional and bidirectional NVLink transfers as well as all-to-host / host-to-all patterns.

```bash
# JSON output (for monitoring / alerting pipelines)
ai-studio-cli nvbandwidth nvlink --json

# Larger buffer and more samples for sustained-bandwidth measurement
ai-studio-cli nvbandwidth nvlink --buffer-size 1024 --samples 10
```

The same flags as `run` and `test` apply.

---

## Usage Notes

- **Sudo Password**: The tool will securely prompt for your sudo password interactively during setup.
Expand Down
2 changes: 1 addition & 1 deletion ai-studio-cli/assets/driversInstallation.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ detect_gpu() {
elif echo "$GPU_UPPER" | grep -qE "A100|A6000|A30 |A10 |RTX 30|AMPERE|RTX A[0-9]"; then ARCH="ampere"
elif echo "$GPU_UPPER" | grep -qE "RTX 40|L40|ADA|RTX 6000 ADA"; then ARCH="ada"
elif echo "$GPU_UPPER" | grep -qE "H100|H200|HOPPER"; then ARCH="hopper"
elif echo "$GPU_UPPER" | grep -qE "RTX 50|B100|B200|BLACKWELL"; then ARCH="blackwell"
elif echo "$GPU_UPPER" | grep -qE "RTX 5090|RTX 5080|RTX 5070|RTX 50|B100|B200|B300|GB202|BLACKWELL"; then ARCH="blackwell"
fi
fi

Expand Down
57 changes: 57 additions & 0 deletions ai-studio-cli/cmd/nvlink.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
package cmd

import (
"github.com/corespan/ai-studio-cli/internal/nvbandwidth"
"github.com/spf13/cobra"
)

// nvlinkRunner shares flags with the top-level nvbwRunner but is a distinct
// instance so the NVLink subcommand can override defaults independently.
var nvlinkRunner = nvbandwidth.NewRunner()

// nvlinkTestcases are the nvbandwidth testcases that exercise NVLink
// peer-to-peer transfers. These are the most relevant benchmarks for
// multi-GPU inference systems such as the CoreSpan 5090.
var nvlinkTestcases = []string{
"device_to_device_memcpy_read_sm",
"device_to_device_memcpy_write_sm",
"device_to_device_bidirectional_memcpy_read_sm",
"device_to_device_bidirectional_memcpy_write_sm",
"all_to_host_memcpy_sm",
"host_to_all_memcpy_sm",
}

var nvlinkCmd = &cobra.Command{
Use: "nvlink",
Short: "Run NVLink peer-to-peer bandwidth benchmarks",
Long: `Runs the nvbandwidth testcases that exercise GPU-to-GPU (NVLink)
bandwidth. Useful for validating multi-GPU interconnect health on systems
such as the CoreSpan 5090 Inference System.

Testcases run:
- device_to_device_memcpy_read_sm
- device_to_device_memcpy_write_sm
- device_to_device_bidirectional_memcpy_read_sm
- device_to_device_bidirectional_memcpy_write_sm
- all_to_host_memcpy_sm
- host_to_all_memcpy_sm`,
RunE: func(cmd *cobra.Command, args []string) error {
for _, tc := range nvlinkTestcases {
if err := nvlinkRunner.Run([]string{"--testcase", tc}); err != nil {
return err
}
}
return nil
},
}

func init() {
nvlinkCmd.Flags().IntVarP(&nvlinkRunner.BufferSize, "buffer-size", "b", 512, "Memcpy buffer size in MiB")
nvlinkCmd.Flags().IntVarP(&nvlinkRunner.TestSamples, "samples", "i", 3, "Number of benchmark iterations")
nvlinkCmd.Flags().BoolVarP(&nvlinkRunner.Verbose, "verbose", "v", false, "Enable verbose output")
nvlinkCmd.Flags().BoolVarP(&nvlinkRunner.JSON, "json", "j", false, "Output results as JSON")
nvlinkCmd.Flags().BoolVarP(&nvlinkRunner.SkipVerify, "skip-verify", "s", false, "Skip data verification after copy")
nvlinkCmd.Flags().BoolVarP(&nvlinkRunner.UseMean, "use-mean", "m", false, "Use arithmetic mean instead of median")

nvbandwidthCmd.AddCommand(nvlinkCmd)
}
Loading