[Primus Preflight] Add cluster rdma env recommender tool & cluster NIC rdma perf tool from clusterSphere (distinf tools)

cluster_sphere_env.md

@@ -0,0 +1,63 @@
+## Cluster Sphere — RDMA environment recommendations
+
+### Host `useocpm2m-097-078`
+
+**Warnings:**
+
+- Multiple firmware versions detected — standardization recommended.
+
+| RDMA | PCI | NETDEV | Firmware | GID idx | GID | Vendor |
+|------|-----|--------|----------|---------|-----|--------|
+| mlx5_0 | 0000:0c:00.0 | rdma0 | 28.47.1900 | 3 | ::ffff:10.224.0.73 | MLNX |
+| mlx5_1 | 0000:1f:00.0 | eth0 | 22.47.1088 | 3 | ::ffff:10.158.212.73 | MLNX |
+| mlx5_2 | 0000:2a:00.0 | rdma1 | 28.47.1900 | 3 | ::ffff:10.224.4.73 | MLNX |
+| mlx5_3 | 0000:41:00.0 | rdma2 | 28.47.1900 | 3 | ::ffff:10.224.8.73 | MLNX |
+| mlx5_4 | 0000:58:00.0 | rdma3 | 28.47.1900 | 3 | ::ffff:10.224.12.73 | MLNX |
+| mlx5_5 | 0000:86:00.0 | rdma4 | 28.47.1900 | 3 | ::ffff:10.224.16.73 | MLNX |
+| mlx5_6 | 0000:9a:00.0 | eth1 | 22.47.1088 | - | N/A | MLNX |
+| mlx5_7 | 0000:a5:00.0 | rdma5 | 28.47.1900 | 3 | ::ffff:10.224.20.73 | MLNX |
+| mlx5_8 | 0000:bd:00.0 | rdma6 | 28.47.1900 | 3 | ::ffff:10.224.24.73 | MLNX |
+| mlx5_9 | 0000:d5:00.0 | rdma7 | 28.47.1900 | 3 | ::ffff:10.224.28.73 | MLNX |
+
+**Suggested NCCL / socket exports:**
+
+```bash
+export NCCL_IGNORE_CPU_AFFINITY=1
+export NCCL_IB_GID_INDEX=3
+export NCCL_IB_HCA=mlx5_0,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_7,mlx5_8,mlx5_9
+export NCCL_SOCKET_IFNAME=eth0
+export GLOO_SOCKET_IFNAME=eth0
+```
+
+**Suggested rocSHMEM exports:**
+
+```bash
+export ROCSHMEM_HEAP_SIZE=7524589824
+export ROCSHMEM_MAX_NUM_CONTEXTS=256
+```
+
+**Example Docker launch (vendor-specific template):**
+
+```bash
+docker run --rm -it \
+    --device /dev/dri \
+    --device /dev/infiniband \
+    --device /dev/kfd \
+    --network host \
+    --ipc host \
+    --privileged \
+    --ulimit memlock=-1:-1 \
+    --group-add video \
+    --cap-add SYS_PTRACE \
+    --security-opt seccomp=unconfined \
+    --shm-size 64G \
+    -v /sys:/sys \
+    -v $HOME/.ssh:/root/.ssh \
+    -v $HOME:$HOME \
+    -v /dev/infiniband:/dev/infiniband \
+    -v /sys/class/infiniband:/sys/class/infiniband:ro \
+    -v /sys/class/net:/sys/class/net:ro \
+    -v /sys/bus/pci:/sys/bus/pci:ro \
+    <image>
+```
+

docs/preflight.md

@@ -53,6 +53,15 @@ Selection:
 - `--network`: network info
 - `--perf-test`: run perf tests only (GEMM + comm). This is slower.
 
+Cluster Sphere (built into Primus under `primus/tools/preflight/cluster_sphere/`):
+- `--cluster-sphere`: enable **both** Cluster Sphere behaviors below when dependencies are present.
+- `--cluster-sphere-env`: add **RDMA env recommendations** (NCCL/GLOO/rocSHMEM export hints from local verbs devices) to the **info** report (`preflight_report*.md`).
+- `--cluster-sphere-rdma-bw`: append **Verbs `ib_write_bw`** results to the **perf** report (`*_perf.md`). Intended for **`WORLD_SIZE=2`** only (two processes, typically one per node). Requires [linux-rdma/perftest](https://github.com/linux-rdma/perftest) (`ib_write_bw` on `PATH`). Optional: set `PRIMUS_IB_WRITE_BW_PORT` (default `2000`).
+
+Optional override: set **`PRIMUS_CLUSTER_SPHERE_ROOT`** to a directory only if you need to point at a custom/fork copy of the integration (defaults to the in-tree package path).
+
+Preflight’s existing perf tests measure **PyTorch / NCCL-style** GPU communication and use sysfs link rate as a roofline; they are **not** a substitute for raw Verbs bandwidth. Cluster Sphere **`ib_write_bw`** validates the **host RDMA path** separately.
+
 Reporting:
 - `--dump-path`: output directory (default: `output/preflight`)
 - `--report-file-name`: base report name (default: `preflight_report`)
@@ -69,10 +78,70 @@ Backward compatibility:
 By default, outputs are written under `output/preflight`.
 
 Typical report files:
-- `preflight_report.md` / `preflight_report.pdf`: **info report** (host/GPU/network)
-- `preflight_report_perf.md` / `preflight_report_perf.pdf`: **perf report** (GEMM + comm tests)
+- `preflight_report.md` / `preflight_report.pdf`: **info report** (host/GPU/network, plus Cluster Sphere env section when enabled)
+- `preflight_report_perf.md` / `preflight_report_perf.pdf`: **perf report** (GEMM + comm tests, plus optional `ib_write_bw` section)
+
+## Slurm: Cluster Sphere without preflight / torchrun
+
+Use this when you only need **host RDMA validation** and **NCCL-style export hints**, not full GPU preflight (no `torchrun`, no `primus-cli preflight` required). Run on **compute nodes** so `/sys/class/infiniband` and `ibv_devinfo` match real jobs.
+
+**Checklist**
+
+- Primus checkout on nodes (or bind-mount); set `PYTHONPATH` to the repo root that contains the `primus/` package.
+- `ibv_devinfo` (often `rdma-core`), `lspci` for PCI vendor detection.
+- For Pipeline B: [perftest](https://github.com/linux-rdma/perftest) (`ib_write_bw` on `PATH`).
+- Your site may require Slurm `-t`, `-p`, `-A`, etc.
+
+### Pipeline A — RDMA env recommender (one node)
+
+Produces a NIC-oriented Markdown report (GID, per-device firmware, a grouped **firmware report** when multiple NICs exist, suggested `NCCL_*` / `GLOO_*` exports).
+
+```bash
+export PRIMUS_ROOT=/path/to/Primus
+export PYTHONPATH="${PRIMUS_ROOT}"
+
+srun -N 1 -n 1 -t 00:30:00 bash -lc '
+  cd "${PRIMUS_ROOT}" && python3 -m primus.tools.preflight.cluster_sphere env --markdown \
+    > cluster_sphere_env_${SLURM_JOB_ID:-local}.md
+'
+```
+
+Or use the helper script: [`scripts/slurm/cluster_sphere_env_single_node.sh`](../scripts/slurm/cluster_sphere_env_single_node.sh).
+
+### Pipeline B — Verbs `ib_write_bw` (two nodes)
+
+Server listens on **node A**; client on **node B** connects to **`SERVER_RDMA_IP`** — the address of **A on the RDMA/RoCE network**, not necessarily the management DNS name. Obtain it from Pipeline A output / `ip` on the HCA netdev.
+
+Manual (two terminals or SSH into allocated nodes):
+
+```bash
+# Node A
+export PYTHONPATH=/path/to/Primus
+python3 -m primus.tools.preflight.cluster_sphere verbs-server --device rdma0
+
+# Node B (after server is up)
+python3 -m primus.tools.preflight.cluster_sphere verbs-client --server-ip <A_RDMA_IP> --device rdma0
+```
+
+Port: `2000` or `PRIMUS_IB_WRITE_BW_PORT`.
+
+**Single `srun` (server + client)** — set `SERVER_RDMA_IP` to the **server host’s** RDMA address (task 0 / first node); task 1 runs the client after a short delay:
+
+```bash
+export SERVER_RDMA_IP=<node_A_RDMA_IP>
+export PYTHONPATH=/path/to/Primus
+srun -N 2 -n 2 -t 00:30:00 env PYTHONPATH="${PYTHONPATH}" SERVER_RDMA_IP="${SERVER_RDMA_IP}" \
+  python3 -m primus.tools.preflight.cluster_sphere verbs-pair --device rdma0
+```
+
+Automated helper (same allocation, two nodes): [`scripts/slurm/cluster_sphere_ib_write_bw_two_node.sh`](../scripts/slurm/cluster_sphere_ib_write_bw_two_node.sh) — set `SERVER_RDMA_IP` before running.
+
+See [`scripts/slurm/README.md`](../scripts/slurm/README.md) for examples.
+
+---
 
 ## Notes
 
 - For multi-node runs, use `primus-cli slurm …` (or your preferred launcher) so distributed environment variables are set correctly.
 - If you only want a quick environment snapshot, prefer `--host --gpu --network`.
+- `--perf-test` skips the info report; use `--cluster-sphere-env` without `--perf-test` if you only need RDMA export hints.

primus/cli/subcommands/preflight.py

@@ -13,6 +13,7 @@
     primus-cli preflight --gpu                        # GPU info only
     primus-cli preflight --network                    # Network info only
     primus-cli preflight --perf-test                  # Perf tests only (skip info)
+    primus-cli preflight --cluster-sphere             # Optional Cluster Sphere RDMA env + ib_write_bw
 """
 
 from __future__ import annotations
@@ -43,6 +44,7 @@ def register_subcommand(subparsers):
         primus-cli preflight --gpu                        # GPU info only
         primus-cli preflight --network                    # Network info only
         primus-cli preflight --perf-test                  # Perf only
+        primus-cli preflight --cluster-sphere             # Cluster Sphere env + ib_write_bw
     """
     from primus.tools.preflight.preflight_args import add_preflight_parser
 

primus/tools/preflight/README.md

@@ -13,7 +13,97 @@ primus-cli preflight \
   --report-file-name preflight_report
 ```
 
-Slurm (multi-node example):
+### Cluster Sphere (optional)
+
+RDMA environment hints (NCCL/GLOO/rocSHMEM) from local InfiniBand/RoCE devices, plus optional Verbs `ib_write_bw` on two ranks:
+
+```bash
+# Info report: export hints only
+primus-cli preflight --cluster-sphere-env --dump-path output/preflight
+
+# Full preflight + Cluster Sphere env + ib_write_bw on the perf report (use WORLD_SIZE=2 for the Verbs test)
+primus-cli slurm srun -N 2 --ntasks-per-node=1 -- primus-cli preflight --cluster-sphere
+```
+
+Cluster Sphere logic ships inside Primus; set **`PRIMUS_CLUSTER_SPHERE_ROOT`** only if overriding that path. Install **perftest** for `ib_write_bw`.
+
+### Cluster Sphere without torchrun (Slurm or local)
+
+Use the **module CLI** when you want RDMA checks **without** `torchrun`, `torch.distributed`, or `primus-cli preflight` (no GPU preflight / NCCL process group). Run on a **compute node** (via `srun` / `salloc`) when possible so verbs devices match real jobs—not only on the login node.
+
+**Setup (every invocation)**
+
+```bash
+export PRIMUS_ROOT=/path/to/Primus          # repo root containing the primus/ package
+export PYTHONPATH="${PRIMUS_ROOT}${PYTHONPATH:+:${PYTHONPATH}}"
+```
+
+Add your site’s Slurm flags (`-t`, `-p`, `-A`, …) where shown. Optional: **`PRIMUS_CLUSTER_SPHERE_ROOT`** only if you override the in-tree integration path.
+
+**Pipeline A — RDMA env recommender (one host, NIC report)**
+
+Uses `ibv_devinfo` / sysfs (no GPU, no Torch). Writes Markdown with device table, a **firmware report** (grouped by NIC firmware version), and suggested NCCL/GLOO/rocSHMEM exports.
+
+```bash
+# From a shell with PYTHONPATH set (local or already on a compute node):
+python3 -m primus.tools.preflight.cluster_sphere env --markdown > cluster_sphere_env.md
+
+# Quick text summary to stderr instead of Markdown:
+python3 -m primus.tools.preflight.cluster_sphere env
+
+# Slurm — one allocate step, write report in cwd (add -p/-A/-t as needed):
+srun -N 1 -n 1 -t 00:30:00 bash -lc \
+  'cd "${HOME}" && python3 -m primus.tools.preflight.cluster_sphere env --markdown > cluster_sphere_env_${SLURM_JOB_ID:-local}.md'
+```
+
+Helper script (same idea; sets `PYTHONPATH` from `PRIMUS_ROOT`):  
+[`scripts/slurm/cluster_sphere_env_single_node.sh`](../../../scripts/slurm/cluster_sphere_env_single_node.sh)
+
+**Pipeline B — Verbs `ib_write_bw` (two hosts)**
+
+Requires [perftest](https://github.com/linux-rdma/perftest) (`ib_write_bw` on `PATH`). Default TCP port **`2000`** or set **`PRIMUS_IB_WRITE_BW_PORT`**. The client must use the server’s **`SERVER_RDMA_IP`** on the **RDMA/RoCE network** (from Pipeline A output or `ip` on the HCA netdev)—not necessarily the hostname’s management IP.
+
+1. Allocate two nodes (`salloc -N 2 -n 2 -t 01:00:00 …`).
+
+2. **Server (node A)** — blocks until the run finishes:
+
+```bash
+python3 -m primus.tools.preflight.cluster_sphere verbs-server --device mlx5_0
+# Omit --device to pick the first device under /sys/class/infiniband
+```
+
+3. **Client (node B)** — after the server is listening:
+
+```bash
+python3 -m primus.tools.preflight.cluster_sphere verbs-client \
+  --server-ip "${SERVER_RDMA_IP}" --device mlx5_0
+```
+
+**Single Slurm step (recommended)** — one `srun` launches server + client (`SERVER_RDMA_IP` = server node’s address on the RDMA network; task 0 is the first node in the allocation, task 1 the second):
+
+```bash
+export SERVER_RDMA_IP=<first_node_RDMA_IP>
+srun -N 2 -n 2 -t 00:30:00 env PYTHONPATH="${PYTHONPATH}" SERVER_RDMA_IP="${SERVER_RDMA_IP}" \
+  python3 -m primus.tools.preflight.cluster_sphere verbs-pair --device mlx5_0
+```
+
+Optional: `--client-delay 15` (default) gives the server time to listen before the client connects.
+
+Example coordinating two separate `srun` steps (replace `node-a` / `node-b` with hosts from `scontrol show hostnames "$SLURM_JOB_NODELIST"`; add `--overlap` on the first line if your Slurm allows overlapping steps):
+
+```bash
+srun -N 1 -n 1 -w node-a python3 -m primus.tools.preflight.cluster_sphere verbs-server --device mlx5_0 &
+sleep 15
+srun -N 1 -n 1 -w node-b python3 -m primus.tools.preflight.cluster_sphere verbs-client \
+  --server-ip "${SERVER_RDMA_IP}" --device mlx5_0
+```
+
+Automated helper (requires **`export SERVER_RDMA_IP=…`**):  
+[`scripts/slurm/cluster_sphere_ib_write_bw_two_node.sh`](../../../scripts/slurm/cluster_sphere_ib_write_bw_two_node.sh) · index: [`scripts/slurm/README.md`](../../../scripts/slurm/README.md).
+
+More detail: **[`docs/preflight.md`](../../../docs/preflight.md)** (*Slurm: Cluster Sphere without preflight / torchrun*).
+
+Slurm (multi-node preflight example):
 
 ```bash
 NUM_NODES=8 srun -N ${NUM_NODES} --ntasks-per-node=1 --cpus-per-task=256 \

primus/tools/preflight/cluster_sphere/__init__.py

@@ -0,0 +1,11 @@
+###############################################################################
+# Copyright (c) 2025, Advanced Micro Devices, Inc. All rights reserved.
+#
+# See LICENSE for license information.
+###############################################################################
+
+"""Cluster Sphere RDMA helpers for Primus Preflight (in-tree under this package)."""
+
+from primus.tools.preflight.cluster_sphere.paths import resolve_cluster_sphere_root
+
+__all__ = ["resolve_cluster_sphere_root"]