diff --git a/mkdocs.yml b/mkdocs.yml index 5de85a37d8..9408576c5c 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -330,8 +330,8 @@ nav: - Qwen 3.6: docs/examples/models/qwen36.md - Accelerators: - AMD: docs/examples/accelerators/amd.md - - TPU: docs/examples/accelerators/tpu.md - Tenstorrent: docs/examples/accelerators/tenstorrent.md + - TPU: docs/examples/accelerators/tpu.md - Reference: - .dstack.yml: - dev-environment: docs/reference/dstack.yml/dev-environment.md diff --git a/mkdocs/docs/concepts/fleets.md b/mkdocs/docs/concepts/fleets.md index d39899bd4a..9ec7ce3920 100644 --- a/mkdocs/docs/concepts/fleets.md +++ b/mkdocs/docs/concepts/fleets.md @@ -114,10 +114,6 @@ To create a fleet, define its configuration in a YAML file. The filename must en [native package manager](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/native-install/index.html) or [AMDGPU installer](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/amdgpu-install.html).) - === "Intel Gaudi" - 2. Hosts with Intel Gaudi accelerators must be pre-installed with [Gaudi software and drivers](https://docs.habana.ai/en/latest/Installation_Guide/Driver_Installation.html#driver-installation). - This must include the drivers, `hl-smi`, and Habana Container Runtime. - === "Tenstorrent" 2. Hosts with Tenstorrent accelerators must be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation). This must include the drivers, `tt-smi`, and HugePages. diff --git a/mkdocs/docs/examples/accelerators/tenstorrent.md b/mkdocs/docs/examples/accelerators/tenstorrent.md index 69971e6895..ecbb56309c 100644 --- a/mkdocs/docs/examples/accelerators/tenstorrent.md +++ b/mkdocs/docs/examples/accelerators/tenstorrent.md @@ -1,16 +1,27 @@ --- title: Tenstorrent -description: Running dev environments, tasks, and services on Tenstorrent Wormhole accelerators +description: Running inference, training, and dev environments on Tenstorrent accelerators --- # Tenstorrent -`dstack` supports running dev environments, tasks, and services on Tenstorrent -[Wormwhole](https://tenstorrent.com/en/hardware/wormhole) accelerators via SSH fleets. +`dstack` supports running inference, training, and dev environments on +Tenstorrent accelerators via SSH fleets. +The examples below assume the Tenstorrent hosts are pre-installed with +[Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation), +including the drivers, `tt-smi`, and HugePages. -??? info "SSH fleets" -
+## Fleets + +Currently, Tenstorrent accelerators are supported via +[SSH fleets](../../concepts/fleets.md#ssh-fleets). + +=== "SSH fleets" + + Start by creating an SSH fleet for your Tenstorrent hosts. + +
```yaml type: fleet @@ -19,16 +30,12 @@ description: Running dev environments, tasks, and services on Tenstorrent Wormho ssh_config: user: root identity_file: ~/.ssh/id_rsa - # Configure any number of hosts with n150 or n300 PCEe boards hosts: - 192.168.2.108 ```
- > Hosts should be pre-installed with [Tenstorrent software](https://docs.tenstorrent.com/getting-started/README.html#software-installation). - This should include the drivers, `tt-smi`, and HugePages. - To apply the fleet configuration, run:
@@ -36,58 +43,62 @@ description: Running dev environments, tasks, and services on Tenstorrent Wormho ```bash $ dstack apply -f tt-fleet.dstack.yml - FLEET RESOURCES PRICE STATUS CREATED - tt-fleet cpu=12 mem=32GB disk=243GB n150:12GB $0 idle 18 sec ago + FLEET RESOURCES PRICE STATUS CREATED + tt-fleet cpu=64 mem=566.1GB disk=749.6GB gpu=tt-galaxy-wh:12GB:32 $0 idle 18 sec ago ```
- For more details on fleet configuration, refer to [SSH fleets](../../concepts/fleets.md#ssh-fleets). - -## Services +## Inference -Here's an example of a service that deploys -[`Llama-3.2-1B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-1B) -using [Tenstorrent Inference Service](https://github.com/tenstorrent/tt-inference-server). +Below is a [service](../../concepts/services.md) that deploys +[`gpt-oss-120b`](https://huggingface.co/openai/gpt-oss-120b) on a +Tenstorrent Galaxy system using +[Tenstorrent Inference Server](https://github.com/tenstorrent/tt-inference-server). -
+
```yaml type: service -name: tt-inference-server +name: gpt-oss-120b + +image: ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-22.04-amd64:0.12.0-805f43d-a45c614 env: - HF_TOKEN - - HF_MODEL_REPO_ID=meta-llama/Llama-3.2-1B-Instruct -image: ghcr.io/tenstorrent/tt-inference-server/vllm-tt-metal-src-release-ubuntu-20.04-amd64:0.0.4-v0.56.0-rc47-e2e0002ac7dc + commands: - - | - . ${PYTHON_ENV_DIR}/bin/activate - pip install "huggingface_hub[cli]" - export LLAMA_DIR="/data/models--$(echo "$HF_MODEL_REPO_ID" | sed 's/\//--/g')/" - huggingface-cli download $HF_MODEL_REPO_ID --local-dir $LLAMA_DIR - python /home/container_app_user/app/src/run_vllm_api_server.py -port: 7000 + - | + ulimit -n 65535 + /home/container_app_user/tt-metal/python_env/bin/python /home/container_app_user/app/src/run_vllm_api_server.py \ + --model gpt-oss-120b \ + --tt-device galaxy + +port: 8000 -model: meta-llama/Llama-3.2-1B-Instruct +model: openai/gpt-oss-120b -# Cache downloaded model volumes: - - /mnt/data/tt-inference-server/data:/data + # Cache model weights and TT runtime artifacts on the host. + - /mnt/data/gpt-oss-120b/cache_root:/home/container_app_user/cache_root + - /mnt/data/gpt-oss-120b/dot-cache:/home/container_app_user/.cache resources: - gpu: n150:1 + shm_size: 32GB + gpu: tt-galaxy-wh:32 ```
-Go ahead and run configuration using `dstack apply`: +Go ahead and run the configuration using `dstack apply`:
- ```bash - $ dstack apply -f service.dstack.yml - ``` +```bash +$ export HF_TOKEN= +$ dstack apply -f service.dstack.yml +``` +
Once the service is up, it will be available via the service endpoint @@ -96,102 +107,199 @@ at `/proxy/services///`.
```shell -$ curl http://127.0.0.1:3000/proxy/services/main/tt-inference-server/v1/chat/completions \ +$ curl http://127.0.0.1:3000/proxy/services/main/gpt-oss-120b/v1/chat/completions \ -X POST \ -H 'Authorization: Bearer <user token>' \ -H 'Content-Type: application/json' \ -d '{ - "model": "meta-llama/Llama-3.2-1B-Instruct", + "model": "openai/gpt-oss-120b", "messages": [ - { - "role": "system", - "content": "You are a helpful assistant." - }, { "role": "user", - "content": "What is Deep Learning?" + "content": "What is 17 + 25? Answer with just the number." } ], - "stream": true, - "max_tokens": 512 + "max_tokens": 128 }' ```
+The response includes both the final answer and the model's reasoning fields +(`reasoning` and `reasoning_content`). + Additionally, the model is available via `dstack`'s control plane UI: ![](https://dstack.ai/static-assets/static-assets/images/dstack-tenstorrent-model-ui.png){ width=800 } -When a [gateway](../../concepts/gateways.md) is configured, the service endpoint +When a [gateway](../../concepts/gateways.md) is configured, the service endpoint is available at `https://./`. -> Services support many options, including authentication, auto-scaling policies, etc. To learn more, refer to [Services](../../concepts/services.md). - -## Tasks +## Training -Below is a task that simply runs `tt-smi -s`. Tasks can be used for training, fine-tuning, batch inference, or antything else. +Below is a minimal [task](../../concepts/tasks.md) that runs a TT-XLA training +smoke test. -
+
```yaml type: task -# The name is optional, if not specified, generated randomly -name: tt-smi +name: tt-xla-train -env: - - HF_TOKEN - -# (Required) Use any image with TT drivers -image: dstackai/tt-smi:latest +image: ghcr.io/tenstorrent/tt-xla-slim:latest -# Use any commands commands: - - tt-smi -s + - | + python - <<'PY' + import jax + import jax.numpy as jnp + + devices = jax.devices("tt") + print("TT devices:", devices) + if not devices: + raise SystemExit("No Tenstorrent devices found by JAX") + + with jax.default_device(devices[0]): + params = { + "w": jnp.ones((32, 32), dtype=jnp.bfloat16), + "b": jnp.zeros((32,), dtype=jnp.bfloat16), + } + x = jnp.ones((32, 32), dtype=jnp.bfloat16) + y = jnp.zeros((32, 32), dtype=jnp.bfloat16) + + def loss_fn(params, x, y): + pred = x @ params["w"] + params["b"] + err = (pred - y).astype(jnp.float32) + return jnp.mean(err * err) + + @jax.jit + def train_step(params, x, y): + loss, grads = jax.value_and_grad(loss_fn)(params, x, y) + next_params = jax.tree_util.tree_map( + lambda p, g: p - jnp.asarray(0.01, dtype=p.dtype) * g.astype(p.dtype), + params, + grads, + ) + return next_params, loss + + for step in range(3): + params, loss = train_step(params, x, y) + loss.block_until_ready() + print(f"step={step} loss={float(jax.device_get(loss)):.6f}") + + print("tiny training smoke test passed") + PY -# Specify the number of accelerators, model, etc resources: - gpu: n150:1 - -# Uncomment if you want to run on a cluster of nodes -#nodes: 2 + gpu: tt-galaxy-wh:32 ```
-> Tasks support many options, including multi-node configuration, max duration, etc. To learn more, refer to [Tasks](../../concepts/tasks.md). +For a single Wormhole PCIe card, use `gpu: n150:1`. + +??? info "Files and repos" + For longer commands, put the Python code in `train.py` next to + `tt-task.dstack.yml` and upload it with `files`: + +
+ + ```yaml + type: task + name: tt-xla-train + + image: ghcr.io/tenstorrent/tt-xla-slim:latest + + files: + - train.py + + commands: + - python train.py + + resources: + gpu: tt-galaxy-wh:32 + ``` + +
+ + If the script is part of a Git repository, use `repos` instead: + +
+ + ```yaml + working_dir: /workspace + + repos: + - .:/workspace + + commands: + - python train.py + ``` + +
+ + For more details, refer to [Files](../../concepts/tasks.md#files) and + [Repos](../../concepts/tasks.md#repos). ## Dev environments -Below is an example of a dev environment configuration. It can be used to provision a dev environemnt that can be accessed via your desktop IDE. +Below is an example [dev environment](../../concepts/dev-environments.md) +configuration. It can be used to provision a dev environment that can be +accessed via your desktop IDE. -
+
```yaml type: dev-environment # The name is optional, if not specified, generated randomly -name: cursor - -# (Optional) List required env variables -env: - - HF_TOKEN +name: vscode image: dstackai/tt-smi:latest # Can be `vscode` or `cursor` -ide: cursor +ide: vscode resources: - gpu: n150:1 + gpu: tt-galaxy-wh:32 ```
If you run it via `dstack apply`, it will output the URL to access it via your desktop IDE. -![](https://dstack.ai/static-assets/static-assets/images/dstack-tenstorrent-cursor.png){ width=800 } +![](https://dstack.ai/static-assets/static-assets/images/dstack-tenstorrent-vscode.png){ width=800 } + +## GPU specification + +`resources.gpu` uses the usual `name:count` format. For Tenstorrent, `count` +is the number of devices reported from the TT-SMI topology. On Galaxy systems, +this corresponds to chips. On PCIe systems, this is usually the card count, but +dual-chip cards can also be reported as per-chip devices. + +```yaml +resources: + gpu: tt-galaxy-wh:32 # Galaxy Wormhole, 32 chips + # gpu: tt-galaxy-bh:32 # Galaxy Blackhole, 32 chips + # gpu: n300:4 # TT-LoudBox or TT-QuietBox Wormhole, 4 n300 cards + # gpu: p150:4 # TT-QuietBox Blackhole, 4 p150 cards + # gpu: p300:64GB:2 # TT-QuietBox 2 Blackhole, 2 p300 cards + # gpu: p300:32GB:4 # TT-QuietBox 2 Blackhole, if exposed per chip +``` + +Use `tt:` only when the workload can run on any Tenstorrent device type. +Use a model name when placement depends on the hardware family: `n150` or +`n300` for Wormhole PCIe cards, `tt-galaxy-wh` for Galaxy Wormhole, `p100a`, +`p150`, or `p300` for Blackhole PCIe cards, and `tt-galaxy-bh` for Galaxy +Blackhole. + +## What's next? -> Dev nevironments support many options, including inactivity and max duration, IDE configuration, etc. To learn more, refer to [Dev environments](../../concepts/tasks.md). +1. Check [Services](../../concepts/services.md), + [Tasks](../../concepts/tasks.md), [Dev environments](../../concepts/dev-environments.md), + and [SSH fleets](../../concepts/fleets.md#ssh-fleets). +2. Browse [Tenstorrent Inference Server](https://github.com/tenstorrent/tt-inference-server), + [TT-XLA](https://github.com/tenstorrent/tt-xla), and + [TT-Metalium](https://github.com/tenstorrent/tt-metal). ??? info "Feedback" Found a bug, or want to request a feature? File it in the [issue tracker](https://github.com/dstackai/dstack/issues), diff --git a/mkdocs/docs/guides/protips.md b/mkdocs/docs/guides/protips.md index 9f800a956e..065556ef11 100644 --- a/mkdocs/docs/guides/protips.md +++ b/mkdocs/docs/guides/protips.md @@ -382,13 +382,14 @@ To run in detached mode, use `-d` with `dstack apply`. ## GPU specification -`dstack` natively supports NVIDIA GPU, AMD GPU, and Google Cloud TPU accelerator chips. +`dstack` natively supports NVIDIA GPUs, AMD GPUs, TPUs, and Tenstorrent +devices. The `gpu` property within [`resources`](../reference/dstack.yml/dev-environment.md#resources) (or the `--gpu` option with [`dstack apply`](../reference/cli/dstack/apply.md) or [`dstack offer`](../reference/cli/dstack/offer.md)) allows specifying not only memory size but also GPU vendor, names, their memory, and quantity. -The general format is: `:::`. +The general format is: `:::`. Each component is optional. @@ -412,13 +413,18 @@ Examples: - `A100:2` (two A100) - `MI300X:4` (four MI300X) - `A100:40GB:2` (two A100 40GB) -- `tpu:v2-8` (`v2` Google Cloud TPU with 8 cores) +- `tpu:v2-8` (`v2` TPU with 8 cores) +- `tt:32` (32 Tenstorrent devices) +- `tt-galaxy-wh:32` (32 Galaxy Wormhole chips) +- `tt-galaxy-bh:32` (32 Galaxy Blackhole chips) +- `p150:8` (eight Tenstorrent Blackhole P150 devices) The GPU vendor is indicated by one of the following case-insensitive values: - `nvidia` (NVIDIA GPUs) - `amd` (AMD GPUs) -- `tpu` (Google Cloud TPUs) +- `tpu` (TPUs) +- `tt` (Tenstorrent devices) ??? info "AMD" Currently, when an AMD GPU is specified, either by name or by vendor, the `image` property must be specified as well. diff --git a/mkdocs/docs/guides/troubleshooting.md b/mkdocs/docs/guides/troubleshooting.md index 7b42bd73f1..f3746a54d4 100644 --- a/mkdocs/docs/guides/troubleshooting.md +++ b/mkdocs/docs/guides/troubleshooting.md @@ -113,7 +113,7 @@ If you want to use non-NVIDIA GPUs or multi-GPU instances, set the `gpu` propert in your configuration. Examples: `gpu: amd` (one AMD GPU), `gpu: A10:4..8` (4 to 8 A10 GPUs), -`gpu: 8:Gaudi2` (8 Gaudi2 accelerators). +`gpu: tt:32` (32 Tenstorrent devices). > If you don't specify the number of GPUs, `dstack` will only select single-GPU instances. diff --git a/pyproject.toml b/pyproject.toml index e4083604c7..0b4569698d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -80,6 +80,8 @@ ignore-case = true [tool.uv.sources] dstack-plugin-server = { path = "examples/plugins/example_plugin_server", editable = true } +# TODO: Release gpuhunt with Tenstorrent Blackhole accelerators and restore the PyPI source. +gpuhunt = { git = "https://github.com/dstackai/gpuhunt.git", rev = "refs/pull/232/head" } [tool.ruff] target-version = "py310" diff --git a/runner/internal/shim/host/gpu.go b/runner/internal/shim/host/gpu.go index 10cbc7307f..eff57f2e00 100644 --- a/runner/internal/shim/host/gpu.go +++ b/runner/internal/shim/host/gpu.go @@ -214,6 +214,7 @@ type ttDeviceInfo struct { type ttBoardInfo struct { BoardType string `json:"board_type"` BoardID string `json:"board_id"` + BusID string `json:"bus_id"` } func unmarshalTtSmiSnapshot(data []byte) (*ttSmiSnapshot, error) { @@ -224,45 +225,87 @@ func unmarshalTtSmiSnapshot(data []byte) (*ttSmiSnapshot, error) { return &snapshot, nil } +func normalizeTtBoardName(name string) string { + switch { + case name == "bh-scrappy" || name == "p100": + return "p100a" + case strings.HasPrefix(name, "p150"): + return "p150" + case strings.HasPrefix(name, "p300"): + return "p300" + default: + return name + } +} + +func splitTtBoardType(boardType string) (name string, suffix string) { + boardType = strings.TrimSpace(boardType) + if strings.HasSuffix(boardType, " L") || strings.HasSuffix(boardType, " R") { + suffix = boardType[len(boardType)-1:] + boardType = strings.TrimSpace(boardType[:len(boardType)-2]) + } + return normalizeTtBoardName(boardType), suffix +} + +func ttBoardVramMib(name string) int { + switch { + case strings.HasPrefix(name, "n150"), + strings.HasPrefix(name, "n300"), + strings.HasPrefix(name, "tt-galaxy-wh"): + return 12 * 1024 + } + switch name { + case "p100a": + return 28 * 1024 + case "p150", "p300", "tt-galaxy-bh": + return 32 * 1024 + default: + return 0 + } +} + +func isTtBlackholeBoard(name string) bool { + switch name { + case "p100a", "p150", "p300", "tt-galaxy-bh": + return true + default: + return false + } +} + +func isRemoteTtDevice(device ttDeviceInfo) bool { + return strings.EqualFold(strings.TrimSpace(device.BoardInfo.BusID), "N/A") +} + func getGpusFromTtSmiSnapshot(snapshot *ttSmiSnapshot) []GpuInfo { - // Create a map to track "L" devices and their corresponding "R" devices - // Each "L" device becomes a separate GPU - lDeviceMap := make(map[string]*GpuInfo) + gpuMap := make(map[string]*GpuInfo) + gpuKeys := []string{} indexCounter := 0 + addGpu := func(key string, gpuInfo GpuInfo) { + gpuMap[key] = &gpuInfo + gpuKeys = append(gpuKeys, key) + } // First pass: identify all "L" and "R" devices for i, device := range snapshot.DeviceInfo { boardID := device.BoardInfo.BoardID - boardType := strings.TrimSpace(device.BoardInfo.BoardType) - - // Determine if this is an "L" device - isLDevice := strings.HasSuffix(boardType, " L") + name, suffix := splitTtBoardType(device.BoardInfo.BoardType) - if isLDevice { + if suffix == "L" { // Create unique identifier for this "L" device uniqueID := fmt.Sprintf("%s_L_%d", boardID, i) - // Extract base name without L suffix - name := boardType[:len(boardType)-2] - // Determine base VRAM based on board type - baseVram := 0 - if strings.HasPrefix(name, "n150") { - baseVram = 12 * 1024 // 12GB in MiB - } else if strings.HasPrefix(name, "n300") { - baseVram = 12 * 1024 // 12GB in MiB - } else if strings.HasPrefix(name, "tt-galaxy-wh") { - baseVram = 12 * 1024 // 12GB in MiB - } + baseVram := ttBoardVramMib(name) // Create new GPU entry for "L" device - lDeviceMap[uniqueID] = &GpuInfo{ + addGpu(uniqueID, GpuInfo{ Vendor: gpu.GpuVendorTenstorrent, Name: name, Vram: baseVram, ID: boardID, Index: strconv.Itoa(indexCounter), - } + }) indexCounter++ } } @@ -270,27 +313,17 @@ func getGpusFromTtSmiSnapshot(snapshot *ttSmiSnapshot) []GpuInfo { // Second pass: add memory from "R" devices to corresponding "L" devices for _, device := range snapshot.DeviceInfo { boardID := device.BoardInfo.BoardID - boardType := strings.TrimSpace(device.BoardInfo.BoardType) + name, suffix := splitTtBoardType(device.BoardInfo.BoardType) - if strings.HasSuffix(boardType, " R") { + if suffix == "R" { // Find the corresponding "L" device with the same board_id // Since we need to match "R" to "L", we'll use the board_id as the key // and add memory to the first "L" device we find with that board_id - for _, gpu := range lDeviceMap { - if gpu.ID == boardID { - // Extract base name without R suffix - name := boardType[:len(boardType)-2] - - // Determine base VRAM based on board type - baseVram := 0 - if strings.HasPrefix(name, "n150") { - baseVram = 12 * 1024 // 12GB in MiB - } else if strings.HasPrefix(name, "n300") { - baseVram = 12 * 1024 // 12GB in MiB - } - + for _, key := range gpuKeys { + gpu := gpuMap[key] + if gpu.ID == boardID && (gpu.Name == name || !isTtBlackholeBoard(name)) { // Add memory to the "L" device - gpu.Vram += baseVram + gpu.Vram += ttBoardVramMib(name) break // Only add to the first matching "L" device } } @@ -300,24 +333,35 @@ func getGpusFromTtSmiSnapshot(snapshot *ttSmiSnapshot) []GpuInfo { // Handle devices without L/R suffix (backward compatibility) for i, device := range snapshot.DeviceInfo { boardID := device.BoardInfo.BoardID - boardType := strings.TrimSpace(device.BoardInfo.BoardType) + name, suffix := splitTtBoardType(device.BoardInfo.BoardType) - if !strings.HasSuffix(boardType, " L") && !strings.HasSuffix(boardType, " R") { + if suffix == "" { // For devices without L/R suffix, treat them as standalone GPUs // This maintains backward compatibility with existing data uniqueID := fmt.Sprintf("%s_standalone_%d", boardID, i) // Determine base VRAM based on board type - baseVram := 0 - if strings.HasPrefix(boardType, "n150") { - baseVram = 12 * 1024 // 12GB in MiB - } else if strings.HasPrefix(boardType, "n300") { - baseVram = 12 * 1024 // 12GB in MiB + baseVram := ttBoardVramMib(name) + + if isTtBlackholeBoard(name) { + if isRemoteTtDevice(device) { + continue + } + addGpu(uniqueID, GpuInfo{ + Vendor: gpu.GpuVendorTenstorrent, + Name: name, + Vram: baseVram, + ID: boardID, + Index: strconv.Itoa(indexCounter), + }) + indexCounter++ + continue } // Check if we already have a GPU with this board_id (old behavior) existingGpu := false - for _, gpu := range lDeviceMap { + for _, key := range gpuKeys { + gpu := gpuMap[key] if gpu.ID == boardID { gpu.Vram += baseVram existingGpu = true @@ -327,26 +371,41 @@ func getGpusFromTtSmiSnapshot(snapshot *ttSmiSnapshot) []GpuInfo { if !existingGpu { // Create new GPU entry - lDeviceMap[uniqueID] = &GpuInfo{ + addGpu(uniqueID, GpuInfo{ Vendor: gpu.GpuVendorTenstorrent, - Name: boardType, + Name: name, Vram: baseVram, ID: boardID, Index: strconv.Itoa(indexCounter), - } + }) indexCounter++ } } } + // Add memory from remote Blackhole chips to the matching local board. + for _, device := range snapshot.DeviceInfo { + boardID := device.BoardInfo.BoardID + name, suffix := splitTtBoardType(device.BoardInfo.BoardType) + if suffix != "" || !isTtBlackholeBoard(name) || !isRemoteTtDevice(device) { + continue + } + for _, key := range gpuKeys { + gpu := gpuMap[key] + if gpu.ID == boardID && gpu.Name == name { + gpu.Vram += ttBoardVramMib(name) + break + } + } + } + // Convert map to slice var gpus []GpuInfo - for _, gpu := range lDeviceMap { - gpus = append(gpus, *gpu) + for _, key := range gpuKeys { + gpus = append(gpus, *gpuMap[key]) } - // Sort by the original index to ensure consistent ordering - // We'll reassign indices sequentially based on the original order + // Reassign indices sequentially based on discovery order. for i := range gpus { gpus[i].Index = strconv.Itoa(i) } diff --git a/runner/internal/shim/host/gpu_test.go b/runner/internal/shim/host/gpu_test.go index 9facf9992a..4110816003 100644 --- a/runner/internal/shim/host/gpu_test.go +++ b/runner/internal/shim/host/gpu_test.go @@ -239,6 +239,28 @@ func TestGetGpusFromTtSmiSnapshotMultipleDevices(t *testing.T) { } } +func TestGetGpusFromTtSmiSnapshotWormholePrefixMemoryCompatibility(t *testing.T) { + snapshot := &ttSmiSnapshot{ + DeviceInfo: []ttDeviceInfo{ + {BoardInfo: ttBoardInfo{BoardType: "n150-custom L", BoardID: "100018000000001"}}, + {BoardInfo: ttBoardInfo{BoardType: "n300-custom L", BoardID: "100014000000001"}}, + {BoardInfo: ttBoardInfo{BoardType: "n300-custom R", BoardID: "100014000000001"}}, + {BoardInfo: ttBoardInfo{BoardType: "tt-galaxy-wh-custom L", BoardID: "100035000000001"}}, + }, + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + expected := []GpuInfo{ + {Vendor: gpu.GpuVendorTenstorrent, Name: "n150-custom", Vram: 12 * 1024, ID: "100018000000001", Index: "0"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "n300-custom", Vram: 24 * 1024, ID: "100014000000001", Index: "1"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "tt-galaxy-wh-custom", Vram: 12 * 1024, ID: "100035000000001", Index: "2"}, + } + if !reflect.DeepEqual(gpus, expected) { + t.Errorf("getGpusFromTtSmiSnapshot() = %v, want %v", gpus, expected) + } +} + func TestGetGpusFromTtSmiSnapshotGalaxy(t *testing.T) { data, err := loadTestData("tenstorrent/galaxy.json") if err != nil { @@ -289,3 +311,185 @@ func TestGetGpusFromTtSmiSnapshotGalaxy(t *testing.T) { t.Errorf("Total VRAM = %d MiB, want %d MiB (384GB)", actualTotalVram, totalVram) } } + +func TestGetGpusFromTtSmiSnapshotBlackholeRevisions(t *testing.T) { + snapshot := &ttSmiSnapshot{ + DeviceInfo: []ttDeviceInfo{ + {BoardInfo: ttBoardInfo{BoardType: "bh-scrappy", BoardID: "0000360000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p100", BoardID: "0000430000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p100a", BoardID: "0000430000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p150a", BoardID: "0000400000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p150b", BoardID: "0000410000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p150c", BoardID: "0000420000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p300b", BoardID: "0000440000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p300a", BoardID: "0000450000000000"}}, + {BoardInfo: ttBoardInfo{BoardType: "p300c", BoardID: "0000460000000000"}}, + }, + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + expectedNames := []string{"p100a", "p100a", "p100a", "p150", "p150", "p150", "p300", "p300", "p300"} + expectedVram := []int{28 * 1024, 28 * 1024, 28 * 1024, 32 * 1024, 32 * 1024, 32 * 1024, 32 * 1024, 32 * 1024, 32 * 1024} + if len(gpus) != len(expectedNames) { + t.Fatalf("getGpusFromTtSmiSnapshot() returned %d GPUs, want %d", len(gpus), len(expectedNames)) + } + for i, expectedName := range expectedNames { + if gpus[i].Vendor != gpu.GpuVendorTenstorrent { + t.Errorf("GPU[%d] vendor = %v, want %v", i, gpus[i].Vendor, gpu.GpuVendorTenstorrent) + } + if gpus[i].Name != expectedName { + t.Errorf("GPU[%d] name = %s, want %s", i, gpus[i].Name, expectedName) + } + if gpus[i].Vram != expectedVram[i] { + t.Errorf("GPU[%d] VRAM = %d, want %d", i, gpus[i].Vram, expectedVram[i]) + } + if gpus[i].Index != strconv.Itoa(i) { + t.Errorf("GPU[%d] index = %s, want %s", i, gpus[i].Index, strconv.Itoa(i)) + } + } +} + +func TestGetGpusFromTtSmiSnapshotBlackholeSourceFixtures(t *testing.T) { + // Synthetic tt-smi snapshot derived from TT-SMI board name mappings and + // TT-Metal UMD Blackhole board descriptors. + data, err := loadTestData("tenstorrent/blackhole_boards.json") + if err != nil { + t.Fatalf("Failed to load test data: %v", err) + } + snapshot, err := unmarshalTtSmiSnapshot(data) + if err != nil { + t.Fatalf("Failed to unmarshal snapshot: %v", err) + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + expected := []GpuInfo{ + {Vendor: gpu.GpuVendorTenstorrent, Name: "p100a", Vram: 28 * 1024, ID: "000004323191b040", Index: "0"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "p150", Vram: 32 * 1024, ID: "0000040100000000", Index: "1"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "p150", Vram: 32 * 1024, ID: "000004123191110e", Index: "2"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "p300", Vram: 32 * 1024, ID: "000004513190f004", Index: "3"}, + {Vendor: gpu.GpuVendorTenstorrent, Name: "p300", Vram: 32 * 1024, ID: "000004513190f004", Index: "4"}, + } + if !reflect.DeepEqual(gpus, expected) { + t.Errorf("getGpusFromTtSmiSnapshot() = %v, want %v", gpus, expected) + } +} + +func TestGetGpusFromTtSmiSnapshotBlackholeEightP150(t *testing.T) { + // Derived from TT-Metal UMD blackhole_8xP150 cluster descriptor. + // The p150b name follows TT-SMI's board ID to board type mapping. + data, err := loadTestData("tenstorrent/blackhole_8xp150.json") + if err != nil { + t.Fatalf("Failed to load test data: %v", err) + } + snapshot, err := unmarshalTtSmiSnapshot(data) + if err != nil { + t.Fatalf("Failed to unmarshal snapshot: %v", err) + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + expectedIDs := []string{ + "0000041231915018", + "0000041231915002", + "0000041231915009", + "000004123191500f", + "0000041231914064", + "0000041231915006", + "0000041231914087", + "000004123191402f", + } + if len(gpus) != len(expectedIDs) { + t.Fatalf("getGpusFromTtSmiSnapshot() returned %d GPUs, want %d", len(gpus), len(expectedIDs)) + } + for i, gpu_ := range gpus { + if gpu_.Vendor != gpu.GpuVendorTenstorrent { + t.Errorf("GPU[%d] vendor = %v, want %v", i, gpu_.Vendor, gpu.GpuVendorTenstorrent) + } + if gpu_.Name != "p150" { + t.Errorf("GPU[%d] name = %s, want p150", i, gpu_.Name) + } + if gpu_.Vram != 32*1024 { + t.Errorf("GPU[%d] VRAM = %d, want %d", i, gpu_.Vram, 32*1024) + } + if gpu_.ID != expectedIDs[i] { + t.Errorf("GPU[%d] ID = %s, want %s", i, gpu_.ID, expectedIDs[i]) + } + if gpu_.Index != strconv.Itoa(i) { + t.Errorf("GPU[%d] index = %s, want %s", i, gpu_.Index, strconv.Itoa(i)) + } + } +} + +func TestGetGpusFromTtSmiSnapshotBlackholeP300SameBoardID(t *testing.T) { + snapshot := &ttSmiSnapshot{ + DeviceInfo: []ttDeviceInfo{ + {BoardInfo: ttBoardInfo{BoardType: "p300a", BoardID: "0000450000000000", BusID: "0000:01:00.0"}}, + {BoardInfo: ttBoardInfo{BoardType: "p300a", BoardID: "0000450000000000", BusID: "0000:02:00.0"}}, + }, + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + if len(gpus) != 2 { + t.Fatalf("getGpusFromTtSmiSnapshot() returned %d GPUs, want 2", len(gpus)) + } + for i, gpu_ := range gpus { + if gpu_.Name != "p300" { + t.Errorf("GPU[%d] name = %s, want p300", i, gpu_.Name) + } + if gpu_.Vram != 32*1024 { + t.Errorf("GPU[%d] VRAM = %d, want %d", i, gpu_.Vram, 32*1024) + } + if gpu_.Index != strconv.Itoa(i) { + t.Errorf("GPU[%d] index = %s, want %s", i, gpu_.Index, strconv.Itoa(i)) + } + } +} + +func TestGetGpusFromTtSmiSnapshotBlackholeP300RemoteChip(t *testing.T) { + snapshot := &ttSmiSnapshot{ + DeviceInfo: []ttDeviceInfo{ + {BoardInfo: ttBoardInfo{BoardType: "p300a", BoardID: "0000450000000000", BusID: "0000:01:00.0"}}, + {BoardInfo: ttBoardInfo{BoardType: "p300a", BoardID: "0000450000000000", BusID: "N/A"}}, + }, + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + expected := []GpuInfo{ + {Vendor: gpu.GpuVendorTenstorrent, Name: "p300", Vram: 64 * 1024, ID: "0000450000000000", Index: "0"}, + } + if !reflect.DeepEqual(gpus, expected) { + t.Errorf("getGpusFromTtSmiSnapshot() = %v, want %v", gpus, expected) + } +} + +func TestGetGpusFromTtSmiSnapshotBlackholeGalaxy(t *testing.T) { + data, err := loadTestData("tenstorrent/blackhole_galaxy.json") + if err != nil { + t.Fatalf("Failed to load test data: %v", err) + } + snapshot, err := unmarshalTtSmiSnapshot(data) + if err != nil { + t.Fatalf("Failed to unmarshal snapshot: %v", err) + } + + gpus := getGpusFromTtSmiSnapshot(snapshot) + + if len(gpus) != 32 { + t.Fatalf("getGpusFromTtSmiSnapshot() returned %d GPUs, want 32", len(gpus)) + } + for i, gpu_ := range gpus { + if gpu_.Name != "tt-galaxy-bh" { + t.Errorf("GPU[%d] name = %s, want tt-galaxy-bh", i, gpu_.Name) + } + if gpu_.Vram != 32*1024 { + t.Errorf("GPU[%d] VRAM = %d, want %d", i, gpu_.Vram, 32*1024) + } + if gpu_.Index != strconv.Itoa(i) { + t.Errorf("GPU[%d] index = %s, want %s", i, gpu_.Index, strconv.Itoa(i)) + } + } +} diff --git a/runner/internal/shim/host/testdata/tenstorrent/README.md b/runner/internal/shim/host/testdata/tenstorrent/README.md new file mode 100644 index 0000000000..db7a8cbc4c --- /dev/null +++ b/runner/internal/shim/host/testdata/tenstorrent/README.md @@ -0,0 +1,19 @@ +# Tenstorrent fixtures + +The Wormhole fixtures are captured `tt-smi -s` snapshots used by the existing +Tenstorrent tests. + +The Blackhole Galaxy fixture is a captured `tt-smi -s` snapshot from a 32-chip +Galaxy Blackhole host. + +The Blackhole PCIe fixtures are source-derived compatibility fixtures: + +- `blackhole_boards.json` covers `tt-smi` Blackhole board names and P300 + same-board dual-MMIO behavior. Board names are based on + `tt_smi/utils.py::get_board_type` and UMD board type mappings. +- `blackhole_8xp150.json` is derived from UMD's + `blackhole_8xP150.yaml` cluster descriptor. The board IDs and PCI bus IDs are + from that descriptor; the `p150b` board name follows the UPI mapping used by + `tt-smi`. +These fixtures are not substitutes for live hardware smoke tests. They preserve +captured `tt-smi` JSON shapes and source-derived UMD topology cases. diff --git a/runner/internal/shim/host/testdata/tenstorrent/blackhole_8xp150.json b/runner/internal/shim/host/testdata/tenstorrent/blackhole_8xp150.json new file mode 100644 index 0000000000..c313c5be0f --- /dev/null +++ b/runner/internal/shim/host/testdata/tenstorrent/blackhole_8xp150.json @@ -0,0 +1,60 @@ +{ + "device_info": [ + { + "board_info": { + "bus_id": "0000:01:00.0", + "board_type": "p150b", + "board_id": "0000041231915018" + } + }, + { + "board_info": { + "bus_id": "0000:21:00.0", + "board_type": "p150b", + "board_id": "0000041231915002" + } + }, + { + "board_info": { + "bus_id": "0000:41:00.0", + "board_type": "p150b", + "board_id": "0000041231915009" + } + }, + { + "board_info": { + "bus_id": "0000:61:00.0", + "board_type": "p150b", + "board_id": "000004123191500f" + } + }, + { + "board_info": { + "bus_id": "0000:81:00.0", + "board_type": "p150b", + "board_id": "0000041231914064" + } + }, + { + "board_info": { + "bus_id": "0000:a1:00.0", + "board_type": "p150b", + "board_id": "0000041231915006" + } + }, + { + "board_info": { + "bus_id": "0000:c1:00.0", + "board_type": "p150b", + "board_id": "0000041231914087" + } + }, + { + "board_info": { + "bus_id": "0000:e1:00.0", + "board_type": "p150b", + "board_id": "000004123191402f" + } + } + ] +} diff --git a/runner/internal/shim/host/testdata/tenstorrent/blackhole_boards.json b/runner/internal/shim/host/testdata/tenstorrent/blackhole_boards.json new file mode 100644 index 0000000000..8dfdd62e64 --- /dev/null +++ b/runner/internal/shim/host/testdata/tenstorrent/blackhole_boards.json @@ -0,0 +1,39 @@ +{ + "device_info": [ + { + "board_info": { + "bus_id": "0000:61:00.0", + "board_type": "p100a", + "board_id": "000004323191b040" + } + }, + { + "board_info": { + "bus_id": "0000:01:00.0", + "board_type": "p150a", + "board_id": "0000040100000000" + } + }, + { + "board_info": { + "bus_id": "0000:02:00.0", + "board_type": "p150b", + "board_id": "000004123191110e" + } + }, + { + "board_info": { + "bus_id": "0000:03:00.0", + "board_type": "p300a", + "board_id": "000004513190f004" + } + }, + { + "board_info": { + "bus_id": "0000:04:00.0", + "board_type": "p300a", + "board_id": "000004513190f004" + } + } + ] +} diff --git a/runner/internal/shim/host/testdata/tenstorrent/blackhole_galaxy.json b/runner/internal/shim/host/testdata/tenstorrent/blackhole_galaxy.json new file mode 100644 index 0000000000..2a85a80786 --- /dev/null +++ b/runner/internal/shim/host/testdata/tenstorrent/blackhole_galaxy.json @@ -0,0 +1,3348 @@ +{ + "time": "2026-05-24T07:51:28.084906", + "host_info": { + "OS": "Linux", + "Distro": "Ubuntu 22.04.5 LTS", + "Kernel": "6.8.0-111-generic", + "Hostname": "de55091aab2e", + "Platform": "x86_64", + "Python": "3.10.12", + "Memory": "566.12 GB", + "Driver": "TT-KMD 2.8.0" + }, + "host_sw_vers": { + "tt_smi": "5.2.0", + "pyluwen": "0.8.5", + "tt_umd": "0.9.5" + }, + "device_info": [ + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e2", + "TDP": "0x27", + "TDC": "0x35", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2d682a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x2fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36363834", + "GDDR_2_3_TEMP": "0x36363838", + "GDDR_4_5_TEMP": "0x36323432", + "GDDR_6_7_TEMP": "0x36323634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x1", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x30da68d3", + "ASIC_ID_LOW": "0x319cdf5f", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:01:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 53.0", + "power": " 39.0", + "aiclk": " 800", + "asic_temperature": "45.4", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x20", + "TDC": "0x2c", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28ba82", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfa", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fdf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36363834", + "GDDR_2_3_TEMP": "0x38363836", + "GDDR_4_5_TEMP": "0x38383636", + "GDDR_6_7_TEMP": "0x38383838", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x2", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xafa174e", + "ASIC_ID_LOW": "0x22393ea0", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:02:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 44.0", + "power": " 32.0", + "aiclk": " 800", + "asic_temperature": "40.7", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x11", + "TDC": "0x18", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28db52", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df6", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3f7f", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36363836", + "GDDR_2_3_TEMP": "0x38363836", + "GDDR_4_5_TEMP": "0x3a363834", + "GDDR_6_7_TEMP": "0x38343836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x3", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xc722c9ef", + "ASIC_ID_LOW": "0x5253bcaa", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:03:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 24.0", + "power": " 17.0", + "aiclk": " 800", + "asic_temperature": "40.9", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x24", + "TDC": "0x31", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2dca9a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfb", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3bff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36323434", + "GDDR_2_3_TEMP": "0x36323834", + "GDDR_4_5_TEMP": "0x3a363636", + "GDDR_6_7_TEMP": "0x3a363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x4", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x5f105885", + "ASIC_ID_LOW": "0x8d9e9a28", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:04:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 49.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "45.8", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e1", + "TDP": "0x26", + "TDC": "0x34", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x295e92", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df7", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ff7", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x302e302e", + "GDDR_2_3_TEMP": "0x32303230", + "GDDR_4_5_TEMP": "0x34323430", + "GDDR_6_7_TEMP": "0x34343632", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x5", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x38a62485", + "ASIC_ID_LOW": "0x910f7598", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:05:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 52.0", + "power": " 38.0", + "aiclk": " 800", + "asic_temperature": "41.4", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x25", + "TDC": "0x33", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a2372", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ffd", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323430", + "GDDR_2_3_TEMP": "0x36363634", + "GDDR_4_5_TEMP": "0x34343432", + "GDDR_6_7_TEMP": "0x36343434", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x6", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x3b0df290", + "ASIC_ID_LOW": "0x34ed9b6c", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:06:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "8" + }, + "telemetry": { + "voltage": "0.74", + "current": " 51.0", + "power": " 37.0", + "aiclk": " 800", + "asic_temperature": "42.1", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x11", + "TDC": "0x18", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x29a032", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3f7f", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36323230", + "GDDR_2_3_TEMP": "0x36343634", + "GDDR_4_5_TEMP": "0x3634342e", + "GDDR_6_7_TEMP": "0x38343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x7", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x470e750f", + "ASIC_ID_LOW": "0xe3463989", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:07:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 24.0", + "power": " 17.0", + "aiclk": " 800", + "asic_temperature": "41.6", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x10", + "TDC": "0x16", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28894a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x1fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323230", + "GDDR_2_3_TEMP": "0x36343834", + "GDDR_4_5_TEMP": "0x3230322e", + "GDDR_6_7_TEMP": "0x32323632", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x8", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x2ee8b99a", + "ASIC_ID_LOW": "0x700a32f4", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:08:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 22.0", + "power": " 16.0", + "aiclk": " 800", + "asic_temperature": "40.5", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e2", + "TDP": "0x2a", + "TDC": "0x39", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2e4dda", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ffe", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x3a363834", + "GDDR_2_3_TEMP": "0x36363836", + "GDDR_4_5_TEMP": "0x36343632", + "GDDR_6_7_TEMP": "0x34343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x1", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xbc2704c4", + "ASIC_ID_LOW": "0x6388dc72", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:41:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 57.0", + "power": " 42.0", + "aiclk": " 800", + "asic_temperature": "46.3", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x24", + "TDC": "0x31", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28ebba", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ffd", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x38363836", + "GDDR_2_3_TEMP": "0x3a383836", + "GDDR_4_5_TEMP": "0x38363836", + "GDDR_6_7_TEMP": "0x38363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x2", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x3bbbc600", + "ASIC_ID_LOW": "0x3117fbe1", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:42:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 49.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "40.9", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x23", + "TDC": "0x30", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2b198a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df6", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fef", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x3a363836", + "GDDR_2_3_TEMP": "0x38363836", + "GDDR_4_5_TEMP": "0x38343836", + "GDDR_6_7_TEMP": "0x3a363834", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x3", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x1c2b9a0", + "ASIC_ID_LOW": "0x58a60566", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:43:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 48.0", + "power": " 35.0", + "aiclk": " 800", + "asic_temperature": "43.1", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e1", + "TDP": "0x27", + "TDC": "0x35", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2cb3b2", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfb", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x2fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323432", + "GDDR_2_3_TEMP": "0x36343632", + "GDDR_4_5_TEMP": "0x38343834", + "GDDR_6_7_TEMP": "0x36363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x4", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x701faaab", + "ASIC_ID_LOW": "0x606f3f29", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:44:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 53.0", + "power": " 39.0", + "aiclk": " 800", + "asic_temperature": "44.7", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e2", + "TDP": "0x24", + "TDC": "0x31", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a757a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfb", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fef", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x302e2e2a", + "GDDR_2_3_TEMP": "0x3430322e", + "GDDR_4_5_TEMP": "0x34303230", + "GDDR_6_7_TEMP": "0x36343432", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x5", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xae417056", + "ASIC_ID_LOW": "0x353a8f9c", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:45:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 49.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "42.5", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x10", + "TDC": "0x16", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28ebba", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3f7f", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323430", + "GDDR_2_3_TEMP": "0x36343634", + "GDDR_4_5_TEMP": "0x36343232", + "GDDR_6_7_TEMP": "0x38343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x6", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x3e58f3fa", + "ASIC_ID_LOW": "0xc0c22e55", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:46:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "8" + }, + "telemetry": { + "voltage": "0.74", + "current": " 22.0", + "power": " 16.0", + "aiclk": " 800", + "asic_temperature": "40.9", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x12", + "TDC": "0x19", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x295e92", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfa", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fdf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34303030", + "GDDR_2_3_TEMP": "0x38343632", + "GDDR_4_5_TEMP": "0x34323432", + "GDDR_6_7_TEMP": "0x36343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x7", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x9d5d96f", + "ASIC_ID_LOW": "0xeed74c01", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:47:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 25.0", + "power": " 18.0", + "aiclk": " 800", + "asic_temperature": "41.4", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e4", + "TDP": "0x22", + "TDC": "0x2f", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2b198a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3bff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323430", + "GDDR_2_3_TEMP": "0x38323634", + "GDDR_4_5_TEMP": "0x3030322e", + "GDDR_6_7_TEMP": "0x3430322e", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x8", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xf3bf945a", + "ASIC_ID_LOW": "0xdcd2b7ef", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:48:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 47.0", + "power": " 34.0", + "aiclk": " 800", + "asic_temperature": "43.1", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e9", + "TDP": "0x10", + "TDC": "0x16", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x28894a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df7", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fdf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36383836", + "GDDR_2_3_TEMP": "0x3a383838", + "GDDR_4_5_TEMP": "0x36343636", + "GDDR_6_7_TEMP": "0x34343636", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x1", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x5cfe5073", + "ASIC_ID_LOW": "0x42cfe4e1", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:81:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 22.0", + "power": " 16.0", + "aiclk": " 800", + "asic_temperature": "40.5", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x24", + "TDC": "0x31", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x292d5a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fdf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x38363636", + "GDDR_2_3_TEMP": "0x3a383838", + "GDDR_4_5_TEMP": "0x38383a38", + "GDDR_6_7_TEMP": "0x36383836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x2", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xff51a2a2", + "ASIC_ID_LOW": "0x23203905", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:82:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 49.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "41.2", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e7", + "TDP": "0x11", + "TDC": "0x18", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a4442", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df7", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3f7f", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36363836", + "GDDR_2_3_TEMP": "0x38383836", + "GDDR_4_5_TEMP": "0x38363836", + "GDDR_6_7_TEMP": "0x38383836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x3", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x8d27d7ec", + "ASIC_ID_LOW": "0xe45d68f6", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:83:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 24.0", + "power": " 17.0", + "aiclk": " 800", + "asic_temperature": "42.3", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e2", + "TDP": "0x27", + "TDC": "0x35", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2c0fa2", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df6", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fbf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323432", + "GDDR_2_3_TEMP": "0x36323632", + "GDDR_4_5_TEMP": "0x36343834", + "GDDR_6_7_TEMP": "0x38363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x4", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xef2552b4", + "ASIC_ID_LOW": "0x90ff9acc", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:84:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 53.0", + "power": " 39.0", + "aiclk": " 800", + "asic_temperature": "44.1", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x14", + "TDC": "0x1b", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a757a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfa", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ff7", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x302e302e", + "GDDR_2_3_TEMP": "0x34303230", + "GDDR_4_5_TEMP": "0x34323432", + "GDDR_6_7_TEMP": "0x38343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x5", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x443a0b54", + "ASIC_ID_LOW": "0x29c8374c", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:85:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 27.0", + "power": " 20.0", + "aiclk": " 800", + "asic_temperature": "42.5", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e1", + "TDP": "0x26", + "TDC": "0x34", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2beed2", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfa", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x2fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36343432", + "GDDR_2_3_TEMP": "0x36363636", + "GDDR_4_5_TEMP": "0x34323230", + "GDDR_6_7_TEMP": "0x36343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x6", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xe2505c71", + "ASIC_ID_LOW": "0xeb8d9097", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:86:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "8" + }, + "telemetry": { + "voltage": "0.74", + "current": " 52.0", + "power": " 38.0", + "aiclk": " 800", + "asic_temperature": "43.9", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x24", + "TDC": "0x32", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a757a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfe", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ffe", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323030", + "GDDR_2_3_TEMP": "0x36343632", + "GDDR_4_5_TEMP": "0x34323630", + "GDDR_6_7_TEMP": "0x34343632", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x7", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x4300de24", + "ASIC_ID_LOW": "0xb9034c67", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:87:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 50.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "42.5", + "fan_speed": " 0", + "heartbeat": "67498" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e9", + "TDP": "0xf", + "TDC": "0x15", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x27f5a2", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df7", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3f7f", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323232", + "GDDR_2_3_TEMP": "0x36343634", + "GDDR_4_5_TEMP": "0x322e322e", + "GDDR_6_7_TEMP": "0x34323430", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x8", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xc4ac0ba6", + "ASIC_ID_LOW": "0x46891f5b", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:88:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 21.0", + "power": " 15.0", + "aiclk": " 800", + "asic_temperature": "40.0", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x20", + "TDC": "0x2c", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2ad7ea", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fbf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x38343634", + "GDDR_2_3_TEMP": "0x38343634", + "GDDR_4_5_TEMP": "0x34323434", + "GDDR_6_7_TEMP": "0x34323632", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x1", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x942a743e", + "ASIC_ID_LOW": "0x637088a4", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c1:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 44.0", + "power": " 32.0", + "aiclk": " 800", + "asic_temperature": "42.8", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x23", + "TDC": "0x30", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2cd482", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df6", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ffe", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x38383836", + "GDDR_2_3_TEMP": "0x36363636", + "GDDR_4_5_TEMP": "0x38363634", + "GDDR_6_7_TEMP": "0x38363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x2", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xc5898618", + "ASIC_ID_LOW": "0x78af4f3e", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c2:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 48.0", + "power": " 35.0", + "aiclk": " 800", + "asic_temperature": "44.8", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x11", + "TDC": "0x17", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2a4442", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x1fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x36363836", + "GDDR_2_3_TEMP": "0x38363832", + "GDDR_4_5_TEMP": "0x3a363834", + "GDDR_6_7_TEMP": "0x38363836", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x3a", + "ASIC_LOCATION": "0x3", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xd8f9ffdc", + "ASIC_ID_LOW": "0x3d1b9f23", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c3:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 23.0", + "power": " 17.0", + "aiclk": " 800", + "asic_temperature": "42.3", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e4", + "TDP": "0x22", + "TDC": "0x2e", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2af8ba", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x2fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34343432", + "GDDR_2_3_TEMP": "0x36343632", + "GDDR_4_5_TEMP": "0x38343638", + "GDDR_6_7_TEMP": "0x36363834", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x38", + "ASIC_LOCATION": "0x4", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xea25c6db", + "ASIC_ID_LOW": "0xab35c1ca", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c4:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 46.0", + "power": " 34.0", + "aiclk": " 800", + "asic_temperature": "43.0", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e3", + "TDP": "0x25", + "TDC": "0x33", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2deb6a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62dfc", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fef", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x302e302c", + "GDDR_2_3_TEMP": "0x34323230", + "GDDR_4_5_TEMP": "0x34323432", + "GDDR_6_7_TEMP": "0x36343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x5", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x9019d0f8", + "ASIC_ID_LOW": "0x5c61ff49", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c5:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 51.0", + "power": " 37.0", + "aiclk": " 800", + "asic_temperature": "45.9", + "fan_speed": " 0", + "heartbeat": "67498" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e8", + "TDP": "0x10", + "TDC": "0x16", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x2847aa", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3ff7", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x32323232", + "GDDR_2_3_TEMP": "0x36343434", + "GDDR_4_5_TEMP": "0x34323430", + "GDDR_6_7_TEMP": "0x36343634", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x6", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x89f9d5c7", + "ASIC_ID_LOW": "0x22a507e9", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c6:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "8" + }, + "telemetry": { + "voltage": "0.74", + "current": " 22.0", + "power": " 16.0", + "aiclk": " 800", + "asic_temperature": "40.3", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e9", + "TDP": "0x11", + "TDC": "0x17", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x293dc2", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x0", + "L2CPUCLK1": "0x0", + "L2CPUCLK2": "0x0", + "L2CPUCLK3": "0x0", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df9", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x1fff", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34303230", + "GDDR_2_3_TEMP": "0x36343432", + "GDDR_4_5_TEMP": "0x34303430", + "GDDR_6_7_TEMP": "0x36323432", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x7", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0xa4d189d2", + "ASIC_ID_LOW": "0x740a2f25", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c7:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 23.0", + "power": " 17.0", + "aiclk": " 800", + "asic_temperature": "41.2", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + }, + { + "smbus_telem": { + "BOARD_ID_HIGH": "0x471", + "BOARD_ID_LOW": "0x31831011", + "ASIC_ID": null, + "HARVESTING_STATE": "0x0", + "UPDATE_TELEM_SPEED": "0x64", + "VCORE": "0x2e9", + "TDP": "0x24", + "TDC": "0x31", + "VDD_LIMITS": "0x38402bc", + "THM_LIMIT_SHUTDOWN": "0x6e", + "ASIC_TEMPERATURE": "0x27e53a", + "VREG_TEMPERATURE": "0x0", + "BOARD_TEMPERATURE": "0x0", + "AICLK": "0x320", + "AXICLK": "0x3c0", + "ARCCLK": "0x320", + "L2CPUCLK0": "0x320", + "L2CPUCLK1": "0x320", + "L2CPUCLK2": "0x320", + "L2CPUCLK3": "0x320", + "ETH_LIVE_STATUS": "0x0", + "DDR_STATUS": "0x5555", + "DDR_SPEED": "0x36b0", + "ETH_FW_VERSION": "0x0", + "GDDR_FW_VERSION": "0x2000b", + "DM_APP_FW_VERSION": "0x0", + "DM_BL_FW_VERSION": "0x0", + "FLASH_BUNDLE_VERSION": "0x13050000", + "CM_FW_VERSION": "0x1b0000", + "L2CPU_FW_VERSION": "0x0", + "FAN_SPEED": "0x0", + "TIMER_HEARTBEAT": "0x62df8", + "TELEMETRY_ENUM_COUNT": "0x45", + "ENABLED_TENSIX_COL": "0x3fdf", + "ENABLED_ETH": "0x3edf", + "ENABLED_GDDR": "0xff", + "ENABLED_L2CPU": "0xf", + "PCIE_USAGE": "0x4", + "NOC_TRANSLATION": "0x1", + "FAN_RPM": "0x0", + "GDDR_0_1_TEMP": "0x34323230", + "GDDR_2_3_TEMP": "0x36323432", + "GDDR_4_5_TEMP": "0x322e302e", + "GDDR_6_7_TEMP": "0x32303030", + "GDDR_0_1_CORR_ERRS": "0xff00ff00", + "GDDR_2_3_CORR_ERRS": "0xff00ff00", + "GDDR_4_5_CORR_ERRS": "0xff00ff00", + "GDDR_6_7_CORR_ERRS": "0xff00ff00", + "GDDR_UNCORR_ERRS": "0xaaaa", + "MAX_GDDR_TEMP": "0x36", + "ASIC_LOCATION": "0x8", + "BOARD_POWER_LIMIT": "0x0", + "TDC_LIMIT_MAX": "0x1f4", + "THM_LIMIT_THROTTLE": "0x5a", + "TT_FLASH_VERSION": null, + "THERM_TRIP_COUNT": "0x0", + "ASIC_ID_HIGH": "0x1092e6d0", + "ASIC_ID_LOW": "0xdbd47817", + "AICLK_LIMIT_MAX": "0x546", + "TDP_LIMIT_MAX": "0x82", + "NUMBER_OF_TAGS": "0x320" + }, + "board_info": { + "bus_id": "0000:c8:00.0", + "board_type": "tt-galaxy-bh", + "board_id": "0000047131831011", + "coords": "N/A", + "dram_status": true, + "dram_speed": "14G", + "pcie_speed": 4, + "pcie_width": "1" + }, + "telemetry": { + "voltage": "0.74", + "current": " 49.0", + "power": " 36.0", + "aiclk": " 800", + "asic_temperature": "39.9", + "fan_speed": " 0", + "heartbeat": "67497" + }, + "firmwares": { + "fw_bundle_version": "19.5.0.0", + "tt_flash_version": "N/A", + "cm_fw": "0.27.0.0", + "cm_fw_date": "2020-00-27", + "eth_fw": "0.0.0", + "dm_bl_fw": "0.0.0.0", + "dm_app_fw": "0.0.0.0", + "gddr_fw": "2.11" + }, + "limits": { + "vdd_min": "0.70", + "vdd_max": "0.90", + "tdp_limit": "130", + "tdc_limit": "500", + "asic_fmax": "1350", + "therm_trip_l1_limit": "90", + "thm_limit": "110", + "bus_peak_limit": 0 + } + } + ] +} diff --git a/src/tests/_internal/cli/services/configurators/test_run.py b/src/tests/_internal/cli/services/configurators/test_run.py index 6238bcb025..97be49405d 100644 --- a/src/tests/_internal/cli/services/configurators/test_run.py +++ b/src/tests/_internal/cli/services/configurators/test_run.py @@ -4,7 +4,7 @@ from unittest.mock import Mock import pytest -from gpuhunt import AcceleratorVendor +from gpuhunt import KNOWN_TENSTORRENT_ACCELERATORS, AcceleratorVendor from dstack._internal.cli.services.configurators import get_run_configurator_class from dstack._internal.cli.services.configurators.run import ( @@ -24,6 +24,10 @@ from dstack._internal.core.models.profiles import Profile from dstack._internal.server.testing.common import get_run_spec +_TENSTORRENT_ACCELERATOR_NAMES = tuple( + sorted({gpu.name for gpu in KNOWN_TENSTORRENT_ACCELERATORS}) +) + class TestApplyArgs: def apply_args( @@ -263,12 +267,21 @@ def test_two_vendors_including_amd_inferred_no_image(self, gpu_spec): ): self.validate(conf) - @pytest.mark.parametrize("gpu_spec", ["n150", "n300"]) + @pytest.mark.parametrize("gpu_spec", _TENSTORRENT_ACCELERATOR_NAMES) def test_tenstorrent_docker_true_no_image(self, gpu_spec): conf = self.prepare_conf(gpu_spec=gpu_spec, docker=True) self.validate(conf) assert conf.resources.gpu.vendor == AcceleratorVendor.TENSTORRENT + @pytest.mark.parametrize("gpu_spec", _TENSTORRENT_ACCELERATOR_NAMES) + def test_tenstorrent_vendor_inferred_no_image(self, gpu_spec): + conf = self.prepare_conf(gpu_spec=gpu_spec) + with pytest.raises( + ConfigurationError, + match=r"`image` is required if `resources.gpu.vendor` is `tenstorrent`", + ): + self.validate(conf) + class TestValidateCPUArchAndImage: def prepare_conf(