From 8fb3792c60621df320f17b05246ef889ed91a0c4 Mon Sep 17 00:00:00 2001 From: William Dawson Date: Tue, 21 Jul 2026 22:22:15 +0900 Subject: [PATCH 1/2] Add BenchKit skill Guide for using and contributing to BenchKit (RIKEN-RCCS): running or testing an existing benchmark locally or via a real batch job, adding a new benchmark application (build.sh/run.sh/list.csv), and registering a new target system/site (runner setup + system.csv/ queue.csv/system_info.csv). Points to, but does not duplicate, BenchKit's own maintenance-level docs (result portal deployment, estimation-package internals, CI infrastructure). Co-Authored-By: Claude Sonnet 5 --- .claude-plugin/marketplace.json | 11 ++ README.md | 1 + benchkit/.codex-plugin/plugin.json | 22 +++ benchkit/skills/benchkit/SKILL.md | 303 +++++++++++++++++++++++++++++ 4 files changed, 337 insertions(+) create mode 100644 benchkit/.codex-plugin/plugin.json create mode 100644 benchkit/skills/benchkit/SKILL.md diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 7f706b6..522199a 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -77,5 +77,16 @@ "description": "Slurm job management, filesystem operations, and documentation search for the RIKEN AI4S supercomputer (NVIDIA GB200, aarch64)", "source": "./rikyu/plugins/rikyu" } + , + { + "id": "benchkit", + "name": "BenchKit", + "description": "Use and contribute to BenchKit, a shell-first framework for building and running HPC benchmarks across multiple systems", + "source": { + "source": "git-subdir", + "url": "git@github.com:RIKEN-RCCS/HPC-Agentic-SDK.git", + "path": "benchkit" + } + } ] } diff --git a/README.md b/README.md index f317b6c..3eef506 100644 --- a/README.md +++ b/README.md @@ -34,6 +34,7 @@ Then open `/plugins` in Codex and install the plugins you need. | [Tadashi](./tadashi/) | Check whether loop transformations are mathematically safe using polyhedral analysis | | [make-vibe](./make-vibe/) | Expose Makefile targets as MCP tools for building and running code on local or remote HPC systems | | [OpenACC](./openacc/) | Automatically insert OpenACC directives into Fortran code using a verified template library for reliable GPU parallelization | +| [BenchKit](./benchkit/) | Use and contribute to BenchKit, a shell-first framework for building and running HPC benchmarks across multiple systems | ### RIKEN R-CCS facility agents diff --git a/benchkit/.codex-plugin/plugin.json b/benchkit/.codex-plugin/plugin.json new file mode 100644 index 0000000..837a7a7 --- /dev/null +++ b/benchkit/.codex-plugin/plugin.json @@ -0,0 +1,22 @@ +{ + "name": "benchkit", + "version": "0.1.0", + "description": "Use and contribute to BenchKit, a shell-first framework for building and running HPC benchmarks across multiple systems.", + "author": { + "name": "RIKEN-RCCS" + }, + "repository": "https://github.com/RIKEN-RCCS/HPC-Agentic-SDK", + "keywords": ["hpc", "benchmarking", "benchkit", "ci", "performance"], + "skills": "./skills/", + "interface": { + "displayName": "BenchKit", + "shortDescription": "Build, run, and contribute HPC benchmarks with BenchKit.", + "longDescription": "Guide for using and contributing to BenchKit: running an existing benchmark locally or via a real batch job, adding a new benchmark application (build.sh/run.sh/list.csv), and registering a new target system/site.", + "developerName": "RIKEN-RCCS", + "category": "Developer Tools", + "capabilities": ["Write"], + "defaultPrompt": [ + "Help me add a new BenchKit application." + ] + } +} diff --git a/benchkit/skills/benchkit/SKILL.md b/benchkit/skills/benchkit/SKILL.md new file mode 100644 index 0000000..286e770 --- /dev/null +++ b/benchkit/skills/benchkit/SKILL.md @@ -0,0 +1,303 @@ +--- +name: benchkit +description: Guide for using and contributing to BenchKit (RIKEN-RCCS), a shell-first benchmarking framework that builds/runs HPC applications across multiple systems, collects results, and reports them to a portal. Use when adding a benchmark application to BenchKit, registering a new target system/site, running or testing a BenchKit benchmark locally, or understanding a `programs//{build.sh,run.sh,list.csv}` layout. +user-invocable: true +--- + +# BenchKit — building and running HPC benchmarks across systems + +BenchKit (from RIKEN-RCCS) is a **shell-first** benchmarking framework: for +any given application, a `build.sh` and `run.sh` any HPC user could read and +run by hand define how it's built and executed on each target system, and a +`list.csv` defines which (system, node-count, rank, thread) combinations to +run. There is no heavy framework abstraction layer between you and the actual +build/run commands — that's a deliberate design choice, not an accident. + +**Repository:** https://github.com/RIKEN-RCCS/benchkit + +Two things you'll do with it: **run/test a benchmark that already exists**, +or **contribute a new application or a new target system**. This skill +covers both. It does not cover maintaining BenchKit's own internals +(the result portal service, the estimation-package algorithms, CI +infrastructure) — see `docs/guides/developer-reference.md` in the repo for +that. + +## Repository layout at a glance + +``` +benchkit/ +├── programs// # one directory per benchmark application +│ ├── build.sh # required — builds the app for a given system +│ ├── run.sh # required — runs it, emits results +│ ├── list.csv # required — which (system,nodes,...) to run +│ ├── estimate.sh # optional — plugs into the estimation layer +│ └── README.md # optional — app-specific notes +├── config/ +│ ├── system.csv # registered systems: mode, runner tags, queue +│ ├── queue.csv # scheduler submit-command templates +│ └── system_info.csv # CPU/GPU/memory info, for portal display +├── scripts/ +│ ├── bk_functions.sh # sourced by every build.sh/run.sh +│ ├── job_functions.sh # sourced helper: CSV lookups (system mode, queue_group, ...) +│ ├── test_submit.sh # submit one list.csv row as a real batch job +│ ├── test_submit_build.sh # build-only variant +│ ├── matrix_generate.sh # list.csv × system.csv -> generated CI pipeline +│ ├── result.sh # results/result -> normalized Result JSON +│ ├── result_server/ # send_results.sh, send_estimate.sh, api.sh +│ ├── site/ # setup_runner.sh, preflight_runner.sh +│ └── estimation/ # estimation pipeline driver/packages +├── docs/ +│ ├── guides/add-app.md # full walkthrough, reference example: qws +│ ├── guides/add-site.md # full walkthrough for new systems +│ ├── guides/add-estimation.md # routes to app-side vs package-side guides +│ └── cx/*.md # CX Framework specs (terminology, architecture) +└── result_server/ # the result portal (Flask app; separate service) +``` + +## Running/testing an existing benchmark + +There is no single `bin/benchkit` driver — you invoke the scripts for a +specific app/system pair directly. + +**Local, no scheduler** (fastest iteration loop, useful for checking the +build/run logic itself works before touching a real job): + +```sh +bash programs//build.sh +bash programs//run.sh +``` + +`run.sh` always takes exactly those four positional args, in that order, +regardless of application. Check `artifacts/` after the build step and +`results/result` after the run step. + +**Real batch job on the target scheduler** (the actual "does this work in +CI" test): + +```sh +bash scripts/test_submit.sh +``` + +`` is the row in `programs//list.csv` to submit (1-based, +counting the header). This resolves `system`/`mode`/`queue_group` from +`config/system.csv`, prints them so you can sanity-check the resolution, and +submits through the right scheduler command for that system (PJM on +Fugaku-family systems, PBS on Miyabi-family, SLURM on R-CCS Cloud/RIKYU/etc. +— the mapping lives in `config/queue.csv`). `scripts/test_submit_build.sh` +is the build-only equivalent, and `scripts/test_estimate_submit.sh` tests +the estimation pipeline the same way. + +**Generating the CI pipeline** (rarely needed by hand — CI runs this +itself, but useful to preview what a `list.csv` change will actually +schedule): + +```sh +bash scripts/matrix_generate.sh [code=] [system=] +``` + +Reads `list.csv` × `system.csv`/`queue.csv`/`system_info.csv` and writes +`.gitlab-ci.generated.yml`. + +## Contributing a new application + +Reference example in the repo: `programs/qws/` (most fully built-out — +has build recipes for essentially every registered system). Full guide: +`docs/guides/add-app.md`. + +1. **Branch and scaffold:** + ```sh + git checkout -b add- + mkdir programs/ + ``` + (or `cp -pr programs/qws/ programs/` to start from a real example). + +2. **`list.csv`** — exact header, no more no less: + ```csv + system,enable,nodes,numproc_node,nthreads,elapse + Fugaku,yes,1,4,12,0:10:00 + ``` + `enable` is `yes`/`no` (to disable a row, set `enable=no` — don't comment + it out with `#`). Note `mode` and `queue_group` are **not** columns + here — those come from `config/system.csv`, one shared source of truth + across every app. Before adding a row for a system, confirm it already + has entries in both `config/system.csv` and `config/system_info.csv` + (if not, that's a site-registration task — see below, not something you + do from the app side). + +3. **`build.sh`** — receives `system` as `$1`. Pattern: + ```sh + #!/bin/bash + set -euo pipefail + system="$1" + source scripts/bk_functions.sh + + bk_fetch_source "" "" "" + cd "" + + case "$system" in + Fugaku) ... module loads, compiler flags, build commands ... ;; + RC_GENOA) ... ;; + *) echo "Unknown system: $system" >&2; exit 1 ;; + esac + + cp "${PWD}/../artifacts/" + ``` + Always use `bk_fetch_source` rather than a raw `git clone` — it writes + `results/source_info.env` with the repo/branch/commit hash, which is the + provenance record the portal surfaces at `/results/usage`. Copy only the + built executable (and anything else `run.sh` genuinely needs) into + `artifacts/`, not the whole source/build tree — that's what CI archives. + +4. **`run.sh`** — receives `system nodes numproc_node nthreads` positionally. + Pattern: + ```sh + #!/bin/bash + set -euo pipefail + system="$1"; nodes="$2"; numproc_node="$3"; nthreads="$4" + source scripts/bk_functions.sh + export OMP_NUM_THREADS="$nthreads" + mkdir -p results && : > results/result + + case "$system" in + Fugaku) mpiexec -n $((nodes*numproc_node)) ./artifacts/ ;; + *) echo "Unknown system: $system" >&2; exit 1 ;; + esac + + bk_emit_result --fom --fom-unit --fom-version