Fix benchmark fairness, profiling attribution, and reportability semantics

README.md

@@ -84,8 +84,7 @@ On the current homogeneous `RTX A6000` pool:
 
 ```bash
 export KTUNE_SCRATCH=/scratch/scratch-space/expires-xxxx/$USER/kerneltuner
-scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312"
-source "$KTUNE_SCRATCH/venv-py312/bin/activate"
+source scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312"
 
 ktune validate-kernel --kernel configs/kernels/gemm.yaml
 ktune run-experiment --experiment configs/experiments/gemm_smoke.yaml
@@ -136,6 +135,54 @@ scripts/slurm/submit_kerneltuner.sh \
   --mem 24GB
 ```
 
+## Slurm Submission Helpers
+
+For cluster execution, the repo includes reusable Slurm scripts:
+
+- `scripts/slurm/run_kerneltuner_array.sbatch`: array worker that maps one array task to one experiment YAML.
+- `scripts/slurm/submit_kerneltuner.sh`: submit wrapper that computes array size from a list file.
+
+Example list file:
+
+- `configs/experiments/slurm_experiment_list.example.txt`
+
+Basic usage:
+
+```bash
+chmod +x scripts/slurm/submit_kerneltuner.sh
+cp configs/experiments/slurm_experiment_list.example.txt configs/experiments/slurm_experiment_list.txt
+scripts/slurm/submit_kerneltuner.sh \
+  --list configs/experiments/slurm_experiment_list.txt \
+  --partition gpunodes \
+  --gpu-type rtx_a2000 \
+  --gpus 1 \
+  --time 0-04:00 \
+  --cpus 4 \
+  --mem 24GB \
+  --log-dir /scratch/scratch-space/expires-xxxx/$USER/kerneltuner_logs \
+  --scratch-root /scratch/scratch-space/expires-xxxx/$USER/kerneltuner \
+  --artifact-root /scratch/scratch-space/expires-xxxx/$USER/kerneltuner_artifacts \
+  --mail-user [email protected] \
+  --mail-type BEGIN,END,FAIL \
+  --alert-email [email protected] \
+  --alert-on-start
+```
+
+Useful environment overrides:
+
+- `RUN_COMMAND_TEMPLATE` (default: `ktune run-experiment --experiment "{experiment}"`)
+- `INSTALL_PACKAGES` (`0` to skip pip install in jobs)
+- `EXTRA_PIP_PACKAGES` (space-separated extra pip packages)
+- `SKIP_IF_ARTIFACTS_EXIST` (`0` to force reruns)
+- `DRY_RUN=1` (worker-level dry run)
+- `ALERT_EMAIL`, `ALERT_ON_START`, `ALERT_ON_END`, `ALERT_ON_FAIL` for worker-level alerts
+
+Notes:
+
+- `--artifact-root` overrides `artifact_root` in experiment YAML at submission time.
+- If no scratch path exists on the node, jobs fall back to `<workspace>/.scratch/$USER`.
+- Slurm native email (`--mail-user/--mail-type`) and worker-level alerts can be used together.
+
 ## What This Repo Is Not
 
 - It is not a Triton compiler redesign effort.

configs/counters/default_calibration.yaml

@@ -4,7 +4,7 @@ tool: ncu
 replay_mode: kernel
 target_processes: all
 counters:
-  - smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg
+  - smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct
   - sm__warps_active.avg.pct_of_peak_sustained_active
   - smsp__inst_executed.sum
 minimum_availability: 0.9

configs/studies/validation_phase.yaml

@@ -1,7 +1,7 @@
 study_id: validation_phase
 hypotheses:
   - hypothesis_id: H1
-    description: Cheap compile signals prune obvious losers, but do not reliably rank near-tied GEMM configs.
+    description: Cheap compile signals prune obvious losers, but produce noticeably unstable ranking quality on representative GEMM workloads.
     clauses:
       - left:
           source: strategy_rows

docs/gpu_job_guide.md

@@ -150,15 +150,13 @@ else
   export KTUNE_SCRATCH="/tmp/$USER/kerneltuner"
 fi
 
-scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312"
-source "$KTUNE_SCRATCH/venv-py312/bin/activate"
+source scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312"
 ```
 
 If you want to choose the virtualenv path explicitly:
 
 ```bash
-scripts/bootstrap_env.sh /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312
-source /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312/bin/activate
+source scripts/bootstrap_env.sh /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312
 ```
 
 The bootstrap script:

docs/research/logs/2026-03-28_profiling_reliability_validation.md

@@ -0,0 +1,90 @@
+# Profiling Reliability Validation On gpunode2
+
+Purpose: record the profiling reliability fix, the counter-set audit, and the first reportable reruns on the pinned validation host.
+Status: Log
+Update Rule: append-only; do not rewrite past observations except to fix factual errors.
+Feeds Paper Sections: operational methodology, profiling validity, limitations.
+Depends On: [../04_signal_and_profiling_plan.md](../04_signal_and_profiling_plan.md), [../06_hypotheses_and_ablation_plan.md](../06_hypotheses_and_ablation_plan.md), [../08_evidence_registry.md](../08_evidence_registry.md), [../../04_experiment_protocol.md](../../04_experiment_protocol.md)
+
+## What Changed
+
+- preserved the CUDA toolchain exports by making `source scripts/bootstrap_env.sh <venv>` the canonical launch contract for both manual GPU-shell runs and the Slurm worker path
+- updated profiler invocation and provenance capture to resolve tool paths through `CUDA_HOME/bin` before falling back to `PATH`
+- tightened profiling diagnostics so artifacts distinguish missing profiler binaries, missing queried metrics, invocation failures, kernel attribution failures, and runs that return null counter values
+- replaced the fragile `default_calibration` long-scoreboard counter with a queryable and populated variant on `gpunode2`
+
+## Original Failure
+
+- baseline smoke run: `artifacts/gemm_smoke/run_20260328T153717Z_fb053320`
+- observed failure: `counter_compatibility.notes` reported `ncu executable not found`
+- root cause: `bootstrap_env.sh` was executed in a subprocess, so the exported CUDA tool paths were lost before `ktune` launched profiling
+
+## Intermediate Validation
+
+- post-bootstrap-fix smoke run: `artifacts/gemm_smoke/run_20260328T155619Z_7fd01b07`
+- result: profiler path was fixed and `counter_compatibility.validation_backend` became `ncu_query_metrics`
+- remaining issue: `smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg` populated `0/4` rows and caused `unsupported_counter` failures in the calibration profile path
+- interpretation: the environment bug was resolved, but the default smoke counter set still contained a metric that was operationally fragile on the pinned host
+
+## Counter Audit Outcome
+
+### `default_calibration`
+
+- replaced `smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg`
+- with `smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct`
+- reason: the original metric queried successfully but returned null values in live smoke profiling on `gpunode2`
+
+### `compute_lite`
+
+- validation command reported `6/6` queried metrics available
+- reportable GEMM run populated every configured counter for profiled strategies
+- no counter replacement was required
+
+### `memory_lite`
+
+- validation command reported `6/6` queried metrics available
+- reportable LayerNorm run populated every configured counter for profiled strategies
+- no counter replacement was required
+
+## Rerun Results
+
+### Smoke
+
+- rerun: `artifacts/gemm_smoke/run_20260328T182208Z_78c5f7dc`
+- `profile_failures` reduced to `{"success": 4}`
+- `counter_set_accepted` was `true`
+- all calibration counters populated `4/4` rows in `counter_availability_report.csv`
+
+### GEMM reportable
+
+- rerun: `artifacts/gemm_reportable/run_20260328T182338Z_6e92e07c`
+- `terminal_status`: `success`
+- `profile_failures`: `{"skipped_budget": 24, "success": 8}`
+- `counter_set_accepted`: `true`
+- all `compute_lite` counters populated `4/4` rows for `prune_rank_profiled` and `prune_rank_revised`
+- `reportability.is_reportable`: `false`
+- limiting reason: `budget_limited_decision_present`
+
+### LayerNorm reportable
+
+- rerun: `artifacts/layernorm_reportable/run_20260328T182601Z_5351e70a`
+- `terminal_status`: `success`
+- `profile_failures`: `{"success": 8, "skipped_budget": 8}`
+- `counter_set_accepted`: `true`
+- all `memory_lite` counters populated `4/4` rows for `prune_rank_profiled` and `prune_rank_revised`
+- `reportability.is_reportable`: `false`
+- limiting reason: `budget_limited_decision_present`
+
+## Interpretation
+
+- profiling is now operationally trustworthy on `gpunode2` for the audited smoke and reportable paths
+- the resolved problem was environment propagation, not a fundamental incompatibility between Nsight Compute and the pinned host
+- the remaining obstacle to study-level hypothesis testing is reportability semantics under matched-budget runs, not counter availability or profiler invocation
+- this means the profiling hypothesis can now be tested operationally, but the current `validation_phase` study spec still does not admit these first reruns as study evidence because it requires reportable runs and broader repeat/seed coverage
+
+## Immediate Follow-up
+
+- treat the profiling tool-path issue as closed for PR `#11`
+- keep `default_calibration` as a smoke-only compatibility set with intentionally robust counters
+- address `budget_limited_decision_present` semantics before expecting `compare-runs --spec configs/studies/validation_phase.yaml` to ingest these reruns
+- expand reportable validation coverage to satisfy the study spec's repeat and seed requirements before drawing hypothesis-level conclusions

scripts/slurm/run_kerneltuner_array.sbatch

@@ -172,7 +172,7 @@ fi
 VENV_PATH="$SCRATCH_ROOT/$VENV_NAME"
 if [[ "${INSTALL_PACKAGES:-1}" == "1" ]]; then
   echo "Bootstrapping environment into $VENV_PATH"
-  "$WORKSPACE_ROOT/scripts/bootstrap_env.sh" "$VENV_PATH"
+  source "$WORKSPACE_ROOT/scripts/bootstrap_env.sh" "$VENV_PATH"
 else
   if [[ ! -d "$VENV_PATH" ]]; then
     echo "ERROR: virtual environment missing at $VENV_PATH and INSTALL_PACKAGES=0" >&2

src/kernel_tuner/analysis/comparison.py

@@ -72,9 +72,12 @@ def compare_runs(
     )
     store.initialize_manifest(manifest)
     try:
-        run_payloads = _resolve_run_payloads(study_spec, study_path)
+        run_payloads, filter_diagnostics = _resolve_run_payloads(study_spec, study_path)
         if not run_payloads:
-            raise ValueError("no study runs matched the configured groups and filters")
+            raise ValueError(
+                "no study runs matched the configured groups and filters"
+                + _format_filter_diagnostics(filter_diagnostics)
+            )
         strategy_rows = _build_strategy_rows(run_payloads)
         if strategy_rows.empty:
             raise ValueError("no held-out strategy rows were available after loading matched study runs")
@@ -179,18 +182,43 @@ def validate_study_from_path(study_path: str | Path) -> dict[str, object]:
     }
 
 
-def _resolve_run_payloads(study_spec: StudySpec, study_path: str | Path | None) -> list[dict[str, Any]]:
+def _resolve_run_payloads(
+    study_spec: StudySpec,
+    study_path: str | Path | None,
+) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
     payloads: list[dict[str, Any]] = []
+    diagnostics: list[dict[str, Any]] = []
     seen_run_dirs: set[Path] = set()
     for group in study_spec.run_groups:
+        group_stats = {
+            "group_id": group.group_id,
+            "candidate_run_count": 0,
+            "matched_run_count": 0,
+            "excluded_reportability": 0,
+            "excluded_environment": 0,
+            "excluded_kernel_family": 0,
+            "excluded_selector_version": 0,
+            "excluded_selector_revision_id": 0,
+            "excluded_counter_set_id": 0,
+            "excluded_budget_id": 0,
+            "excluded_execution_mode": 0,
+            "excluded_seed": 0,
+            "excluded_repeat_index": 0,
+        }
         for run_dir in _resolve_group_run_dirs(group, study_spec, study_path):
             if run_dir in seen_run_dirs:
                 continue
             seen_run_dirs.add(run_dir)
+            group_stats["candidate_run_count"] += 1
             payload = _load_run_payload(run_dir, group.group_id)
-            if _passes_filters(payload, study_spec):
+            accepted, reason = _passes_filters(payload, study_spec)
+            if accepted:
                 payloads.append(payload)
-    return payloads
+                group_stats["matched_run_count"] += 1
+            elif reason is not None:
+                group_stats[reason] += 1
+        diagnostics.append(group_stats)
+    return payloads, diagnostics
 
 
 def _resolve_group_run_dirs(group, study_spec: StudySpec, study_path: str | Path | None) -> list[Path]:
@@ -229,7 +257,7 @@ def _load_run_payload(run_dir: Path, group_id: str) -> dict[str, Any]:
         summarize_run(run_dir)
     summary = json.loads(summary_path.read_text(encoding="utf-8"))
     source_experiment_path = run_dir / "experiment_spec.yaml"
-    if manifest.invocation.experiment_config_path:
+    if not source_experiment_path.exists() and manifest.invocation.experiment_config_path:
         candidate = Path(manifest.invocation.experiment_config_path)
         if candidate.exists():
             source_experiment_path = candidate
@@ -260,37 +288,64 @@ def _load_run_payload(run_dir: Path, group_id: str) -> dict[str, Any]:
     }
 
 
-def _passes_filters(payload: dict[str, Any], study_spec: StudySpec) -> bool:
+def _passes_filters(payload: dict[str, Any], study_spec: StudySpec) -> tuple[bool, str | None]:
     summary = payload["summary"]
     manifest = payload["manifest"]
     labels = payload.get("run_labels") or {}
     group_id = payload["group_id"]
     group = next((item for item in study_spec.run_groups if item.group_id == group_id), None)
     if study_spec.reportability_filter and not summary.get("reportability", {}).get("is_reportable", False):
-        return False
+        return False, "excluded_reportability"
     for field, expected in study_spec.environment_filter.items():
         actual = getattr(manifest.environment, field, None)
         if actual != expected:
-            return False
+            return False, "excluded_environment"
     if group is None:
-        return True
+        return True, None
     if group.kernel_family and labels.get("kernel_family") != group.kernel_family:
-        return False
+        return False, "excluded_kernel_family"
     if group.selector_version and labels.get("selector_version") != group.selector_version:
-        return False
+        return False, "excluded_selector_version"
     if group.selector_revision_id and labels.get("selector_revision_id") != group.selector_revision_id:
-        return False
+        return False, "excluded_selector_revision_id"
     if group.counter_set_id and labels.get("counter_set_id") != group.counter_set_id:
-        return False
+        return False, "excluded_counter_set_id"
     if group.budget_id and labels.get("budget_id") != group.budget_id:
-        return False
+        return False, "excluded_budget_id"
     if group.execution_mode and labels.get("execution_mode") != group.execution_mode:
-        return False
+        return False, "excluded_execution_mode"
     if group.seeds and labels.get("seed") not in group.seeds:
-        return False
+        return False, "excluded_seed"
     if group.repeat_indices and labels.get("repeat_index") not in group.repeat_indices:
-        return False
-    return True
+        return False, "excluded_repeat_index"
+    return True, None
+
+
+def _format_filter_diagnostics(filter_diagnostics: list[dict[str, Any]]) -> str:
+    if not filter_diagnostics:
+        return ""
+    parts: list[str] = []
+    for row in filter_diagnostics:
+        details = [
+            f"candidates={row['candidate_run_count']}",
+            f"matched={row['matched_run_count']}",
+        ]
+        for key, label in [
+            ("excluded_reportability", "reportability"),
+            ("excluded_environment", "environment"),
+            ("excluded_kernel_family", "kernel_family"),
+            ("excluded_selector_version", "selector_version"),
+            ("excluded_selector_revision_id", "selector_revision_id"),
+            ("excluded_counter_set_id", "counter_set_id"),
+            ("excluded_budget_id", "budget_id"),
+            ("excluded_execution_mode", "execution_mode"),
+            ("excluded_seed", "seed"),
+            ("excluded_repeat_index", "repeat_index"),
+        ]:
+            if row[key]:
+                details.append(f"{label}={row[key]}")
+        parts.append(f"{row['group_id']}(" + ", ".join(details) + ")")
+    return ": " + "; ".join(parts)
 
 
 def _build_strategy_rows(run_payloads: list[dict[str, Any]]) -> pd.DataFrame:
@@ -680,6 +735,9 @@ def _aggregate_frontier_diagnostics(run_payloads: list[dict[str, Any]]) -> pd.Da
         decorated = frame.copy()
         decorated["group_id"] = payload["group_id"]
         decorated["run_id"] = payload["summary"]["run_id"]
+        if "run_ids" in decorated.columns:
+            missing_run_ids = decorated["run_ids"].astype(str).eq("") | decorated["run_ids"].isna()
+            decorated.loc[missing_run_ids, "run_ids"] = payload["summary"]["run_id"]
         decorated["experiment_id"] = payload["experiment_spec"].experiment_id
         rows.append(decorated)
     if not rows: