diff --git a/README.md b/README.md index 90621ac..0a523cd 100644 --- a/README.md +++ b/README.md @@ -84,8 +84,7 @@ On the current homogeneous `RTX A6000` pool: ```bash export KTUNE_SCRATCH=/scratch/scratch-space/expires-xxxx/$USER/kerneltuner -scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312" -source "$KTUNE_SCRATCH/venv-py312/bin/activate" +source scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312" ktune validate-kernel --kernel configs/kernels/gemm.yaml ktune run-experiment --experiment configs/experiments/gemm_smoke.yaml @@ -136,6 +135,54 @@ scripts/slurm/submit_kerneltuner.sh \ --mem 24GB ``` +## Slurm Submission Helpers + +For cluster execution, the repo includes reusable Slurm scripts: + +- `scripts/slurm/run_kerneltuner_array.sbatch`: array worker that maps one array task to one experiment YAML. +- `scripts/slurm/submit_kerneltuner.sh`: submit wrapper that computes array size from a list file. + +Example list file: + +- `configs/experiments/slurm_experiment_list.example.txt` + +Basic usage: + +```bash +chmod +x scripts/slurm/submit_kerneltuner.sh +cp configs/experiments/slurm_experiment_list.example.txt configs/experiments/slurm_experiment_list.txt +scripts/slurm/submit_kerneltuner.sh \ + --list configs/experiments/slurm_experiment_list.txt \ + --partition gpunodes \ + --gpu-type rtx_a2000 \ + --gpus 1 \ + --time 0-04:00 \ + --cpus 4 \ + --mem 24GB \ + --log-dir /scratch/scratch-space/expires-xxxx/$USER/kerneltuner_logs \ + --scratch-root /scratch/scratch-space/expires-xxxx/$USER/kerneltuner \ + --artifact-root /scratch/scratch-space/expires-xxxx/$USER/kerneltuner_artifacts \ + --mail-user you@example.com \ + --mail-type BEGIN,END,FAIL \ + --alert-email you@example.com \ + --alert-on-start +``` + +Useful environment overrides: + +- `RUN_COMMAND_TEMPLATE` (default: `ktune run-experiment --experiment "{experiment}"`) +- `INSTALL_PACKAGES` (`0` to skip pip install in jobs) +- `EXTRA_PIP_PACKAGES` (space-separated extra pip packages) +- `SKIP_IF_ARTIFACTS_EXIST` (`0` to force reruns) +- `DRY_RUN=1` (worker-level dry run) +- `ALERT_EMAIL`, `ALERT_ON_START`, `ALERT_ON_END`, `ALERT_ON_FAIL` for worker-level alerts + +Notes: + +- `--artifact-root` overrides `artifact_root` in experiment YAML at submission time. +- If no scratch path exists on the node, jobs fall back to `/.scratch/$USER`. +- Slurm native email (`--mail-user/--mail-type`) and worker-level alerts can be used together. + ## What This Repo Is Not - It is not a Triton compiler redesign effort. diff --git a/configs/counters/default_calibration.yaml b/configs/counters/default_calibration.yaml index 9e26099..5b8a661 100644 --- a/configs/counters/default_calibration.yaml +++ b/configs/counters/default_calibration.yaml @@ -4,7 +4,7 @@ tool: ncu replay_mode: kernel target_processes: all counters: - - smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg + - smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct - sm__warps_active.avg.pct_of_peak_sustained_active - smsp__inst_executed.sum minimum_availability: 0.9 diff --git a/configs/studies/validation_phase.yaml b/configs/studies/validation_phase.yaml index 77e15a8..529add9 100644 --- a/configs/studies/validation_phase.yaml +++ b/configs/studies/validation_phase.yaml @@ -1,7 +1,7 @@ study_id: validation_phase hypotheses: - hypothesis_id: H1 - description: Cheap compile signals prune obvious losers, but do not reliably rank near-tied GEMM configs. + description: Cheap compile signals prune obvious losers, but produce noticeably unstable ranking quality on representative GEMM workloads. clauses: - left: source: strategy_rows diff --git a/docs/gpu_job_guide.md b/docs/gpu_job_guide.md index 928370e..fc46f5c 100644 --- a/docs/gpu_job_guide.md +++ b/docs/gpu_job_guide.md @@ -150,15 +150,13 @@ else export KTUNE_SCRATCH="/tmp/$USER/kerneltuner" fi -scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312" -source "$KTUNE_SCRATCH/venv-py312/bin/activate" +source scripts/bootstrap_env.sh "$KTUNE_SCRATCH/venv-py312" ``` If you want to choose the virtualenv path explicitly: ```bash -scripts/bootstrap_env.sh /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312 -source /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312/bin/activate +source scripts/bootstrap_env.sh /scratch/scratch-space/expires-xxxx/$USER/kerneltuner/venv-py312 ``` The bootstrap script: diff --git a/docs/research/logs/2026-03-28_profiling_reliability_validation.md b/docs/research/logs/2026-03-28_profiling_reliability_validation.md new file mode 100644 index 0000000..7093459 --- /dev/null +++ b/docs/research/logs/2026-03-28_profiling_reliability_validation.md @@ -0,0 +1,90 @@ +# Profiling Reliability Validation On gpunode2 + +Purpose: record the profiling reliability fix, the counter-set audit, and the first reportable reruns on the pinned validation host. +Status: Log +Update Rule: append-only; do not rewrite past observations except to fix factual errors. +Feeds Paper Sections: operational methodology, profiling validity, limitations. +Depends On: [../04_signal_and_profiling_plan.md](../04_signal_and_profiling_plan.md), [../06_hypotheses_and_ablation_plan.md](../06_hypotheses_and_ablation_plan.md), [../08_evidence_registry.md](../08_evidence_registry.md), [../../04_experiment_protocol.md](../../04_experiment_protocol.md) + +## What Changed + +- preserved the CUDA toolchain exports by making `source scripts/bootstrap_env.sh ` the canonical launch contract for both manual GPU-shell runs and the Slurm worker path +- updated profiler invocation and provenance capture to resolve tool paths through `CUDA_HOME/bin` before falling back to `PATH` +- tightened profiling diagnostics so artifacts distinguish missing profiler binaries, missing queried metrics, invocation failures, kernel attribution failures, and runs that return null counter values +- replaced the fragile `default_calibration` long-scoreboard counter with a queryable and populated variant on `gpunode2` + +## Original Failure + +- baseline smoke run: `artifacts/gemm_smoke/run_20260328T153717Z_fb053320` +- observed failure: `counter_compatibility.notes` reported `ncu executable not found` +- root cause: `bootstrap_env.sh` was executed in a subprocess, so the exported CUDA tool paths were lost before `ktune` launched profiling + +## Intermediate Validation + +- post-bootstrap-fix smoke run: `artifacts/gemm_smoke/run_20260328T155619Z_7fd01b07` +- result: profiler path was fixed and `counter_compatibility.validation_backend` became `ncu_query_metrics` +- remaining issue: `smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg` populated `0/4` rows and caused `unsupported_counter` failures in the calibration profile path +- interpretation: the environment bug was resolved, but the default smoke counter set still contained a metric that was operationally fragile on the pinned host + +## Counter Audit Outcome + +### `default_calibration` + +- replaced `smsp__average_warps_issue_stalled_long_scoreboard_per_issue_active.avg` +- with `smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct` +- reason: the original metric queried successfully but returned null values in live smoke profiling on `gpunode2` + +### `compute_lite` + +- validation command reported `6/6` queried metrics available +- reportable GEMM run populated every configured counter for profiled strategies +- no counter replacement was required + +### `memory_lite` + +- validation command reported `6/6` queried metrics available +- reportable LayerNorm run populated every configured counter for profiled strategies +- no counter replacement was required + +## Rerun Results + +### Smoke + +- rerun: `artifacts/gemm_smoke/run_20260328T182208Z_78c5f7dc` +- `profile_failures` reduced to `{"success": 4}` +- `counter_set_accepted` was `true` +- all calibration counters populated `4/4` rows in `counter_availability_report.csv` + +### GEMM reportable + +- rerun: `artifacts/gemm_reportable/run_20260328T182338Z_6e92e07c` +- `terminal_status`: `success` +- `profile_failures`: `{"skipped_budget": 24, "success": 8}` +- `counter_set_accepted`: `true` +- all `compute_lite` counters populated `4/4` rows for `prune_rank_profiled` and `prune_rank_revised` +- `reportability.is_reportable`: `false` +- limiting reason: `budget_limited_decision_present` + +### LayerNorm reportable + +- rerun: `artifacts/layernorm_reportable/run_20260328T182601Z_5351e70a` +- `terminal_status`: `success` +- `profile_failures`: `{"success": 8, "skipped_budget": 8}` +- `counter_set_accepted`: `true` +- all `memory_lite` counters populated `4/4` rows for `prune_rank_profiled` and `prune_rank_revised` +- `reportability.is_reportable`: `false` +- limiting reason: `budget_limited_decision_present` + +## Interpretation + +- profiling is now operationally trustworthy on `gpunode2` for the audited smoke and reportable paths +- the resolved problem was environment propagation, not a fundamental incompatibility between Nsight Compute and the pinned host +- the remaining obstacle to study-level hypothesis testing is reportability semantics under matched-budget runs, not counter availability or profiler invocation +- this means the profiling hypothesis can now be tested operationally, but the current `validation_phase` study spec still does not admit these first reruns as study evidence because it requires reportable runs and broader repeat/seed coverage + +## Immediate Follow-up + +- treat the profiling tool-path issue as closed for PR `#11` +- keep `default_calibration` as a smoke-only compatibility set with intentionally robust counters +- address `budget_limited_decision_present` semantics before expecting `compare-runs --spec configs/studies/validation_phase.yaml` to ingest these reruns +- expand reportable validation coverage to satisfy the study spec's repeat and seed requirements before drawing hypothesis-level conclusions diff --git a/scripts/slurm/run_kerneltuner_array.sbatch b/scripts/slurm/run_kerneltuner_array.sbatch index d778ec2..5ce46a1 100755 --- a/scripts/slurm/run_kerneltuner_array.sbatch +++ b/scripts/slurm/run_kerneltuner_array.sbatch @@ -172,7 +172,7 @@ fi VENV_PATH="$SCRATCH_ROOT/$VENV_NAME" if [[ "${INSTALL_PACKAGES:-1}" == "1" ]]; then echo "Bootstrapping environment into $VENV_PATH" - "$WORKSPACE_ROOT/scripts/bootstrap_env.sh" "$VENV_PATH" + source "$WORKSPACE_ROOT/scripts/bootstrap_env.sh" "$VENV_PATH" else if [[ ! -d "$VENV_PATH" ]]; then echo "ERROR: virtual environment missing at $VENV_PATH and INSTALL_PACKAGES=0" >&2 diff --git a/src/kernel_tuner/analysis/comparison.py b/src/kernel_tuner/analysis/comparison.py index 6171d42..7dcc94d 100644 --- a/src/kernel_tuner/analysis/comparison.py +++ b/src/kernel_tuner/analysis/comparison.py @@ -72,9 +72,12 @@ def compare_runs( ) store.initialize_manifest(manifest) try: - run_payloads = _resolve_run_payloads(study_spec, study_path) + run_payloads, filter_diagnostics = _resolve_run_payloads(study_spec, study_path) if not run_payloads: - raise ValueError("no study runs matched the configured groups and filters") + raise ValueError( + "no study runs matched the configured groups and filters" + + _format_filter_diagnostics(filter_diagnostics) + ) strategy_rows = _build_strategy_rows(run_payloads) if strategy_rows.empty: raise ValueError("no held-out strategy rows were available after loading matched study runs") @@ -179,18 +182,43 @@ def validate_study_from_path(study_path: str | Path) -> dict[str, object]: } -def _resolve_run_payloads(study_spec: StudySpec, study_path: str | Path | None) -> list[dict[str, Any]]: +def _resolve_run_payloads( + study_spec: StudySpec, + study_path: str | Path | None, +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: payloads: list[dict[str, Any]] = [] + diagnostics: list[dict[str, Any]] = [] seen_run_dirs: set[Path] = set() for group in study_spec.run_groups: + group_stats = { + "group_id": group.group_id, + "candidate_run_count": 0, + "matched_run_count": 0, + "excluded_reportability": 0, + "excluded_environment": 0, + "excluded_kernel_family": 0, + "excluded_selector_version": 0, + "excluded_selector_revision_id": 0, + "excluded_counter_set_id": 0, + "excluded_budget_id": 0, + "excluded_execution_mode": 0, + "excluded_seed": 0, + "excluded_repeat_index": 0, + } for run_dir in _resolve_group_run_dirs(group, study_spec, study_path): if run_dir in seen_run_dirs: continue seen_run_dirs.add(run_dir) + group_stats["candidate_run_count"] += 1 payload = _load_run_payload(run_dir, group.group_id) - if _passes_filters(payload, study_spec): + accepted, reason = _passes_filters(payload, study_spec) + if accepted: payloads.append(payload) - return payloads + group_stats["matched_run_count"] += 1 + elif reason is not None: + group_stats[reason] += 1 + diagnostics.append(group_stats) + return payloads, diagnostics def _resolve_group_run_dirs(group, study_spec: StudySpec, study_path: str | Path | None) -> list[Path]: @@ -229,7 +257,7 @@ def _load_run_payload(run_dir: Path, group_id: str) -> dict[str, Any]: summarize_run(run_dir) summary = json.loads(summary_path.read_text(encoding="utf-8")) source_experiment_path = run_dir / "experiment_spec.yaml" - if manifest.invocation.experiment_config_path: + if not source_experiment_path.exists() and manifest.invocation.experiment_config_path: candidate = Path(manifest.invocation.experiment_config_path) if candidate.exists(): source_experiment_path = candidate @@ -260,37 +288,64 @@ def _load_run_payload(run_dir: Path, group_id: str) -> dict[str, Any]: } -def _passes_filters(payload: dict[str, Any], study_spec: StudySpec) -> bool: +def _passes_filters(payload: dict[str, Any], study_spec: StudySpec) -> tuple[bool, str | None]: summary = payload["summary"] manifest = payload["manifest"] labels = payload.get("run_labels") or {} group_id = payload["group_id"] group = next((item for item in study_spec.run_groups if item.group_id == group_id), None) if study_spec.reportability_filter and not summary.get("reportability", {}).get("is_reportable", False): - return False + return False, "excluded_reportability" for field, expected in study_spec.environment_filter.items(): actual = getattr(manifest.environment, field, None) if actual != expected: - return False + return False, "excluded_environment" if group is None: - return True + return True, None if group.kernel_family and labels.get("kernel_family") != group.kernel_family: - return False + return False, "excluded_kernel_family" if group.selector_version and labels.get("selector_version") != group.selector_version: - return False + return False, "excluded_selector_version" if group.selector_revision_id and labels.get("selector_revision_id") != group.selector_revision_id: - return False + return False, "excluded_selector_revision_id" if group.counter_set_id and labels.get("counter_set_id") != group.counter_set_id: - return False + return False, "excluded_counter_set_id" if group.budget_id and labels.get("budget_id") != group.budget_id: - return False + return False, "excluded_budget_id" if group.execution_mode and labels.get("execution_mode") != group.execution_mode: - return False + return False, "excluded_execution_mode" if group.seeds and labels.get("seed") not in group.seeds: - return False + return False, "excluded_seed" if group.repeat_indices and labels.get("repeat_index") not in group.repeat_indices: - return False - return True + return False, "excluded_repeat_index" + return True, None + + +def _format_filter_diagnostics(filter_diagnostics: list[dict[str, Any]]) -> str: + if not filter_diagnostics: + return "" + parts: list[str] = [] + for row in filter_diagnostics: + details = [ + f"candidates={row['candidate_run_count']}", + f"matched={row['matched_run_count']}", + ] + for key, label in [ + ("excluded_reportability", "reportability"), + ("excluded_environment", "environment"), + ("excluded_kernel_family", "kernel_family"), + ("excluded_selector_version", "selector_version"), + ("excluded_selector_revision_id", "selector_revision_id"), + ("excluded_counter_set_id", "counter_set_id"), + ("excluded_budget_id", "budget_id"), + ("excluded_execution_mode", "execution_mode"), + ("excluded_seed", "seed"), + ("excluded_repeat_index", "repeat_index"), + ]: + if row[key]: + details.append(f"{label}={row[key]}") + parts.append(f"{row['group_id']}(" + ", ".join(details) + ")") + return ": " + "; ".join(parts) def _build_strategy_rows(run_payloads: list[dict[str, Any]]) -> pd.DataFrame: @@ -680,6 +735,9 @@ def _aggregate_frontier_diagnostics(run_payloads: list[dict[str, Any]]) -> pd.Da decorated = frame.copy() decorated["group_id"] = payload["group_id"] decorated["run_id"] = payload["summary"]["run_id"] + if "run_ids" in decorated.columns: + missing_run_ids = decorated["run_ids"].astype(str).eq("") | decorated["run_ids"].isna() + decorated.loc[missing_run_ids, "run_ids"] = payload["summary"]["run_id"] decorated["experiment_id"] = payload["experiment_spec"].experiment_id rows.append(decorated) if not rows: diff --git a/src/kernel_tuner/analysis/reporting.py b/src/kernel_tuner/analysis/reporting.py index ec7d074..16fc3fa 100644 --- a/src/kernel_tuner/analysis/reporting.py +++ b/src/kernel_tuner/analysis/reporting.py @@ -337,7 +337,7 @@ def summarize_run(run_dir: str | Path) -> dict[str, object]: source_experiment_path = experiment_spec_path if not experiment_spec_path.exists(): source_experiment_path = None - if manifest.invocation.experiment_config_path: + if source_experiment_path is None and manifest.invocation.experiment_config_path: candidate = Path(manifest.invocation.experiment_config_path) if candidate.exists(): source_experiment_path = candidate @@ -527,17 +527,12 @@ def summarize_run(run_dir: str | Path) -> dict[str, object]: ) comparison_classes = {value for value in comparison_class_by_strategy.values() if value is not None} matched_budget_only = bool(comparison_classes) and comparison_classes == {"matched_budget"} - decision_statuses = ( - set(selection_decisions["decision_status"].dropna().astype(str)) - if "decision_status" in selection_decisions.columns - else set() - ) + decision_statuses = set(selection_decisions["decision_status"].dropna().astype(str)) if "decision_status" in selection_decisions.columns else set() has_budget_limited_decision = any(status.endswith("budget_limited") for status in decision_statuses) is_reportable = bool( experiment_spec.study_kind == "reportable" and held_out_shape_count >= policy.minimum_held_out_shapes and matched_budget_only - and not has_budget_limited_decision and (not counter_compatibility or counter_compatibility.get("acceptable", True)) and (not counter_availability_ok or all(counter_availability_ok.values())) and ( @@ -580,6 +575,10 @@ def summarize_run(run_dir: str | Path) -> dict[str, object]: interpretation_notes.append("No usable profiler counter availability rows were recorded.") elif not availability_frame.empty and not availability_frame["acceptable"].all(): interpretation_notes.append("At least one requested profiler counter failed the availability threshold.") + if has_budget_limited_decision: + interpretation_notes.append( + "At least one strategy exhausted its matched budget before evaluating every requested profile or benchmark." + ) result = ExperimentResult( schema_version=SCHEMA_VERSION, @@ -614,7 +613,7 @@ def summarize_run(run_dir: str | Path) -> dict[str, object]: reportability={ "target": experiment_spec.analysis_settings.reportability_target, "is_reportable": is_reportable, - "comparison_class": "matched_budget" if is_reportable else "non_comparable", + "comparison_class": "matched_budget" if matched_budget_only else "non_comparable", "matched_budget_only": matched_budget_only, "budget_limited_decision_present": has_budget_limited_decision, "counter_set_accepted": bool(counter_compatibility.get("acceptable", True)) diff --git a/src/kernel_tuner/baselines/strategies.py b/src/kernel_tuner/baselines/strategies.py index f1f86bb..c10b51f 100644 --- a/src/kernel_tuner/baselines/strategies.py +++ b/src/kernel_tuner/baselines/strategies.py @@ -11,6 +11,7 @@ BaselineMode, CandidateConfig, ComparisonClass, + MeasurementPhase, RuntimeMeasurement, RuntimeStatus, SelectionDecision, @@ -146,6 +147,12 @@ def run_baseline_mode( runtime_records.extend(request_benchmark(config_id)) runtime_scores = aggregate_runtime_scores(runtime_records) selected = min(runtime_scores, key=lambda config_id: (runtime_scores[config_id], config_id)) if runtime_scores else None + consumed_benchmarks = sum( + 1 + for record in runtime_records + if record.measurement_phase == MeasurementPhase.CALIBRATION + and record.status != RuntimeStatus.SKIPPED_BUDGET + ) budget_limited = any(record.status == RuntimeStatus.SKIPPED_BUDGET for record in runtime_records) if selected is None: decision_status = "failed_no_successful_measurements" @@ -164,7 +171,7 @@ def run_baseline_mode( ranked_config_ids=benchmark_ids, pruned_config_ids=[], candidates_considered=len(ordered_ids), - benchmarks_requested=len(benchmark_ids), + benchmarks_requested=consumed_benchmarks, profiles_requested=0, decision_wall_clock_s=time.perf_counter() - started, rationale_summary="; ".join(rationale), diff --git a/src/kernel_tuner/common/provenance.py b/src/kernel_tuner/common/provenance.py index 3633a6d..4b87c42 100644 --- a/src/kernel_tuner/common/provenance.py +++ b/src/kernel_tuner/common/provenance.py @@ -27,6 +27,16 @@ def _run_command(args: list[str], cwd: str | Path | None = None) -> str | None: return completed.stdout.strip() +def resolve_tool_path(name: str) -> str: + cuda_home = os.environ.get("CUDA_HOME") + if cuda_home: + candidate = Path(cuda_home) / "bin" / name + if candidate.exists(): + return str(candidate) + resolved = which(name) + return resolved or name + + def _import_version(module_name: str) -> str | None: try: module = __import__(module_name) @@ -85,7 +95,7 @@ def _capture_git(repo_root: str | Path) -> dict[str, str | bool | None]: def _capture_ncu_version() -> str | None: - output = _run_command(["ncu", "--version"]) + output = _run_command([resolve_tool_path("ncu"), "--version"]) if output: line = output.splitlines()[-1].strip() return line @@ -104,8 +114,8 @@ def _cache_roots() -> dict[str, str]: def _tool_paths() -> dict[str, str]: tools = {} for name in ["python3", "ncu", "nsys", "nvcc"]: - path = which(name) - if path: + path = resolve_tool_path(name) + if path != name: tools[name] = path return tools diff --git a/src/kernel_tuner/experiments/orchestrator.py b/src/kernel_tuner/experiments/orchestrator.py index f338b0e..98e25b3 100644 --- a/src/kernel_tuner/experiments/orchestrator.py +++ b/src/kernel_tuner/experiments/orchestrator.py @@ -114,9 +114,21 @@ def _validate_environment(spec: ExperimentSpec, environment) -> None: def _profile_shapes(spec: ExperimentSpec, calibration_shapes) -> list: if not calibration_shapes: return [] + first_calibration = calibration_shapes[0] + + def _shape_work_score(shape) -> tuple[int, str]: + score = 1 + for value in shape.dimensions.values(): + score *= value + return score, shape.shape_id + + def _representative_shape(shapes): + ordered = sorted(shapes, key=_shape_work_score) + return ordered[len(ordered) // 2] + policy = spec.profile_policy if policy is None: - return [calibration_shapes[0]] + return [first_calibration] mode = policy.shape_sampling_mode if mode == "all_calibration": selected = list(calibration_shapes) @@ -124,18 +136,18 @@ def _profile_shapes(spec: ExperimentSpec, calibration_shapes) -> list: explicit_ids = set(policy.explicit_shape_ids) selected = [shape for shape in calibration_shapes if shape.shape_id in explicit_ids] elif mode == "first_calibration": - selected = [calibration_shapes[0]] + selected = [first_calibration] elif mode == "per_workload_class_top1": by_class: dict[str, list] = {} for shape in calibration_shapes: by_class.setdefault(shape.workload_class or "__unlabeled__", []).append(shape) selected = [ - sorted(shapes, key=lambda shape: shape.shape_id)[0] + _representative_shape(shapes) for _, shapes in sorted(by_class.items()) if shapes ] else: - selected = [sorted(calibration_shapes, key=lambda shape: shape.shape_id)[0]] + selected = [first_calibration] max_shapes = policy.max_shapes_per_config if policy else None if max_shapes is not None: selected = selected[:max_shapes] @@ -328,6 +340,8 @@ def __init__( self.selector_revision = selector_revision self.deadline_s = deadline_s self.measurement_order_counter = measurement_order_counter + self.remaining_benchmarks = experiment_spec.budgets.max_benchmarks + self.remaining_profiles = experiment_spec.budgets.max_profiles self._candidate_lookup = { (candidate.shape_id, candidate.config_id): candidate for candidate in candidate_records } @@ -344,6 +358,12 @@ def _next_measurement_order_index(self) -> int: def _budget_exhausted(self) -> bool: return self.deadline_s is not None and time.perf_counter() >= self.deadline_s + def _benchmark_budget_exhausted(self) -> bool: + return self.remaining_benchmarks <= 0 + + def _profile_budget_exhausted(self) -> bool: + return self.remaining_profiles <= 0 + def benchmark_calibration(self, config_id: str) -> list[RuntimeMeasurement]: return self._benchmark_shapes(config_id, self.calibration_shapes, MeasurementPhase.CALIBRATION) @@ -357,7 +377,8 @@ def _benchmark_shapes(self, config_id: str, shapes, phase: MeasurementPhase) -> records: list[RuntimeMeasurement] = [] for index, shape in enumerate(shapes): - if self._budget_exhausted(): + enforce_budget = phase == MeasurementPhase.CALIBRATION + if enforce_budget and (self._budget_exhausted() or self._benchmark_budget_exhausted()): skipped = RuntimeMeasurement( run_id=self.run_id, strategy_id=self.strategy_id, @@ -376,12 +397,18 @@ def _benchmark_shapes(self, config_id: str, shapes, phase: MeasurementPhase) -> status=RuntimeStatus.SKIPPED_BUDGET, timing_backend=self.experiment_spec.benchmark_settings.timing_backend, measurement_order_index=self._next_measurement_order_index(), - error_message="wall_clock_limit_s exhausted before benchmark", + error_message=( + "wall_clock_limit_s exhausted before benchmark" + if self._budget_exhausted() + else "benchmark budget exhausted before measurement" + ), ) records.append(skipped) self.runtime_records.append(skipped) continue candidate = self._candidate_lookup[(shape.shape_id, config_id)] + if enforce_budget: + self.remaining_benchmarks -= 1 measurement_order_index = self._next_measurement_order_index() outcome = benchmark_candidate( run_id=self.run_id, @@ -414,7 +441,7 @@ def profile_calibration(self, config_id: str) -> list[ProfileMeasurement]: measurements: list[ProfileMeasurement] = [] for shape in self.profile_shapes: - if self._budget_exhausted(): + if self._budget_exhausted() or self._profile_budget_exhausted(): measurement = ProfileMeasurement( run_id=self.run_id, strategy_id=self.strategy_id, @@ -423,12 +450,17 @@ def profile_calibration(self, config_id: str) -> list[ProfileMeasurement]: config_id=config_id, counter_set_id=self.counter_set.counter_set_id, profile_status=ProfileStatus.SKIPPED_BUDGET, - notes="wall_clock_limit_s exhausted before profiling", + notes=( + "wall_clock_limit_s exhausted before profiling" + if self._budget_exhausted() + else "profile budget exhausted before measurement" + ), ) self.profile_records.append(measurement) measurements.append(measurement) continue candidate = self._candidate_lookup[(shape.shape_id, config_id)] + self.remaining_profiles -= 1 outcome = profile_candidate( run_id=self.run_id, strategy_id=self.strategy_id, @@ -544,6 +576,7 @@ def run_experiment( kernel = resolve_kernel(kernel_spec) profile_shapes = _profile_shapes(experiment_spec, calibration_shapes) + candidate_records = generate_candidate_records(experiment_spec, experiment_path=experiment_path) candidate_bundle = generate_candidate_bundle(experiment_spec, experiment_path=experiment_path) candidate_records = candidate_bundle["records"] store.write_json_artifact( diff --git a/src/kernel_tuner/profiling/adapter.py b/src/kernel_tuner/profiling/adapter.py index 245aae3..80cc0a8 100644 --- a/src/kernel_tuner/profiling/adapter.py +++ b/src/kernel_tuner/profiling/adapter.py @@ -20,7 +20,7 @@ load_experiment_spec, load_kernel_spec, ) -from kernel_tuner.common.provenance import python_command +from kernel_tuner.common.provenance import python_command, resolve_tool_path from kernel_tuner.common.schema import ( CandidateConfig, CounterSetSpec, @@ -149,8 +149,9 @@ def _parse_counter_map( @lru_cache(maxsize=1) def _ncu_version() -> str | None: try: + ncu = resolve_tool_path("ncu") completed = subprocess.run( - ["ncu", "--version"], + [ncu, "--version"], check=False, capture_output=True, text=True, @@ -172,6 +173,7 @@ def _base_profiler_metadata( ) -> dict[str, Any]: return { "command": command, + "resolved_profiler_path": command[0], "requested_counters": list(counter_set.counters), "target_processes": counter_set.target_processes or "all", "replay_mode": counter_set.replay_mode or experiment_spec.profiling_settings.replay_mode, @@ -190,8 +192,9 @@ def _build_profile_command( profiling_settings, payload: str, ) -> list[str]: + ncu = resolve_tool_path("ncu") command = [ - "ncu", + ncu, "--csv", "--page", "raw", @@ -262,6 +265,7 @@ def profile_candidate( **base_metadata, "returncode": None, "duration_s": time.perf_counter() - command_started, + "failure_reason": "tool_unavailable", }, notes="ncu executable not found", ) @@ -281,6 +285,7 @@ def profile_candidate( **base_metadata, "returncode": None, "duration_s": time.perf_counter() - command_started, + "failure_reason": "timeout", }, notes=f"ncu timed out: {exc}", ) @@ -298,18 +303,23 @@ def profile_candidate( ) status = ProfileStatus.SUCCESS notes = None + failure_reason = None attribution_failed = matched_kernel_name is None or diagnostics.get("kernel_attribution_status") != "matched" if completed.returncode != 0 and unsupported_metric: status = ProfileStatus.UNSUPPORTED_COUNTER + failure_reason = "unsupported_metric" notes = completed.stderr.strip() or completed.stdout.strip() or "unsupported counter" elif completed.returncode != 0: status = ProfileStatus.INVOCATION_FAILED + failure_reason = "invocation_failed" notes = completed.stderr.strip() or completed.stdout.strip() or "ncu invocation failed" elif attribution_failed: status = ProfileStatus.NO_PROFILE_DATA + failure_reason = f"kernel_attribution_{diagnostics.get('kernel_attribution_status')}" notes = f"no attributable profiler row: {diagnostics.get('kernel_attribution_status')}" elif missing_counters: status = ProfileStatus.UNSUPPORTED_COUNTER + failure_reason = "missing_counter_values" notes = "missing or unsupported counters: " + ", ".join(sorted(missing_counters)) measurement = ProfileMeasurement( @@ -327,6 +337,7 @@ def profile_candidate( "duration_s": time.perf_counter() - command_started, "matched_kernel_name": matched_kernel_name, "missing_counters": missing_counters, + "failure_reason": failure_reason, **diagnostics, }, notes=notes, diff --git a/src/kernel_tuner/profiling/compatibility.py b/src/kernel_tuner/profiling/compatibility.py index c650860..9333d62 100644 --- a/src/kernel_tuner/profiling/compatibility.py +++ b/src/kernel_tuner/profiling/compatibility.py @@ -13,6 +13,7 @@ load_experiment_spec, load_kernel_spec, ) +from kernel_tuner.common.provenance import resolve_tool_path from kernel_tuner.common.schema import CounterCompatibilityRecord, ExperimentSpec @@ -85,8 +86,9 @@ def validate_counter_set( try: completed = None if backend == "ncu_query_metrics": + ncu = resolve_tool_path("ncu") completed = subprocess.run( - ["ncu", "--query-metrics"], + [ncu, "--query-metrics"], check=False, capture_output=True, text=True, @@ -100,6 +102,10 @@ def validate_counter_set( discovered = set(re.findall(r"([A-Za-z0-9_.]+)", metrics_blob)) available = [counter for counter in requested if _metric_match(counter, discovered)] missing = [counter for counter in requested if counter not in available] + if missing: + notes_parts.append( + "missing or unsupported queried metrics: " + ", ".join(sorted(missing)) + ) elif completed is not None: backend = "ncu_query_failed" notes_parts.append(completed.stderr.strip() or completed.stdout.strip() or "ncu --query-metrics failed") diff --git a/src/kernel_tuner/selector/engine.py b/src/kernel_tuner/selector/engine.py index 544b4c9..b4fa114 100644 --- a/src/kernel_tuner/selector/engine.py +++ b/src/kernel_tuner/selector/engine.py @@ -741,6 +741,17 @@ def run_selector_mode( else None ) selected = _select_with_tolerance(runtime_scores, benchmark_order) + consumed_benchmarks = sum( + 1 + for record in runtime_records + if record.measurement_phase == MeasurementPhase.CALIBRATION + and record.status != RuntimeStatus.SKIPPED_BUDGET + ) + consumed_profiles = sum( + 1 + for record in profiled_records + if record.profile_status != ProfileStatus.SKIPPED_BUDGET + ) budget_limited = any( record.status == RuntimeStatus.SKIPPED_BUDGET for record in runtime_records ) or any(record.profile_status == ProfileStatus.SKIPPED_BUDGET for record in profiled_records) @@ -750,7 +761,7 @@ def run_selector_mode( else: decision_status = "selected_budget_limited" if budget_limited else "selected" rationale.append( - f"benchmarked {len(benchmark_ids)} configs and selected the best score within a 2% tie band" + f"consumed {consumed_benchmarks} calibration benchmark rows and selected the best score within a 2% tie band" ) diagnostic_limit = min(len(compile_rank_ids), max(budgets.max_benchmarks, budgets.max_profiles)) @@ -785,10 +796,8 @@ def run_selector_mode( ranked_config_ids=benchmark_order, pruned_config_ids=pruned_ids, candidates_considered=len(candidate_groups), - benchmarks_requested=len(benchmark_ids), - profiles_requested=len( - {record.config_id for record in profiled_records if record.profile_status != ProfileStatus.SKIPPED_BUDGET} - ), + benchmarks_requested=consumed_benchmarks, + profiles_requested=consumed_profiles, decision_wall_clock_s=time.perf_counter() - started, rationale_summary="; ".join(rationale), decision_status=decision_status, diff --git a/tests/integration/test_slurm_scripts.py b/tests/integration/test_slurm_scripts.py index a42ee63..1947951 100644 --- a/tests/integration/test_slurm_scripts.py +++ b/tests/integration/test_slurm_scripts.py @@ -20,3 +20,8 @@ def test_submit_script_dry_run_supports_nodelist(): text=True, ) assert "--nodelist=gpunode2" in result.stdout + + +def test_worker_script_sources_bootstrap_env_so_cuda_paths_persist(): + script = Path("scripts/slurm/run_kerneltuner_array.sbatch").read_text() + assert 'source "$WORKSPACE_ROOT/scripts/bootstrap_env.sh" "$VENV_PATH"' in script diff --git a/tests/unit/test_phase2_analysis.py b/tests/unit/test_phase2_analysis.py index 052e030..85420a9 100644 --- a/tests/unit/test_phase2_analysis.py +++ b/tests/unit/test_phase2_analysis.py @@ -9,6 +9,9 @@ _build_strategy_rows, _compare_hypothesis_values, _evaluate_hypotheses, + _format_filter_diagnostics, + _load_run_payload, + _passes_filters, ) from kernel_tuner.common.schema import ( HypothesisClause, @@ -26,6 +29,7 @@ from kernel_tuner.common.config import load_experiment_spec, load_kernel_spec, load_study_spec from kernel_tuner.common.provenance import capture_environment_metadata, capture_invocation_metadata from kernel_tuner.common.schema import Manifest +from kernel_tuner.storage import RunStore def test_counter_availability_records_apply_threshold(): @@ -180,6 +184,118 @@ def test_build_strategy_rows_preserves_kernel_family_and_workload_class(): assert set(rows["workload_class"]) == {"small_batch"} +def test_passes_filters_accepts_reportable_budget_limited_runs(): + study = StudySpec( + study_id="study", + hypotheses=[], + run_groups=[], + reportability_filter=True, + ) + payload = { + "group_id": "group", + "summary": { + "reportability": { + "is_reportable": True, + "budget_limited_decision_present": True, + } + }, + "manifest": Manifest( + experiment_id="exp", + run_id="run_001", + created_at_utc=datetime.now(timezone.utc), + environment=capture_environment_metadata("."), + invocation=capture_invocation_metadata("pytest"), + artifact_files=[], + ), + "run_labels": {}, + } + + accepted, reason = _passes_filters(payload, study) + + assert accepted is True + assert reason is None + + +def test_format_filter_diagnostics_mentions_seed_and_repeat_exclusions(): + message = _format_filter_diagnostics( + [ + { + "group_id": "gemm_representative", + "candidate_run_count": 3, + "matched_run_count": 0, + "excluded_reportability": 0, + "excluded_environment": 0, + "excluded_kernel_family": 0, + "excluded_selector_version": 0, + "excluded_selector_revision_id": 0, + "excluded_counter_set_id": 0, + "excluded_budget_id": 0, + "excluded_execution_mode": 0, + "excluded_seed": 2, + "excluded_repeat_index": 1, + } + ] + ) + + assert "gemm_representative" in message + assert "seed=2" in message + assert "repeat_index=1" in message + + +def test_load_run_payload_prefers_run_local_experiment_spec_over_manifest_path(tmp_path): + original_spec = load_experiment_spec(Path("configs/experiments/gemm_reportable.yaml")) + mutated_spec = original_spec.model_copy(deep=True) + mutated_spec.seed = 43 + store = RunStore(tmp_path / "artifacts", mutated_spec.experiment_id, "run_payload") + manifest = Manifest( + experiment_id=mutated_spec.experiment_id, + run_id="run_payload", + created_at_utc=datetime.now(timezone.utc), + environment=capture_environment_metadata("."), + invocation=capture_invocation_metadata( + "pytest", + experiment_config_path=str(Path("configs/experiments/gemm_reportable.yaml").resolve()), + ), + artifact_files=[], + ) + store.initialize_manifest(manifest) + store.write_experiment_spec(mutated_spec) + (store.run_dir / "summary.json").write_text( + '{"run_id":"run_payload","reportability":{"is_reportable":true}}', + encoding="utf-8", + ) + pd.DataFrame( + [{"strategy_id": "prune_rank", "selected_config_id": "cfg_a"}] + ).to_parquet(store.run_dir / "selection_decisions.parquet", index=False) + pd.DataFrame( + [ + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": mutated_spec.shapes[0].shape_id, + "strategy_id": "prune_rank", + "latency_median_us": 9.0, + "config_id": "cfg_a", + } + ] + ).to_parquet(store.run_dir / "runtime_measurements.parquet", index=False) + pd.DataFrame( + [ + { + "shape_id": mutated_spec.shapes[0].shape_id, + "workload_class": mutated_spec.shapes[0].workload_class, + "strategy_id": "prune_rank", + "latency_median_us": 9.0, + "winner_on_shape": True, + } + ] + ).to_csv(store.run_dir / "held_out_per_shape.csv", index=False) + + payload = _load_run_payload(store.run_dir, "group") + + assert payload["experiment_spec"].seed == 43 + + def test_evaluate_hypotheses_uses_clause_based_metrics(): strategy_rows = pd.DataFrame( [ diff --git a/tests/unit/test_profiling.py b/tests/unit/test_profiling.py index 3dd4faf..3b28683 100644 --- a/tests/unit/test_profiling.py +++ b/tests/unit/test_profiling.py @@ -1,8 +1,12 @@ +from datetime import datetime, timezone from pathlib import Path import subprocess +import pandas as pd + from kernel_tuner.common.config import load_counter_set, load_experiment_spec -from kernel_tuner.common.schema import CandidateConfig, ProfileMeasurement, ProfileStatus +from kernel_tuner.common.provenance import capture_environment_metadata, capture_invocation_metadata +from kernel_tuner.common.schema import CandidateConfig, Manifest, ProfileMeasurement, ProfileStatus from kernel_tuner.experiments.orchestrator import _profile_shapes from kernel_tuner.profiling.adapter import ProfileOutcome, _parse_counter_map, profile_candidate, profile_experiment from kernel_tuner.profiling.compatibility import validate_counter_set @@ -57,8 +61,11 @@ def test_profile_experiment_prefers_valid_candidate(monkeypatch): ) monkeypatch.setattr( - "kernel_tuner.profiling.adapter.generate_candidate_records", - lambda *args, **kwargs: [invalid, valid], + "kernel_tuner.profiling.adapter.generate_candidate_bundle", + lambda *args, **kwargs: { + "records": [invalid, valid], + "metadata": {}, + }, ) monkeypatch.setattr( "kernel_tuner.experiments.orchestrator._shape_split", @@ -167,3 +174,49 @@ def test_profile_candidate_marks_ambiguous_kernel_attribution_as_no_profile_data assert outcome.measurement.profile_status == ProfileStatus.NO_PROFILE_DATA assert outcome.measurement.profiler_metadata["kernel_attribution_status"] == "regex_ambiguous" + + +def test_profile_candidate_records_missing_counter_failure_reason(monkeypatch): + experiment_spec = load_experiment_spec(Path("configs/experiments/gemm_reportable.yaml")) + counter_set = load_counter_set(Path("configs/counters/compute_lite.yaml")) + shape = experiment_spec.shapes[0] + candidate = CandidateConfig( + experiment_id=experiment_spec.experiment_id, + kernel_id=experiment_spec.kernels[0], + shape_id=shape.shape_id, + config_id="cfg_valid", + config={"block_m": 128}, + is_valid=True, + ) + + stdout = "\n".join( + [ + '"ID","Process ID","Process Name","Host Name","Kernel Name","gpu__time_duration.sum","sm__warps_active.avg.pct_of_peak_sustained_active"', + '1,10,"python","host","matmul_kernel",5.0,1.0', + ] + ) + + monkeypatch.setattr( + subprocess, + "run", + lambda *args, **kwargs: subprocess.CompletedProcess( + args=args[0], + returncode=0, + stdout=stdout, + stderr="", + ), + ) + + outcome = profile_candidate( + run_id="run_001", + strategy_id="prune_rank_profiled", + kernel_id=experiment_spec.kernels[0], + shape=shape, + candidate=candidate, + counter_set=counter_set, + experiment_spec=experiment_spec, + ) + + assert outcome.measurement.profile_status == ProfileStatus.UNSUPPORTED_COUNTER + assert outcome.measurement.profiler_metadata["failure_reason"] == "missing_counter_values" + assert "missing or unsupported counters" in (outcome.measurement.notes or "") diff --git a/tests/unit/test_profiling_compatibility.py b/tests/unit/test_profiling_compatibility.py index 442920d..672c100 100644 --- a/tests/unit/test_profiling_compatibility.py +++ b/tests/unit/test_profiling_compatibility.py @@ -1,7 +1,9 @@ +import os from pathlib import Path from subprocess import CompletedProcess from kernel_tuner.common.config import load_counter_set +from kernel_tuner.common.provenance import resolve_tool_path from kernel_tuner.profiling.compatibility import validate_counter_set @@ -44,3 +46,50 @@ def test_shared_diag_uses_queryable_suffix_metrics(): "l1tex__data_pipe_lsu_wavefronts_mem_shared.avg", "smsp__warp_issue_stalled_short_scoreboard_per_warp_active.pct", ] + + +def test_default_calibration_uses_queryable_suffix_metrics(): + counter_set = load_counter_set(Path("configs/counters/default_calibration.yaml")) + + assert counter_set.counters == [ + "smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct", + "sm__warps_active.avg.pct_of_peak_sustained_active", + "smsp__inst_executed.sum", + ] + + +def test_resolve_tool_path_prefers_cuda_home_bin(tmp_path, monkeypatch): + cuda_bin = tmp_path / "cuda" / "bin" + cuda_bin.mkdir(parents=True) + ncu = cuda_bin / "ncu" + ncu.write_text("") + monkeypatch.setenv("CUDA_HOME", str(tmp_path / "cuda")) + monkeypatch.setattr("kernel_tuner.common.provenance.which", lambda name: None) + + assert resolve_tool_path("ncu") == str(ncu) + + +def test_validate_counter_set_records_missing_metrics_in_notes(monkeypatch): + counter_set = load_counter_set(Path("configs/counters/compute_lite.yaml")) + + monkeypatch.setattr( + "kernel_tuner.profiling.compatibility.subprocess.run", + lambda *args, **kwargs: CompletedProcess( + args=args[0], + returncode=0, + stdout="\n".join( + [ + "sm__warps_active", + "smsp__inst_executed", + ] + ), + stderr="", + ), + ) + + record = validate_counter_set(counter_set, kernel_family="gemm") + + assert record.acceptable is False + assert record.validation_backend == "ncu_query_metrics" + assert "missing or unsupported queried metrics" in (record.notes or "") + assert "smsp__inst_executed_pipe_tensor_op_hmma.avg" in (record.notes or "") diff --git a/tests/unit/test_reporting.py b/tests/unit/test_reporting.py index b8b5dc3..a2de84c 100644 --- a/tests/unit/test_reporting.py +++ b/tests/unit/test_reporting.py @@ -1,6 +1,6 @@ +import math from datetime import datetime, timezone from pathlib import Path -import math import pandas as pd import pytest @@ -56,16 +56,50 @@ def _write_minimal_required_tables(store: RunStore, *, include_profile: bool = F def test_pairwise_speedups_uses_geometric_mean_of_per_shape_ratios(): strategy_metrics = pd.DataFrame( [ - {"strategy_id": "default_config", "held_out_latency_mean_us": 15.0, "held_out_throughput_mean": 1.0, "held_out_measurements": 2}, - {"strategy_id": "candidate", "held_out_latency_mean_us": 15.0, "held_out_throughput_mean": 1.0, "held_out_measurements": 2}, + { + "strategy_id": "default_config", + "held_out_latency_mean_us": 15.0, + "held_out_throughput_mean": 1.0, + "held_out_measurements": 2, + }, + { + "strategy_id": "candidate", + "held_out_latency_mean_us": 15.0, + "held_out_throughput_mean": 1.0, + "held_out_measurements": 2, + }, ] ) runtime_measurements = pd.DataFrame( [ - {"measurement_phase": "held_out", "status": "success", "shape_id": "shape_a", "strategy_id": "default_config", "latency_median_us": 10.0}, - {"measurement_phase": "held_out", "status": "success", "shape_id": "shape_b", "strategy_id": "default_config", "latency_median_us": 20.0}, - {"measurement_phase": "held_out", "status": "success", "shape_id": "shape_a", "strategy_id": "candidate", "latency_median_us": 5.0}, - {"measurement_phase": "held_out", "status": "success", "shape_id": "shape_b", "strategy_id": "candidate", "latency_median_us": 30.0}, + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": "shape_a", + "strategy_id": "default_config", + "latency_median_us": 10.0, + }, + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": "shape_b", + "strategy_id": "default_config", + "latency_median_us": 20.0, + }, + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": "shape_a", + "strategy_id": "candidate", + "latency_median_us": 5.0, + }, + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": "shape_b", + "strategy_id": "candidate", + "latency_median_us": 30.0, + }, ] ) @@ -105,7 +139,32 @@ def test_summarize_run_falls_back_to_manifest_experiment_config(tmp_path): assert summary["run_id"] == "run_001" -def test_summarize_run_marks_budget_limited_runs_non_reportable(tmp_path): +def test_summarize_run_prefers_run_local_experiment_spec_over_manifest_path(tmp_path): + original_spec = load_experiment_spec(Path("configs/experiments/gemm_reportable.yaml")) + mutated_spec = original_spec.model_copy(deep=True) + mutated_spec.seed = 19 + store = RunStore(tmp_path / "artifacts", "test_experiment", "run_local_spec") + manifest = Manifest( + experiment_id="test_experiment", + run_id="run_local_spec", + created_at_utc=datetime.now(timezone.utc), + environment=capture_environment_metadata("."), + invocation=capture_invocation_metadata( + "pytest", + experiment_config_path=str(Path("configs/experiments/gemm_reportable.yaml").resolve()), + ), + artifact_files=[], + ) + store.initialize_manifest(manifest) + store.write_experiment_spec(mutated_spec) + _write_minimal_required_tables(store, include_profile=True) + + summary = summarize_run(store.run_dir) + + assert summary["aggregate_metrics"]["seed"] == 19 + + +def test_summarize_run_allows_budget_limited_runs_when_other_reportability_requirements_pass(tmp_path): experiment_spec = load_experiment_spec(Path("configs/experiments/gemm_reportable.yaml")) store = RunStore(tmp_path, experiment_spec.experiment_id, "run_001") manifest = Manifest( @@ -136,6 +195,105 @@ def test_summarize_run_marks_budget_limited_runs_non_reportable(tmp_path): } ] ).to_parquet(store.run_dir / "compile_signals.parquet", index=False) + held_out_shape_ids = [ + experiment_spec.shapes[0].shape_id, + experiment_spec.shapes[3].shape_id, + experiment_spec.shapes[6].shape_id, + experiment_spec.shapes[9].shape_id, + ] + runtime_rows = [] + for idx, shape_id in enumerate(held_out_shape_ids): + runtime_rows.extend( + [ + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": shape_id, + "strategy_id": "default_config", + "latency_median_us": 10.0 + idx, + "throughput_value": 1.0, + "config_id": "cfg_default", + }, + { + "measurement_phase": "held_out", + "status": "success", + "shape_id": shape_id, + "strategy_id": "prune_rank", + "latency_median_us": 9.0 + idx, + "throughput_value": 1.0, + "config_id": "cfg_a", + }, + ] + ) + pd.DataFrame(runtime_rows).to_parquet(store.run_dir / "runtime_measurements.parquet", index=False) + pd.DataFrame( + [ + { + "strategy_id": "prune_rank", + "counter_set_id": experiment_spec.counter_set_id, + "profile_status": "success", + "counter_map": ( + '{"sm__warps_active.avg.pct_of_peak_sustained_active": 1.0, ' + '"smsp__inst_executed.sum": 2.0, ' + '"smsp__inst_executed_pipe_tensor_op_hmma.avg": 3.0, ' + '"smsp__pipe_tensor_op_hmma_cycles_active.avg": 4.0, ' + '"smsp__warp_issue_stalled_math_pipe_throttle_per_warp_active.pct": 5.0, ' + '"smsp__warp_issue_stalled_long_scoreboard_per_warp_active.pct": 6.0}' + ), + "config_id": "cfg_a", + } + ] + ).to_parquet(store.run_dir / "profile_measurements.parquet", index=False) + pd.DataFrame( + [ + { + "strategy_id": "default_config", + "selected_config_id": "cfg_default", + "benchmarks_requested": 1, + "profiles_requested": 0, + "decision_status": "selected", + "comparison_class": "matched_budget", + }, + { + "strategy_id": "prune_rank", + "selected_config_id": "cfg_a", + "benchmarks_requested": 1, + "profiles_requested": 1, + "decision_status": "selected_budget_limited", + "comparison_class": "matched_budget", + }, + ] + ).to_parquet(store.run_dir / "selection_decisions.parquet", index=False) + (store.run_dir / "counter_compatibility.json").write_text( + '{"acceptable": true, "counter_set_id": "compute_lite"}', + encoding="utf-8", + ) + + summary = summarize_run(store.run_dir) + + assert summary["reportability"]["is_reportable"] is True + assert summary["reportability"]["comparison_class"] == "matched_budget" + assert summary["reportability"]["budget_limited_decision_present"] is True + + +def test_summarize_run_marks_mixed_comparison_class_runs_non_reportable(tmp_path): + experiment_spec = load_experiment_spec(Path("configs/experiments/gemm_reportable.yaml")) + store = RunStore(tmp_path, experiment_spec.experiment_id, "run_mixed") + manifest = Manifest( + experiment_id=experiment_spec.experiment_id, + run_id="run_mixed", + created_at_utc=datetime.now(timezone.utc), + environment=capture_environment_metadata("."), + invocation=capture_invocation_metadata( + "pytest", + experiment_config_path=str(Path("configs/experiments/gemm_reportable.yaml").resolve()), + ), + artifact_files=[], + ) + store.initialize_manifest(manifest) + store.write_experiment_spec(experiment_spec) + + _write_minimal_required_tables(store, include_profile=True) pd.DataFrame( [ { @@ -158,17 +316,6 @@ def test_summarize_run_marks_budget_limited_runs_non_reportable(tmp_path): }, ] ).to_parquet(store.run_dir / "runtime_measurements.parquet", index=False) - pd.DataFrame( - [ - { - "strategy_id": "prune_rank", - "counter_set_id": experiment_spec.counter_set_id, - "profile_status": "success", - "counter_map": '{"sm__warps_active.avg.pct_of_peak_sustained_active": 1.0}', - "config_id": "cfg_a", - } - ] - ).to_parquet(store.run_dir / "profile_measurements.parquet", index=False) pd.DataFrame( [ { @@ -184,8 +331,8 @@ def test_summarize_run_marks_budget_limited_runs_non_reportable(tmp_path): "selected_config_id": "cfg_a", "benchmarks_requested": 1, "profiles_requested": 1, - "decision_status": "selected_budget_limited", - "comparison_class": "matched_budget", + "decision_status": "selected", + "comparison_class": "non_comparable", }, ] ).to_parquet(store.run_dir / "selection_decisions.parquet", index=False) @@ -197,9 +344,7 @@ def test_summarize_run_marks_budget_limited_runs_non_reportable(tmp_path): summary = summarize_run(store.run_dir) assert summary["reportability"]["is_reportable"] is False - assert summary["reportability"]["budget_limited_decision_present"] is True - - + assert summary["reportability"]["comparison_class"] == "non_comparable" def test_summarize_run_marks_counter_set_unaccepted_when_compatibility_fails(tmp_path): experiment_path = Path("configs/experiments/gemm_reportable.yaml").resolve() experiment_spec = load_experiment_spec(experiment_path)