Summary
The benchmark harness only catches exceptions from the first untimed launch. Failures during reference generation, output validation, warmup, or the timed loop can escape and abort the experiment instead of being persisted as non-success measurements. Early untimed failures are also all labeled compile_failed, even when they are really runtime faults.
Evidence
src/kernel_tuner/benchmark/harness.py:103
src/kernel_tuner/benchmark/harness.py:122
src/kernel_tuner/benchmark/harness.py:141
src/kernel_tuner/benchmark/harness.py:116
docs/specs/benchmark_harness.md:74
docs/04_experiment_protocol.md:103
Why This Matters
Bad measurements need to be recorded, not dropped. Otherwise failure analysis and candidate accounting become incomplete, and compile/runtime failure modes are blurred.
Expected Fix
- Catch and classify failures across the full benchmark lifecycle.
- Distinguish compile-time failures from runtime launch/execution failures where possible.
- Persist explicit status records instead of aborting the whole experiment.
- Extend tests around harness failure handling.
Acceptance Criteria
- Failures in correctness, warmup, and timed execution are persisted as explicit non-success measurements.
- Runtime faults are not mislabeled as compile failures.
- A single bad config does not abort the run.
Summary
The benchmark harness only catches exceptions from the first untimed launch. Failures during reference generation, output validation, warmup, or the timed loop can escape and abort the experiment instead of being persisted as non-success measurements. Early untimed failures are also all labeled
compile_failed, even when they are really runtime faults.Evidence
src/kernel_tuner/benchmark/harness.py:103src/kernel_tuner/benchmark/harness.py:122src/kernel_tuner/benchmark/harness.py:141src/kernel_tuner/benchmark/harness.py:116docs/specs/benchmark_harness.md:74docs/04_experiment_protocol.md:103Why This Matters
Bad measurements need to be recorded, not dropped. Otherwise failure analysis and candidate accounting become incomplete, and compile/runtime failure modes are blurred.
Expected Fix
Acceptance Criteria