Skip to content

Harden benchmark harness failure capture and classification #4

Description

@lullu57

Summary

The benchmark harness only catches exceptions from the first untimed launch. Failures during reference generation, output validation, warmup, or the timed loop can escape and abort the experiment instead of being persisted as non-success measurements. Early untimed failures are also all labeled compile_failed, even when they are really runtime faults.

Evidence

  • src/kernel_tuner/benchmark/harness.py:103
  • src/kernel_tuner/benchmark/harness.py:122
  • src/kernel_tuner/benchmark/harness.py:141
  • src/kernel_tuner/benchmark/harness.py:116
  • docs/specs/benchmark_harness.md:74
  • docs/04_experiment_protocol.md:103

Why This Matters

Bad measurements need to be recorded, not dropped. Otherwise failure analysis and candidate accounting become incomplete, and compile/runtime failure modes are blurred.

Expected Fix

  • Catch and classify failures across the full benchmark lifecycle.
  • Distinguish compile-time failures from runtime launch/execution failures where possible.
  • Persist explicit status records instead of aborting the whole experiment.
  • Extend tests around harness failure handling.

Acceptance Criteria

  • Failures in correctness, warmup, and timed execution are persisted as explicit non-success measurements.
  • Runtime faults are not mislabeled as compile failures.
  • A single bad config does not abort the run.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions