Summary
Held-out evaluation is performed sequentially by strategy immediately after each strategy finishes calibration, rather than in a paired or alternating order across the competing final configurations.
Evidence
src/kernel_tuner/experiments/orchestrator.py:576
src/kernel_tuner/experiments/orchestrator.py:589
src/kernel_tuner/experiments/orchestrator.py:618
docs/04_experiment_protocol.md:116
Why This Matters
Sequential held-out evaluation is exposed to time-order effects such as thermal drift, cache state, and other host/GPU noise. That weakens the fairness of final comparisons.
Expected Fix
- Rework held-out evaluation to benchmark competing selected configs in a paired or alternating order on the same held-out shapes.
- Record the held-out measurement order explicitly.
- Add tests for ordering behavior where feasible.
Acceptance Criteria
- Held-out evaluation uses a shared ordering policy across compared strategies.
- Measurement ordering is persisted for auditability.
- The protocol and implementation match.
Summary
Held-out evaluation is performed sequentially by strategy immediately after each strategy finishes calibration, rather than in a paired or alternating order across the competing final configurations.
Evidence
src/kernel_tuner/experiments/orchestrator.py:576src/kernel_tuner/experiments/orchestrator.py:589src/kernel_tuner/experiments/orchestrator.py:618docs/04_experiment_protocol.md:116Why This Matters
Sequential held-out evaluation is exposed to time-order effects such as thermal drift, cache state, and other host/GPU noise. That weakens the fairness of final comparisons.
Expected Fix
Acceptance Criteria