Skip to content

Make held-out evaluation paired or alternating across strategies #5

Description

@lullu57

Summary

Held-out evaluation is performed sequentially by strategy immediately after each strategy finishes calibration, rather than in a paired or alternating order across the competing final configurations.

Evidence

  • src/kernel_tuner/experiments/orchestrator.py:576
  • src/kernel_tuner/experiments/orchestrator.py:589
  • src/kernel_tuner/experiments/orchestrator.py:618
  • docs/04_experiment_protocol.md:116

Why This Matters

Sequential held-out evaluation is exposed to time-order effects such as thermal drift, cache state, and other host/GPU noise. That weakens the fairness of final comparisons.

Expected Fix

  • Rework held-out evaluation to benchmark competing selected configs in a paired or alternating order on the same held-out shapes.
  • Record the held-out measurement order explicitly.
  • Add tests for ordering behavior where feasible.

Acceptance Criteria

  • Held-out evaluation uses a shared ordering policy across compared strategies.
  • Measurement ordering is persisted for auditability.
  • The protocol and implementation match.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions