Skip to content

Fix matched-budget accounting for benchmark and profile measurements #2

Description

@lullu57

Summary

SelectionBudget.max_benchmarks and SelectionBudget.max_profiles are currently enforced per config_id, but each broker request expands across multiple calibration/profile shapes. This makes the recorded budget usage smaller than the real number of measurements and breaks the matched-budget contract.

Evidence

  • src/kernel_tuner/selector/engine.py:560
  • src/kernel_tuner/baselines/strategies.py:143
  • src/kernel_tuner/experiments/orchestrator.py:327
  • docs/04_experiment_protocol.md:42
  • docs/04_experiment_protocol.md:138

Why This Matters

The paper framing depends on fair matched-budget comparisons. Right now a budget of K can become K * num_shapes actual runtime or profile measurements while the system still reports only K.

Expected Fix

  • Redefine broker request interfaces or accounting so benchmark/profile budgets are consumed per measurement, not per config.
  • Ensure selector and baselines consume budgets under identical semantics.
  • Update reported benchmarks_requested and profiles_requested to match real measurement counts.
  • Add tests that fail if one config request silently expands beyond the budget.

Acceptance Criteria

  • A runtime benchmark request for one config over N calibration shapes consumes N benchmark budget units.
  • A profile request for one config over N selected profile shapes consumes N profile budget units.
  • Decision artifacts report true budget usage.
  • Tests cover selector and baseline paths.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions