Summary
SelectionBudget.max_benchmarks and SelectionBudget.max_profiles are currently enforced per config_id, but each broker request expands across multiple calibration/profile shapes. This makes the recorded budget usage smaller than the real number of measurements and breaks the matched-budget contract.
Evidence
src/kernel_tuner/selector/engine.py:560
src/kernel_tuner/baselines/strategies.py:143
src/kernel_tuner/experiments/orchestrator.py:327
docs/04_experiment_protocol.md:42
docs/04_experiment_protocol.md:138
Why This Matters
The paper framing depends on fair matched-budget comparisons. Right now a budget of K can become K * num_shapes actual runtime or profile measurements while the system still reports only K.
Expected Fix
- Redefine broker request interfaces or accounting so benchmark/profile budgets are consumed per measurement, not per config.
- Ensure selector and baselines consume budgets under identical semantics.
- Update reported
benchmarks_requested and profiles_requested to match real measurement counts.
- Add tests that fail if one config request silently expands beyond the budget.
Acceptance Criteria
- A runtime benchmark request for one config over
N calibration shapes consumes N benchmark budget units.
- A profile request for one config over
N selected profile shapes consumes N profile budget units.
- Decision artifacts report true budget usage.
- Tests cover selector and baseline paths.
Summary
SelectionBudget.max_benchmarksandSelectionBudget.max_profilesare currently enforced perconfig_id, but each broker request expands across multiple calibration/profile shapes. This makes the recorded budget usage smaller than the real number of measurements and breaks the matched-budget contract.Evidence
src/kernel_tuner/selector/engine.py:560src/kernel_tuner/baselines/strategies.py:143src/kernel_tuner/experiments/orchestrator.py:327docs/04_experiment_protocol.md:42docs/04_experiment_protocol.md:138Why This Matters
The paper framing depends on fair matched-budget comparisons. Right now a budget of
Kcan becomeK * num_shapesactual runtime or profile measurements while the system still reports onlyK.Expected Fix
benchmarks_requestedandprofiles_requestedto match real measurement counts.Acceptance Criteria
Ncalibration shapes consumesNbenchmark budget units.Nselected profile shapes consumesNprofile budget units.