Summary
Profile-shape sampling and standalone profiling target selection are deterministic but not necessarily representative or valid for evidence. per_workload_class_top1 selects lexicographically smallest shape_id, and the standalone profiling path profiles the first generated candidate rather than a known compile-success/correctness-passing candidate.
Evidence
src/kernel_tuner/experiments/orchestrator.py:121
src/kernel_tuner/experiments/orchestrator.py:130
src/kernel_tuner/experiments/orchestrator.py:136
src/kernel_tuner/profiling/adapter.py:252
src/kernel_tuner/profiling/adapter.py:257
docs/specs/profiling_adapter.md:63
Why This Matters
Tier 1 profiling is supposed to be matched-budget evidence on a calibration subset. ID-order shape selection and arbitrary standalone target selection can bias or weaken what the counters actually mean.
Expected Fix
- Make profile-shape selection explicitly representative or at least clearly documented as deterministic-only.
- Ensure standalone profiling targets only compile-success/correctness-passing candidates.
- Add tests around profile-shape and standalone-target selection.
Acceptance Criteria
- Profile-shape selection policy is explicit and defensible.
- Standalone profiling never targets an arbitrary invalid candidate.
- Tests cover both behaviors.
Summary
Profile-shape sampling and standalone profiling target selection are deterministic but not necessarily representative or valid for evidence.
per_workload_class_top1selects lexicographically smallestshape_id, and the standalone profiling path profiles the first generated candidate rather than a known compile-success/correctness-passing candidate.Evidence
src/kernel_tuner/experiments/orchestrator.py:121src/kernel_tuner/experiments/orchestrator.py:130src/kernel_tuner/experiments/orchestrator.py:136src/kernel_tuner/profiling/adapter.py:252src/kernel_tuner/profiling/adapter.py:257docs/specs/profiling_adapter.md:63Why This Matters
Tier 1 profiling is supposed to be matched-budget evidence on a calibration subset. ID-order shape selection and arbitrary standalone target selection can bias or weaken what the counters actually mean.
Expected Fix
Acceptance Criteria