Skip to content

Make profile-shape and standalone profiling target selection defensible #8

Description

@lullu57

Summary

Profile-shape sampling and standalone profiling target selection are deterministic but not necessarily representative or valid for evidence. per_workload_class_top1 selects lexicographically smallest shape_id, and the standalone profiling path profiles the first generated candidate rather than a known compile-success/correctness-passing candidate.

Evidence

  • src/kernel_tuner/experiments/orchestrator.py:121
  • src/kernel_tuner/experiments/orchestrator.py:130
  • src/kernel_tuner/experiments/orchestrator.py:136
  • src/kernel_tuner/profiling/adapter.py:252
  • src/kernel_tuner/profiling/adapter.py:257
  • docs/specs/profiling_adapter.md:63

Why This Matters

Tier 1 profiling is supposed to be matched-budget evidence on a calibration subset. ID-order shape selection and arbitrary standalone target selection can bias or weaken what the counters actually mean.

Expected Fix

  • Make profile-shape selection explicitly representative or at least clearly documented as deterministic-only.
  • Ensure standalone profiling targets only compile-success/correctness-passing candidates.
  • Add tests around profile-shape and standalone-target selection.

Acceptance Criteria

  • Profile-shape selection policy is explicit and defensible.
  • Standalone profiling never targets an arbitrary invalid candidate.
  • Tests cover both behaviors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions