Skip to content

Scope wall-clock selection budgets per strategy instead of per run #3

Description

@lullu57

Summary

wall_clock_limit_s starts once for the whole run, before compile-signal collection, and is then shared by all strategy brokers. Later strategies can be penalized because earlier strategies or pre-selection work consumed the clock.

Evidence

  • src/kernel_tuner/experiments/orchestrator.py:318
  • src/kernel_tuner/experiments/orchestrator.py:522
  • src/kernel_tuner/experiments/orchestrator.py:561
  • src/kernel_tuner/experiments/orchestrator.py:603
  • docs/04_experiment_protocol.md:45

Why This Matters

The protocol defines this as a strategy selection budget. Sharing one wall-clock deadline across the entire run makes cross-strategy comparisons unfair.

Expected Fix

  • Scope wall-clock accounting to each strategy decision, not the entire run.
  • Exclude pre-selection work like compile-signal collection from per-strategy selection deadlines unless explicitly documented otherwise.
  • Preserve explicit skipped_budget records when a strategy genuinely runs out of time.

Acceptance Criteria

  • Each selector/baseline starts with the same wall-clock selection budget.
  • Pre-selection phases do not silently consume strategy budget.
  • Tests prove one strategy cannot starve later strategies of time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions