Summary
Run-level reportability and study-level filtering are too loose. Study comparison does not enforce run-group workload scoping, and summary.json reportability is weaker than the documented contract while still being used as the admission gate for study aggregation.
Evidence
src/kernel_tuner/analysis/reporting.py:392
src/kernel_tuner/analysis/reporting.py:403
src/kernel_tuner/analysis/comparison.py:232
src/kernel_tuner/analysis/comparison.py:291
configs/studies/validation_phase.yaml:120
docs/specs/analysis_and_reporting.md:79
Why This Matters
Study-level evidence can silently mix workload classes or trust a run summary that did not fully re-check reportability requirements. That weakens paper-facing conclusions.
Expected Fix
- Enforce run-group workload scoping in study comparison.
- Tighten run-level reportability checks to match the documented contract.
- Make study admission depend on the stronger contract, not a loose boolean.
Acceptance Criteria
- Study run-groups respect configured workload scoping.
is_reportable reflects the documented policy, not a reduced subset of checks.
- Tests cover study filtering and reportability enforcement.
Summary
Run-level reportability and study-level filtering are too loose. Study comparison does not enforce run-group workload scoping, and
summary.jsonreportability is weaker than the documented contract while still being used as the admission gate for study aggregation.Evidence
src/kernel_tuner/analysis/reporting.py:392src/kernel_tuner/analysis/reporting.py:403src/kernel_tuner/analysis/comparison.py:232src/kernel_tuner/analysis/comparison.py:291configs/studies/validation_phase.yaml:120docs/specs/analysis_and_reporting.md:79Why This Matters
Study-level evidence can silently mix workload classes or trust a run summary that did not fully re-check reportability requirements. That weakens paper-facing conclusions.
Expected Fix
Acceptance Criteria
is_reportablereflects the documented policy, not a reduced subset of checks.