feat(experiments): test-case-weighted rollups (aggregate per test case before pooling)

[shanaiabuggy]

What

Two-level ("test-case-weighted") rollups for the Experiments leaderboard — ASE-561.

When a test case is run k times, metrics are now rolled up per test case first (pass@1 = mean over its attempts), then aggregated across test cases — so a heavily-retried test case counts once instead of dominating the pool. This also corrects medians/percentiles, which were previously computed over the wrong distribution (all attempts vs. per-test-case values).

Changes

Backend (evaluation_rollup_repository.py)

• Group key = test_case_id when set, else session_id — so unlabeled runs pool per-attempt exactly like the old flat rollup (no empty-string bucket collapse), and there's no hard requirement on test_case_id.
• Scores: pass@1 (mean over a test case's attempts) → distribution across test cases.
• Cost/latency: avg per attempt within a test case (so the number doesn't scale with k) → distribution across test cases; attempts missing a value are excluded, not counted as zero.
• New test_case_count rollup field (distinct test cases) on the evaluation response, sortable.

Frontend (Studio)

• Leaderboard "Run Count" → "Total test cases" (test_case_count), sortable.
• Avg-cell tooltip now states the calculation transparently, e.g. Mean cost over 2 test cases — each test case averaged over its attempts.

SDK regenerated for the new test_case_count field.

Testing

• Unit + integration (real ClickHouse), including an unbalanced-k case that proves test-case-weighting (score mean 0.5, not the pooled 0.75; cost avg-per-attempt) and a missing-test_case_id degrade case (pools per attempt).

Note

run_count (total attempts) stays on the API and still feeds the tooltip's context; only the leaderboard column moved to test cases.

🤖 Generated with Claude Code

Summary by CodeRabbit

Summary by CodeRabbit

• New Features

  • Added test_case_count to evaluation and rollup responses, exposed as a sortable “Total test cases” metric in experiment group tables.
  • Updated API sorting to support ordering by total test cases.

• Bug Fixes

  • Updated rollup aggregation to compute statistics using test-case-level weighting (not raw attempts).
  • Evaluations with no test_case_id no longer contribute to test-case-weighted metrics (test_case_count returns 0).

• Tests

  • Extended integration and repository tests to validate hydration, sorting support, and the new rollup semantics.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6b84e0bc-73e0-4228-8e71-b968fdd751c0

📥 Commits

Reviewing files that changed from the base of the PR and between 8ba9577 and 6d3aaee.

📒 Files selected for processing (3)

• services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py
• services/intake/tests/integration/spans/test_experiment_rollups.py
• services/intake/tests/test_experiment_rollup_repository.py

🚧 Files skipped from review as they are similar to previous changes (3)

• services/intake/tests/test_experiment_rollup_repository.py
• services/intake/tests/integration/spans/test_experiment_rollups.py
• services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py

---

📝 Walkthrough

Walkthrough

Evaluation rollups now expose test_case_count, aggregate scores and metrics per test case before pooling, support sorting and filtering by the new metric, and display test-case counts with updated weighted-average tooltip text in Studio.

Changes

Evaluation rollup metrics

Layer / File(s)	Summary
Test-case rollup aggregation services/intake/src/nmp/intake/spans/evaluation_rollup_repository.py	Rollup queries derive distinct test-case keys, aggregate scores and cost/latency per test case, and populate test_case_count.
Evaluation response and API metric handling services/intake/src/nmp/intake/api/v2/experiments/schemas.py, services/intake/src/nmp/intake/api/v2/experiments/endpoints.py, openapi/...	The response schema, OpenAPI documentation, metric validation, hydration, filtering, and sorting support test_case_count.
Rollup behavior validation services/intake/tests/integration/spans/test_experiment_rollups.py, services/intake/tests/test_experiment_rollup_repository.py	Tests validate test-case counts, weighted aggregation, missing identifiers, and generated SQL.
Experiment group table display web/packages/studio/src/components/dataViews/ExperimentGroupDataView/*	Studio replaces run-count sorting and display with test-case-count sorting and updates mean metric tooltip wording.

Sequence Diagram(s)

sequenceDiagram
  participant Client
  participant EvaluationAPI
  participant ClickHouse
  Client->>EvaluationAPI: Request evaluations sorted by test_case_count
  EvaluationAPI->>ClickHouse: Read evaluation rollup data
  ClickHouse-->>EvaluationAPI: Return test_case_count rollup
  EvaluationAPI-->>Client: Return sorted evaluation response

Loading

Possibly related PRs

• NVIDIA-NeMo/nemo-platform#714: Intersects with the same evaluation sort parsing and validation logic.

Suggested reviewers: briannewsom

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: test-case-weighted rollups aggregated per test case before pooling.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch sbuggy/ase-561

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	25556/32773	78.0%	62.6%
Integration Tests	14747/31422	46.9%	19.2%