Skip to content

feat: add Result.filter() for dataset filtering#87

Open
acere wants to merge 1 commit into
awslabs:mainfrom
acere:feat/result-filter
Open

feat: add Result.filter() for dataset filtering#87
acere wants to merge 1 commit into
awslabs:mainfrom
acere:feat/result-filter

Conversation

@acere

@acere acere commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a filter() method to Result that returns a new Result containing only responses matching specified criteria. This enables users to generate datasets from benchmark runs that match specific conditions without manual list comprehensions.

Usage

# Remove error responses
clean = result.filter(errors=False)

# Only cached responses  
cached_only = result.filter(cached=True)

# Combine criteria (AND logic)
fast_cached = result.filter(cached=True, errors=False, max_ttlt=1.0)

# Custom predicate
reasoning = result.filter(
    predicate=lambda r: r.num_tokens_output_reasoning is not None
)

Supported Filters

Parameter Description
errors True = only errors, False = only successes
cached True = only cache hits, False = no cache
min/max_tokens_output Output token count bounds
min/max_tokens_input Input token count bounds
min/max_ttft Time to first token bounds
min/max_ttlt Time to last token bounds
has_response_text Filter by presence of response text
predicate Arbitrary callable for custom conditions

Design Decisions

  • Returns a new Result (immutable pattern) — original is unchanged
  • Metadata (model_id, provider, run_name, etc.) is preserved
  • total_requests is updated to reflect filtered count
  • Stats are recomputed from the filtered responses
  • output_path is set to None on filtered results to avoid accidental overwrites
  • Responses with None values for a numeric field are excluded when that field's bounds are specified

Testing

Verified with existing test suite (pytest tests/unit/test_results.py — all 32 tests pass) and manual smoke tests covering all filter combinations.

Closes #86

…riteria

Add a filter() method to the Result class that returns a new Result
containing only responses matching given criteria. Supports:

- errors: filter by success/failure
- cached: filter by prompt cache usage
- min/max_tokens_output: output token bounds
- min/max_tokens_input: input token bounds
- min/max_ttft: time to first token bounds
- min/max_ttlt: time to last token bounds
- has_response_text: filter by presence of response text
- predicate: arbitrary callable for custom conditions

All criteria combine with AND logic. The filtered Result preserves
metadata and recomputes stats from the filtered responses.

Closes awslabs#86
@athewsey

Copy link
Copy Markdown
Collaborator

IMO for this to be especially useful, it'd need to go materially beyond just replacing a result.responses list comprehension that users could write themselves... Perhaps the real underlying problem could be that too many of our stats/analysis methods need a Result instead of being able to operate on just a list of responses?

I agree that re-computing Result stats and metadata fields based on the filtering seems like a rabbit-hole - especially when we consider stuff like Callbacks contributing arbitrary stats. However, if we think there's too much potential confusion or magic in tackling that, then maybe the filter method should just return the filtered responses list - if you think that's still a nice convenience method?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add result filtering to generate datasets matching specific criteria

2 participants