feat: add Result.filter() for dataset filtering#87
Open
acere wants to merge 1 commit into
Open
Conversation
…riteria Add a filter() method to the Result class that returns a new Result containing only responses matching given criteria. Supports: - errors: filter by success/failure - cached: filter by prompt cache usage - min/max_tokens_output: output token bounds - min/max_tokens_input: input token bounds - min/max_ttft: time to first token bounds - min/max_ttlt: time to last token bounds - has_response_text: filter by presence of response text - predicate: arbitrary callable for custom conditions All criteria combine with AND logic. The filtered Result preserves metadata and recomputes stats from the filtered responses. Closes awslabs#86
Collaborator
|
IMO for this to be especially useful, it'd need to go materially beyond just replacing a I agree that re-computing Result stats and metadata fields based on the filtering seems like a rabbit-hole - especially when we consider stuff like Callbacks contributing arbitrary stats. However, if we think there's too much potential confusion or magic in tackling that, then maybe the filter method should just return the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
filter()method toResultthat returns a newResultcontaining only responses matching specified criteria. This enables users to generate datasets from benchmark runs that match specific conditions without manual list comprehensions.Usage
Supported Filters
errorsTrue= only errors,False= only successescachedTrue= only cache hits,False= no cachemin/max_tokens_outputmin/max_tokens_inputmin/max_ttftmin/max_ttlthas_response_textpredicateDesign Decisions
Result(immutable pattern) — original is unchangedtotal_requestsis updated to reflect filtered countoutput_pathis set toNoneon filtered results to avoid accidental overwritesNonevalues for a numeric field are excluded when that field's bounds are specifiedTesting
Verified with existing test suite (
pytest tests/unit/test_results.py— all 32 tests pass) and manual smoke tests covering all filter combinations.Closes #86