fix(review-pr): handle large PRs by filtering non-reviewable files#47
Conversation
|
Warning Review limit reached
More reviews will be available in 41 minutes and 36 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe Changesreview-pr Skill Overhaul
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
pr-review-toolkit/skills/review-pr/SKILL.md (2)
104-113: ⚡ Quick winClarify the dynamic jq filter construction for large reviewable sets.
Line 112 states "For large reviewable sets, build the jq filter from the reviewable filename list," but the mechanism is unspecified. For a reviewer or maintainer implementing this step, clarify the exact approach:
- How is the list of filenames converted into a jq filter condition? (e.g., concatenate into a chain of
or .filename == "...", use an array input, build a regex?)- Should the filter construction avoid exceeding any command-line length limits?
While this may be acceptable as an agent-level implementation detail, explicit guidance here would improve consistency and debuggability.
💡 Example clarification for jq filter construction
For large reviewable sets, construct a jq filter by: 1. Joining reviewable filenames with `" or "` and `.filename == "..."` e.g., `.filename == "file1" or .filename == "file2" or ...` 2. Wrapping the filter as: `jq '[.[] | select(<filter>) | {filename, patch, status}]' <saved-response-file>` 3. If the filter exceeds command-line limits, process pages sequentially or split the reviewable set into batches.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pr-review-toolkit/skills/review-pr/SKILL.md` around lines 104 - 113, The documentation in Step 3 of the SKILL.md file (the section describing jq filter construction for extracting file data) lacks concrete implementation details for handling large reviewable sets. Add explicit guidance explaining how to build the dynamic jq filter by constructing a select() condition that joins the reviewable filenames with "or .filename ==" operators. Additionally, include guidance on handling potential command-line length limits by either processing the reviewable set in batches or handling pages sequentially. This will clarify the mechanism referenced in the final sentence and provide clear, actionable steps for maintainers implementing this review workflow.
132-137: ⚡ Quick winDefine quantitative thresholds for deduplication matching.
Phase 3 deduplication (lines 132–137) references matching by file, "approximate line," and "similar description," but these criteria lack precise definitions:
- What constitutes an "approximate line" match? (e.g., ±5 lines, same line number only, ignore line numbers entirely?)
- What metric defines "similar description"? (e.g., string similarity threshold, keyword matching, semantic distance?)
Without explicit thresholds, different implementations of this step may deduplicate inconsistently, potentially missing duplicates or false-matching unrelated comments.
💡 Example deduplication criteria definition
### Deduplication criteria Two findings are considered duplicates if: 1. **File match**: Both reference the same file 2. **Line match**: Line numbers are within ±5 lines (or exact match preferred) 3. **Description match**: Description text shares > 70% character-level similarity or contains the same capitalized keywords (case-insensitive match on 5+ char words) Apply filters in order; all three must match to drop a finding as duplicate.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pr-review-toolkit/skills/review-pr/SKILL.md` around lines 132 - 137, The Phase 3 deduplication section lacks precise definitions for "approximate line" and "similar description" matching criteria. Define concrete quantitative thresholds for each deduplication component: specify the line number tolerance (e.g., ±5 lines or exact match), provide a measurable similarity metric for descriptions (e.g., character-level similarity percentage or keyword matching rules), and clearly state that all three criteria (file, line, description) must match before a finding is dropped as a duplicate. Replace the vague language in the get_review_comments and get_reviews sections with these explicit thresholds.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pr-review-toolkit/skills/review-pr/SKILL.md`:
- Around line 104-113: The documentation in Step 3 of the SKILL.md file (the
section describing jq filter construction for extracting file data) lacks
concrete implementation details for handling large reviewable sets. Add explicit
guidance explaining how to build the dynamic jq filter by constructing a
select() condition that joins the reviewable filenames with "or .filename =="
operators. Additionally, include guidance on handling potential command-line
length limits by either processing the reviewable set in batches or handling
pages sequentially. This will clarify the mechanism referenced in the final
sentence and provide clear, actionable steps for maintainers implementing this
review workflow.
- Around line 132-137: The Phase 3 deduplication section lacks precise
definitions for "approximate line" and "similar description" matching criteria.
Define concrete quantitative thresholds for each deduplication component:
specify the line number tolerance (e.g., ±5 lines or exact match), provide a
measurable similarity metric for descriptions (e.g., character-level similarity
percentage or keyword matching rules), and clearly state that all three criteria
(file, line, description) must match before a finding is dropped as a duplicate.
Replace the vague language in the get_review_comments and get_reviews sections
with these explicit thresholds.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 153c4568-3bfe-4dae-9a80-040af5046c9d
📒 Files selected for processing (3)
pr-review-toolkit/.claude-plugin/plugin.jsonpr-review-toolkit/skills/review-pr/SKILL.mdpr-review-toolkit/skills/review-pr/review-pr.js
cblecker
force-pushed
the
fix/pr-review-large-prs
branch
from
d8c558b to
bfe7e77
Compare
…fore fetching patches The review toolkit failed on PRs with hundreds of files because it tried to pass full patch content for every file through the workflow args, exceeding token limits. Split Phase 1 into a pipeline: discover filenames, categorize into vendor/generated/reviewable, then fetch merge-base patches only for reviewable files. This keeps authoritative diffs in the agent context (preventing hallucination) while dropping the volume that caused failures. Other changes: - Add git fetch to ensure merge-base matches GitHub's PR diff - Add excludedFileSummary so agents know what was filtered - Narrow allowed-tools to read-only git and jq commands - Forbid python for JSON extraction Assisted-by: Claude:claude-opus-4-6
cblecker
force-pushed
the
fix/pr-review-large-prs
branch
from
bfe7e77 to
cfe127d
Compare
1 participant
Summary
git fetchto ensure merge-base matches GitHub's PR diff; narrow allowed-tools to read-only git and jq commands*_generated.go,*.pb.go,*_string.go,bindata.go) and short-circuit when all files are excludedTest plan
/reviewon a small PR (<10 files) — verify status shows in file summary and diff headings/reviewon a large PR with vendor files — verify vendor files are excluded andexcludedFileSummaryappears in agent context/reviewon a PR where local HEAD matches — verify isLocal path usesgit diffwith merge-base/reviewon a PR where local HEAD does not match — verify not-isLocal path usesget_files+ jqSummary by CodeRabbit
Chores
Documentation