fix(pr-review-toolkit): restore scoring and embed inline diffs#46
Conversation
The agent prompts already instruct reviewers to produce 0-100 confidence scores, but the FINDING_SCHEMA lacked a confidence field so scores were silently discarded. Add the field to the schema and use it as a tiebreaker in the sort order (higher confidence first within the same severity level). Assisted-by: Claude:claude-opus-4-6
|
Warning Review limit reached
More reviews will be available in 41 minutes and 22 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe Changesreview-pr skill: confidence, richer context, and updated docs
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
pr-review-toolkit/skills/review-pr/review-pr.js (1)
94-95: ⚡ Quick winTighten doc-file detection to avoid false positives and unnecessary agent runs.
The regex at Line 94 matches any filename containing
doc, which can enablecomment-analyzerfor unrelated code files. Prefer anchored extension/path checks (e.g.,.md/.txt/.rst,README,docs/) to reduce unnecessary analyzer invocations.Suggested diff
- if (changedFiles.some(f => /\.md$|\.txt$|\.rst$|doc|readme/i.test(f.filename)) + if (changedFiles.some(f => /(\.md$|\.txt$|\.rst$|(^|\/)readme(\.|$)|(^|\/)docs?(\/|$))/i.test(f.filename)) || changedFiles.length >= 3) { agents.push('comment-analyzer') }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pr-review-toolkit/skills/review-pr/review-pr.js` around lines 94 - 95, The regex pattern used in the documentation file detection is overly broad because the "doc" portion lacks anchoring, causing it to match unrelated code files like "document.js" or "docker.yml". Refactor the regex pattern to use anchored extension and path checks instead - specifically, ensure documentation file detection uses patterns like checking for `.md`, `.txt`, `.rst` extensions (which are already properly anchored with $) and replace the loose "doc" and "readme" patterns with more specific anchored checks such as anchoring to directory names like `docs/` at the beginning of the path or anchoring README to the start and end of the filename. This will prevent false positives and reduce unnecessary invocations of the comment-analyzer.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@pr-review-toolkit/skills/review-pr/review-pr.js`:
- Around line 20-24: The schema definition at line 24 requires both severity and
confidence fields in the structured output, but only the code-reviewer prompt
explicitly requests confidence scoring while the other four agent prompts use
free-form formats. This causes structured-output validation to fail silently,
and invalid results are filtered out at line 559. Update all five agent prompts
(code-reviewer, security-analyzer, performance-optimizer,
maintainability-scorer, and documentation-checker) to explicitly request
severity field with values critical/important/suggestion and confidence field
with integer values from 0-100 in their structured output requirements.
---
Nitpick comments:
In `@pr-review-toolkit/skills/review-pr/review-pr.js`:
- Around line 94-95: The regex pattern used in the documentation file detection
is overly broad because the "doc" portion lacks anchoring, causing it to match
unrelated code files like "document.js" or "docker.yml". Refactor the regex
pattern to use anchored extension and path checks instead - specifically, ensure
documentation file detection uses patterns like checking for `.md`, `.txt`,
`.rst` extensions (which are already properly anchored with $) and replace the
loose "doc" and "readme" patterns with more specific anchored checks such as
anchoring to directory names like `docs/` at the beginning of the path or
anchoring README to the start and end of the filename. This will prevent false
positives and reduce unnecessary invocations of the comment-analyzer.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 67a2b921-e3b9-4375-89d5-b4a0562c026a
📒 Files selected for processing (3)
pr-review-toolkit/.claude-plugin/plugin.jsonpr-review-toolkit/skills/review-pr/SKILL.mdpr-review-toolkit/skills/review-pr/review-pr.js
Pass full file objects (filename, status, patch, additions, deletions) instead of bare filename strings so agents receive the actual diff content upfront rather than needing extra tool calls. Includes a 100KB guard for oversized patches and accurate language around truncated diffs. Also normalizes var/function() to const/let/arrows for consistency with the rest of the file. Assisted-by: Claude:claude-opus-4-6
cblecker
force-pushed
the
fix/pr-review-scoring
branch
from
b00edf8 to
6edcfbd
Compare
1 participant
Summary
var/function()toconst/let/arrows for consistencyTest plan
/reviewon a small PR to verify findings includeconfidencescores/reviewon a large PR (>100 changed files or large diffs) to verify the size guard and fallback behaviorclaude plugin validate ./pr-review-toolkitSummary by CodeRabbit