ai-code-reviewer

An AI code review agent that reviews pull requests like a senior engineer — catching real bugs (race conditions, N+1 queries, missing error handling, security issues), not style nits. Backed by a versioned eval harness that measures precision, recall, false-positive rate, and cost per review on a hand-labeled test set of known-buggy PRs.

Status: week 1 of 3. Local CLI works on static diffs. GitHub App integration (week 2) and full eval harness (week 3) coming.

Why this project exists

Most new-grad AI projects skip two hard things: real agents (tool use, iteration, termination) and real evals (structured ground truth, regression tracking, negative cases). This project is built around those two. The README, the docs/ folder, and the evals/ folder are all first-class deliverables — not afterthoughts.

Architecture (week 1)

 ┌────────┐      ┌──────────────────────────┐      ┌─────────────┐
 │ diff + │ ───► │  ReAct agent loop        │ ───► │  findings   │
 │ repo   │      │  (reviewer/agent/loop.py)│      │  (JSON)     │
 └────────┘      │                          │      └─────────────┘
                 │  Tools:                  │
                 │   - read_file            │
                 │   - grep_repo            │
                 │   - submit_findings      │
                 └──────────────────────────┘

The agent receives a unified diff and a path to the repo. It has three tools. It loops: reason → tool call → observe → reason → ... → submit_findings. Hard caps at 8 iterations or $0.50.

Full architectural rationale in docs/week-1-guide.md.

Setup

Prerequisites: Python 3.11+ and uv.

# Install uv if you haven't. On Windows (PowerShell):
#   powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# On macOS/Linux:
#   curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone this repo, then from the project root:
uv venv                     # creates a .venv/
uv pip install -e ".[dev]"  # installs deps + this package in editable mode

# Copy the env template and fill in your LLM key
cp .env.example .env
# Then open .env and paste your key on the LLM_API_KEY line.

Getting an LLM API key (free tier — recommended)

The default config uses Google Gemini's free tier (no credit card, ~1500 requests/day):

1. Go to aistudio.google.com/app/apikey and sign in with a Google account.
2. Click Create API key → copy it.
3. Paste it on the LLM_API_KEY= line in your .env.

That's it — LLM_BASE_URL and REVIEW_MODEL already default to Gemini.

Switching providers

The agent talks to any OpenAI-compatible endpoint. To swap providers, change LLM_BASE_URL, LLM_API_KEY, and REVIEW_MODEL in your .env — no code changes. See .env.example for presets (OpenAI, Groq, local Ollama).

Run it

uv run python -m reviewer.cli \
    --diff tests/fixtures/diffs/race-condition.patch \
    --repo tests/fixtures/example_repo

You should see a JSON review printed to stdout. The included fixture has a planted race-condition bug in counter.py; the agent should flag it.

What's coming

• Week 2: GitHub App, webhook receiver, Postgres persistence, deployed to Fly/Railway, posts inline review comments on real PRs.
• Week 3: eval harness (50+ hand-labeled cases), regression tracker, LLM-as-judge for fuzzy scoring, metrics on this README, and a written blog post.

Repo layout

reviewer/              # the Python package
  agent/               # ReAct loop, tools, prompts, Finding schema
  llm/                 # LLM client + cost accounting
  diff/                # unified diff parsing + skip list
  config.py            # loads .env via pydantic-settings
  cli.py               # `python -m reviewer.cli ...`
evals/                 # YAML test cases + runner (week 3)
tests/                 # pytest unit tests + fixtures
docs/                  # architecture notes, eval design, blog post