End-to-End Agent Evaluation with LangSmith

A reference project that builds a LangGraph agent, evaluates it against a curated dataset, and publishes results to LangSmith — all in one python main.py run.

What This Project Does

1. Agent — A ReAct agent (LangGraph + ChatGroq) that answers some sample finance questions using two tools: retrieve_documents and lookup_control.
2. Dataset — Six hand-crafted test cases with expected answers and expected tool-call sequences, pushed to LangSmith as a versioned dataset.
3. Graders — Two scoring functions wired as LangSmith evaluators:
   • tool_sequence_accuracy — deterministic Jaccard similarity over tool calls (no LLM needed).
   • answer_quality — LLM-as-judge factual-accuracy check via ChatGroq.
4. Experiments — Runs the agent with two different models, records each as a separate LangSmith experiment, and lets you compare them side-by-side in the LangSmith UI.

Project Structure

field-guide/
├── main.py                       ← single entry point
├── agent/
│   ├── tools.py                  ← retrieve_documents + lookup_control
│   └── graph.py                  ← LangGraph ReAct agent
└── evals/
    ├── schemas.py                ← Pydantic v2 data models
    ├── graders.py                ← Grader protocol + two grader implementations
    ├── dataset.py                ← 6 OWASP test cases
    ├── langsmith_dataset.py      ← push dataset → LangSmith
    └── runner.py                 ← make_target + evaluators + run_eval()

Quick Start

1. Install dependencies

uv sync

2. Set environment variables

Create a .env file:

GROQ_API_KEY=gsk_...
LANGCHAIN_API_KEY=ls__...
LANGCHAIN_TRACING_V2=true
LANGCHAIN_PROJECT=eval-agent

3. Run the evaluation

python main.py

This will:

• Push the 6-case dataset to LangSmith
• Evaluate llama-3.3-70b-versatile → experiment llama-3.3-70b
• Evaluate openai/gpt-oss-120b → experiment gpt-oss-120b
• Print a link to compare results

4. Compare in LangSmith

Open LangSmith → Datasets & Experiments → agent-eval, select both experiments, and click Compare.

Extending

Add a model — add another run_eval() call in main.py with a different model and experiment_prefix.

Add a grader — implement the grade(GradeInput) → GraderResult interface in evals/graders.py, then wire it as an evaluator in evals/runner.py.