LLM Evaluation & Testing Framework

A comprehensive framework for evaluating LLM-based systems, particularly RAG (Retrieval-Augmented Generation) applications.

Features

• Multiple Evaluation Metrics

  • Retrieval metrics: Recall@k, Precision@k, Hit Rate, MRR
  • Generation metrics: Exact Match, Keyword Overlap, Contains Match
  • Advanced metrics: BLEU, ROUGE, METEOR, Semantic Similarity
  • LLM-as-Judge scoring for answer quality

• Test Suite Management

  • Create and manage test suites
  • Import/export from JSON and CSV
  • Categorize test cases by type and difficulty
  • Version control for benchmarks

• A/B Testing

  • Compare different models/configurations
  • Side-by-side metric comparisons
  • Statistical significance testing

• Reporting & Visualization

  • Text reports
  • HTML reports with interactive visualizations
  • CSV exports for further analysis
  • Metric distribution plots
  • Execution time analysis

Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Create a Test Suite

from app.evaluation import TestSuite, TestCase

# Create a new test suite
suite = TestSuite(
    name="my_benchmark",
    description="My evaluation benchmark"
)

# Add test cases
test_case = TestCase(
    id="tc1",
    question="What are the main risk factors?",
    expected_answer="The main risk factors include...",
    expected_keywords=["risk", "volatility", "market"],
    expected_sources=["document1.pdf"],
    category="risk_analysis",
    difficulty="medium"
)

suite.add_test_case(test_case)

3. Run Evaluation

from app.evaluation import LLMEvaluator
from app.retrieval.vector_store import get_vector_store
from app.llm.qa_chain import QAChain
from app.ingestion.load_portfolios import PortfolioLoader

# Initialize components
vector_store = get_vector_store()
portfolio_loader = PortfolioLoader()
qa_chain = QAChain(vector_store=vector_store, portfolio_loader=portfolio_loader)

# Create evaluator
evaluator = LLMEvaluator(qa_chain=qa_chain, vector_store=vector_store)

# Run evaluation
results = evaluator.evaluate_test_suite(suite)

# Generate reports
from app.evaluation import EvaluationReporter
reporter = EvaluationReporter()
reporter.generate_html_report(results)
reporter.generate_csv_report(results)

Command Line Interface

Create a Test Suite

python run_evaluation.py create-suite \
    --name my_benchmark \
    --description "My evaluation benchmark" \
    --test-cases-file test_cases.json \
    --output test_suites

Run Evaluation

python run_evaluation.py evaluate \
    --suite test_suites/my_benchmark.json \
    --reports \
    --visualize

Compare Two Evaluations

python run_evaluation.py compare \
    --eval1 evaluation_results/evaluation_model1_20240101_120000.json \
    --eval2 evaluation_results/evaluation_model2_20240101_120000.json \
    --report

Add Test Case to Suite

python run_evaluation.py add-test-case \
    --suite test_suites/my_benchmark.json \
    --id tc_new \
    --question "What is the investment strategy?" \
    --expected-answer "The strategy focuses on..." \
    --keywords "strategy,investment,growth" \
    --category "strategy" \
    --difficulty "easy"

List Test Suites

python run_evaluation.py list-suites --directory test_suites

Test Suite Format

JSON Format

{
  "name": "my_benchmark",
  "description": "My evaluation benchmark",
  "test_cases": [
    {
      "id": "tc1",
      "question": "What are the main risk factors?",
      "expected_answer": "The main risk factors include...",
      "expected_keywords": ["risk", "volatility", "market"],
      "expected_sources": ["document1.pdf"],
      "category": "risk_analysis",
      "difficulty": "medium",
      "metadata": {}
    }
  ],
  "metadata": {},
  "created_at": "2024-01-01T12:00:00",
  "updated_at": "2024-01-01T12:00:00"
}

CSV Format

id,question,expected_answer,expected_keywords,expected_sources,category,difficulty
tc1,"What are the main risk factors?","The main risk factors include...","risk,volatility,market","document1.pdf",risk_analysis,medium

Advanced Usage

Using Advanced Metrics

from app.evaluation import AdvancedMetrics

metrics = AdvancedMetrics()

# Calculate semantic similarity
similarity = metrics.calculate_semantic_similarity(
    predicted="The answer is...",
    reference="The expected answer is..."
)

# Calculate BLEU score
bleu = metrics.calculate_bleu_score(
    predicted="The answer is...",
    reference="The expected answer is..."
)

# Calculate ROUGE scores
rouge_scores = metrics.calculate_rouge_scores(
    predicted="The answer is...",
    reference="The expected answer is..."
)

# Calculate all metrics
all_metrics = metrics.calculate_all_metrics(
    predicted="The answer is...",
    reference="The expected answer is..."
)

Benchmark Management

from app.evaluation import BenchmarkManager

manager = BenchmarkManager()

# Create from JSON
suite = manager.create_from_json(
    json_path=Path("benchmarks/my_benchmark.json"),
    suite_name="my_benchmark",
    description="My benchmark"
)

# Create from CSV
suite = manager.create_from_csv(
    csv_path=Path("benchmarks/my_benchmark.csv"),
    suite_name="my_benchmark"
)

# Export to different formats
manager.export_to_json(suite, Path("output/my_benchmark.json"))
manager.export_to_csv(suite, Path("output/my_benchmark.csv"))

# List available benchmarks
benchmarks = manager.list_benchmarks()

A/B Testing

# Run evaluation with Model 1
evaluator1 = LLMEvaluator(qa_chain=qa_chain_model1)
results1 = evaluator1.evaluate_test_suite(suite)

# Run evaluation with Model 2
evaluator2 = LLMEvaluator(qa_chain=qa_chain_model2)
results2 = evaluator2.evaluate_test_suite(suite)

# Compare results
comparison = evaluator1.compare_evaluations(results1, results2)

# Generate comparison report
reporter = EvaluationReporter()
reporter.generate_comparison_report(comparison)

Evaluation Metrics

Retrieval Metrics

• Recall@k: Fraction of expected sources found in top-k results
• Precision@k: Fraction of top-k results that are expected sources
• Hit Rate: Whether at least one expected source was retrieved
• MRR (Mean Reciprocal Rank): Average of reciprocal ranks of first relevant result

Generation Metrics

• Exact Match: Whether predicted answer exactly matches expected
• Contains Match: Whether predicted answer contains expected answer
• Keyword Overlap: Fraction of expected keywords found in predicted answer

Advanced Metrics

• BLEU: N-gram precision score
• ROUGE-1/2/L: Recall-oriented metrics for summarization
• METEOR: Metric considering synonyms and paraphrasing
• Semantic Similarity: Cosine similarity of embeddings
• Factual Consistency: Overlap of numbers and dates

Output Structure

evaluation_results/
├── evaluation_my_benchmark_20240101_120000.json  # Full results
evaluation_reports/
├── report_my_benchmark_20240101_120000.txt      # Text report
├── report_my_benchmark_20240101_120000.html     # HTML report
├── report_my_benchmark_20240101_120000.csv       # CSV report
└── plots/
    ├── my_benchmark_metric_distribution.png      # Metric distributions
    └── my_benchmark_execution_time.png           # Execution time plot

Integration with Existing Code

The framework integrates seamlessly with the existing RAG system:

from app.retrieval.vector_store import get_vector_store
from app.llm.qa_chain import QAChain
from app.ingestion.load_portfolios import PortfolioLoader
from app.evaluation import LLMEvaluator, TestSuite

# Use existing components
vector_store = get_vector_store()
portfolio_loader = PortfolioLoader()
qa_chain = QAChain(vector_store=vector_store, portfolio_loader=portfolio_loader)

# Create evaluator
evaluator = LLMEvaluator(qa_chain=qa_chain, vector_store=vector_store)

# Run evaluation
results = evaluator.evaluate_test_suite(your_test_suite)

Best Practices

1. Test Suite Design

   • Include diverse question types
   • Cover different difficulty levels
   • Include edge cases and failure modes
   • Regularly update benchmarks

2. Evaluation Frequency

   • Run evaluations after major changes
   • Track metrics over time
   • Compare against baselines

3. Metric Selection

   • Use retrieval metrics for RAG systems
   • Use generation metrics when ground truth answers exist
   • Combine multiple metrics for comprehensive evaluation

4. Reporting

   • Generate reports for all evaluations
   • Archive results for historical comparison
   • Share reports with stakeholders

Troubleshooting

Import Errors

Make sure all dependencies are installed:

pip install -r requirements.txt

NLTK Data Missing

The framework will automatically download required NLTK data, but if you encounter issues:

import nltk
nltk.download('punkt')
nltk.download('wordnet')
nltk.download('omw-1.4')

Visualization Not Working

If matplotlib/seaborn visualizations fail, check that the packages are installed:

pip install matplotlib seaborn

License

This evaluation framework is part of the FinSight Copilot project.