Quality Estimation Tools

This directory contains tools for evaluating the quality and relevance of query-response pairs using LLMs.

Setup

Install dependencies:

cd /home/pasha/azeroth/search/quality
uv venv
source .venv/bin/activate
uv pip install pandas langchain-openai langchain-core tqdm

Set your OpenRouter API key:

export OPENROUTER_API_KEY="your-api-key-here"

Get your API key from: https://openrouter.ai/keys

Usage

Basic Usage

Estimate relevance for query-response pairs from one or more input files:

python estimate_relevance.py --input data.csv --output estimates.csv

Multiple Input Files

Process multiple input files at once:

python estimate_relevance.py --input file1.csv file2.tsv file3.csv --output estimates.csv

Force Re-estimation

Force re-estimation even if estimates already exist:

python estimate_relevance.py --input data.csv --output estimates.csv --force

Custom Model

Use a different OpenRouter model:

# Use Gemini 2.5 Pro (recommended for quality, requires credits)
python estimate_relevance.py --input data.csv --output estimates.csv --model google/gemini-2.5-pro

# Use Gemini Pro 1.5
python estimate_relevance.py --input data.csv --output estimates.csv --model google/gemini-pro-1.5

# Use Gemini Flash 1.5 (faster, cheaper)
python estimate_relevance.py --input data.csv --output estimates.csv --model google/gemini-flash-1.5

# Use Gemini 2.0 Flash (free tier, default)
python estimate_relevance.py --input data.csv --output estimates.csv --model google/gemini-2.0-flash-exp:free

Note: Google Gemini 2.5 Pro is the recommended model for highest quality evaluations, but requires OpenRouter credits. The default is Gemini 2.0 Flash (free tier) which is still quite capable.

Available models on OpenRouter: https://openrouter.ai/models

Queries File

The test queries are stored in queries.csv with the following format:

query_id,query
1,What is quantum computing
2,How does photosynthesis work?
...

You can edit this file to add or modify queries for evaluation.

Generating Test Data

Before estimating relevance, you need query-response data. Use the provider-specific scripts to generate AI-powered answers:

Space Frontiers

Use generate_example_data_spacefrontiers.py to generate answers from Space Frontiers API:

# Set your API key (required)
export SPACE_FRONTIERS_API_KEY="your-api-key-here"

# Generate data using queries from queries.csv
python generate_example_data_spacefrontiers.py --output example_data_spacefrontiers.csv

Or pass the API key directly:

python generate_example_data_spacefrontiers.py --output example_data_spacefrontiers.csv --api-key YOUR_API_KEY

This uses the Space Frontiers conversation endpoint to generate AI-powered answers to the queries using the specified LLM model.

Perplexity

Use generate_example_data_perplexity.py to generate answers from Perplexity API:

# Set your API key (required)
export PERPLEXITY_API_KEY="your-api-key-here"

# Generate data using queries from queries.csv
python generate_example_data_perplexity.py --output example_data_perplexity.csv

Custom Queries

You can provide queries directly on the command line to override the queries.csv file:

python generate_example_data_spacefrontiers.py --output test_data.csv --queries "What is AI?" "How does ML work?" "Explain neural networks"

Different LLM Models (Space Frontiers)

Choose which AI model to use for generating answers:

# Use Gemini 2.5 Flash (default, balanced performance)
python generate_example_data_spacefrontiers.py --output test_data.csv --model google/gemini-2.5-flash

# Use Gemini 2.5 Pro (highest quality)
python generate_example_data_spacefrontiers.py --output test_data.csv --model google/gemini-2.5-pro

# Use Claude Sonnet 4.5
python generate_example_data_spacefrontiers.py --output test_data.csv --model anthropic/claude-sonnet-4.5

# Use GPT-4.1
python generate_example_data_spacefrontiers.py --output test_data.csv --model openai/gpt-4.1

Multiple Sources (Space Frontiers)

Query multiple data sources at once:

# Search across library and Reddit
python generate_example_data_spacefrontiers.py --output multi_source_data.csv --sources library reddit

# Search across all sources
python generate_example_data_spacefrontiers.py --output all_sources_data.csv --sources library reddit telegram youtube

# Single source (default: all sources)
python generate_example_data_spacefrontiers.py --output library_data.csv --sources library

Note: The conversation endpoint will retrieve and combine results from all specified sources when generating the AI answer.

Custom Queries File

Use a different queries file:

python generate_example_data_spacefrontiers.py --output test_data.csv --queries-file my_queries.csv
python generate_example_data_perplexity.py --output test_data.csv --queries-file my_queries.csv

Environment Configuration

Create a .env file in the search/quality directory:

# .env file
SPACE_FRONTIERS_API_KEY=your-api-key-here
SPACE_FRONTIERS_API_ENDPOINT=http://localhost:8080
PERPLEXITY_API_KEY=your-perplexity-api-key-here
OPENROUTER_API_KEY=your-openrouter-api-key-here

All scripts will automatically load these variables.

Complete Workflow

Generate test data and estimate relevance:

# Set your API keys
export SPACE_FRONTIERS_API_KEY="your-api-key-here"
export PERPLEXITY_API_KEY="your-perplexity-api-key-here"
export OPENROUTER_API_KEY="your-openrouter-api-key-here"

# 1. Generate test data from Space Frontiers using conversation endpoint
python generate_example_data_spacefrontiers.py --output example_data_spacefrontiers.csv --model google/gemini-2.5-flash

# 2. Generate test data from Perplexity
python generate_example_data_perplexity.py --output example_data_perplexity.csv --model sonar-pro

# 3. Estimate relevance using LLM for both providers
python estimate_relevance.py --input example_data_spacefrontiers.csv example_data_perplexity.csv --output estimates.csv --model google/gemini-2.5-pro

# 4. Analyze results
# The estimates.csv file now contains relevance, diversity, and trustness scores for both providers

Note: The generate scripts use their respective APIs to generate AI-powered answers using the specified models. The estimate_relevance.py script then evaluates those answers for relevance, diversity, and trustworthiness.

Input File Format

Input files should be CSV or TSV format with the following columns:

• query_id: Unique identifier for the query
• query: The search query text
• provider_name: Name of the provider (e.g., "space-frontiers")
• response: The response text to evaluate

Example CSV:

query_id,query,provider_name,response
abc123,What is quantum computing?,space-frontiers,Quantum computing is a type of computation...
abc124,How does photosynthesis work?,space-frontiers,Photosynthesis is the process by which plants...

Example TSV:

query_id	query	provider_name	response
abc123	What is quantum computing?	space-frontiers	Quantum computing is a type of computation...
abc124	How does photosynthesis work?	space-frontiers	Photosynthesis is the process by which plants...

Output File Format

The output file contains the following columns:

• query_id: Unique identifier for the query
• provider_name: Name of the provider
• relevance: Score from 1-5 (how relevant is the response)
• diversity: Score from 1-5 (how rich and comprehensive is the response)
• trustness: Score from 1-5 (how trustworthy is the response)
• reasoning: Brief explanation of the ratings

Evaluation Metrics

Relevance (1-5)

• 1: Completely irrelevant, doesn't address the query at all
• 2: Barely relevant, touches on the topic but doesn't answer the query
• 3: Somewhat relevant, partially answers the query but misses key aspects
• 4: Mostly relevant, answers the query well with minor gaps
• 5: Perfectly relevant, directly and completely answers the query

Diversity (1-5)

• 1: Very narrow, single perspective or piece of information
• 2: Limited scope, covers only the basics
• 3: Moderate diversity, includes some additional relevant information
• 4: Good diversity, covers multiple aspects and perspectives
• 5: Excellent diversity, comprehensive with many relevant facets

Trustness (1-5)

• 1: Highly suspicious, likely contains misinformation
• 2: Questionable, lacks credibility or evidence
• 3: Moderately trustworthy, seems reasonable but not well-supported
• 4: Trustworthy, appears accurate with some supporting evidence
• 5: Highly trustworthy, well-sourced and authoritative

Features

• Incremental Processing: Automatically checks for existing estimates and only processes new rows
• Force Mode: Use --force flag to re-estimate all rows
• Multiple Input Files: Process multiple files in a single run
• Format Detection: Automatically detects CSV vs TSV format
• Progress Tracking: Shows progress bar during estimation
• Error Handling: Gracefully handles errors and marks failed estimations
• Summary Statistics: Displays mean, median, and standard deviation for each metric

Examples

Example 1: Initial Estimation

# First run - estimates all rows
python estimate_relevance.py --input queries_batch1.csv --output estimates.csv
# Output: Processed 100 rows, saved to estimates.csv

Example 2: Incremental Update

# Second run with more data - only estimates new rows
python estimate_relevance.py --input queries_batch1.csv queries_batch2.csv --output estimates.csv
# Output: Found 100 existing estimates, processed 50 new rows

Example 3: Re-evaluation

# Force re-estimation of all rows (e.g., after model improvement)
python estimate_relevance.py --input queries_batch1.csv --output estimates.csv --force
# Output: Processed all 100 rows