Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
156 lines (119 loc) · 4.83 KB

ARCHITECTURE.md

File metadata and controls

156 lines (119 loc) · 4.83 KB
Raw

Architecture Overview

This document explains how all the pieces fit together in this example.

System Components

1. The Agent (agent/)

A simple FAQ bot that answers customer questions:

  • Input: Customer question (string)
  • Output: Answer with confidence score and latency
  • Implementation: In-memory knowledge base with keyword matching
  • Purpose: Represents a real AI agent that needs evaluation

2. The Evals (evals/)

Using Promptfoo to evaluate the agent:

  • Framework: Promptfoo (open-source, easy to run locally)
  • Tests: 10+ test cases covering common questions
  • Metrics: Pass rate, latency, cost, etc.
  • Output: JSON results file (outputs/results.json)

Why Promptfoo?

  • Open-source and free
  • Easy to run locally (no API keys needed)
  • Well-documented
  • Geval has built-in adapter support

3. Geval Contracts (contracts/)

Decision contracts that define release gates:

  • Production: Strict thresholds (90% pass rate, <500ms latency)
  • Staging: Moderate thresholds (70% pass rate, <1000ms latency)
  • Development: Permissive (only blocks toxicity)

Contract Features Demonstrated:

  • ✅ Policy-based rules
  • ✅ Environment-specific policies
  • ✅ Eval metric conditions
  • ✅ Signal support (via --signals flag)
  • ✅ Default actions

4. Geval Decision Layer

Geval consumes eval results and produces decisions:

  • Input: Eval results + Contract + Signals (optional)
  • Output: Decision (PASS / BLOCK / REQUIRES_APPROVAL)
  • Artifacts: Decision record with cryptographic hash

5. CI/CD Integration (.github/workflows/)

GitHub Actions workflow that:

  • Runs evals on every PR
  • Checks results with Geval
  • Blocks merge if contract violated
  • Requires approval for REQUIRES_APPROVAL status

Data Flow

┌──────────────┐
│   Developer  │
│  Makes Change│
└──────┬───────┘
       │
       ▼
┌──────────────┐
│     Agent     │  FAQ bot code changes
│   (Code)      │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Promptfoo   │  Runs test suite
│   (Evals)     │  → outputs/results.json
└──────┬───────┘
       │
       ▼
┌──────────────┐
│    Geval      │  Evaluates against contract
│  (Decision)   │  → PASS / BLOCK / REQUIRES_APPROVAL
└──────┬───────┘
       │
       ▼
┌──────────────┐
│    CI/CD      │  Enforces decision
│  (Enforce)    │  → Deploy or Block
└──────────────┘

Key Design Decisions

Why This Example?

  1. Simple but Realistic: FAQ bot is easy to understand but demonstrates real patterns
  2. Complete Flow: Shows agent → evals → decision → enforcement
  3. Local First: Everything runs locally, no external dependencies
  4. Extensible: Easy to add more complex scenarios

Why Promptfoo?

  • Open Source: No vendor lock-in
  • Local Execution: No API keys or cloud services needed
  • Geval Integration: Built-in adapter support
  • Well Documented: Easy for users to understand and extend

Why Policy-Based Contracts?

  • Flexible: Supports both eval metrics and signals
  • Environment-Aware: Different rules for dev/staging/prod
  • Future-Proof: Easy to extend with new signal types

Extending This Example

Add More Metrics

  1. Edit evals/provider.js to calculate new metrics
  2. Update contracts/*.yaml to use new metrics
  3. Run npm run workflow to test

Add Signal-Based Rules

  1. Create signal files in signals/
  2. Update contracts to use signal conditions
  3. Run with --signals flag

Add Baseline Comparison

  1. Save a baseline eval result
  2. Update contract to use baseline: previous
  3. Pass --baseline flag to geval check

Integrate Real LLM

  1. Replace agent/src/bot.ts with real API calls
  2. Update evals/provider.js to call real endpoint
  3. Add API keys to environment variables

Geval Capabilities Demonstrated

✅ Eval-based contracts - Quality gates on metrics
✅ Policy-based contracts - Signal-driven decisions
✅ Environment-aware - Different rules per environment
✅ Baseline comparison - Regression detection (via --baseline)
✅ Signal integration - Human reviews, risk flags
✅ Decision records - Auditable artifacts
✅ CI/CD integration - Exit codes and automation

Next Steps

  1. Run the example: npm install && npm run workflow
  2. Modify contracts: Edit contracts/production.yaml
  3. Add tests: Extend evals/promptfoo.yaml
  4. Integrate: Copy patterns to your own project

See README.md for quick start guide.