Skip to content

CLAUDE.md

Latest commit

 

History

History
56 lines (41 loc) · 1.95 KB

CLAUDE.md

File metadata and controls

56 lines (41 loc) · 1.95 KB
Raw

ModelRegression.com

Independent automated benchmarking of frontier AI models. Tracks performance regressions so the community knows when models degrade.

Architecture

  • Website: Next.js 15 (App Router), TailwindCSS, Recharts, Framer Motion
  • Benchmark Engine: Python 3.13, SQLite, runs on DGX Sparks
  • Deployment: Linode server, PM2, Nginx reverse proxy

Commands

npm run dev          # Dev server on port 3002
npm run build        # Production build
npm start            # Production server on port 3002
bash deploy.sh       # Deploy to production

Benchmark Engine

cd benchmark
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python runner.py --schedule morning    # Run benchmarks
python export_json.py --output ../public/data  # Export JSON

Cron Schedule (DGX)

Single daily run at 3am ET via benchmark/run_benchmarks.sh. Outage monitor every 30 min. PATH must include ~/.local/bin and ~/.npm-global/bin for CLI tools.

Project Structure

  • app/ — Next.js pages (dashboard, models, categories, compare, evidence, outages, methodology, about)
  • components/ — React components (shared, charts, dashboard)
  • lib/ — Types, utilities, data loading, model/category metadata
  • public/data/ — JSON data files (generated by benchmark engine)
  • benchmark/ — Python benchmark suite (config, db, runner, tests, export)
  • config/ — Nginx configuration template
  • deploy.sh — Blue-green atomic deployment script

Models Tested

Claude Opus 5, Claude Sonnet 5, GPT-5.6 Sol, Grok 4.5. Claude aliases and Codex/Grok catalogs resolve at runtime so scheduled benchmarks follow each provider's current frontier model.

Key Files

  • benchmark/runner.py — Main benchmark orchestrator
  • benchmark/export_json.py — SQLite to JSON exporter
  • benchmark/run_benchmarks.sh — Cron entry point
  • deploy.sh — Production deployment
  • ecosystem.config.js — PM2 process config