Compute Efficiency Benchmark

Overview

LLMs are becoming part of production systems across many industries. As adoption grows, compute is becoming an increasingly important engineering concern. Larger models, longer prompts, and more complex workflows improve capabilities, but they also increase cost, latency, and infrastructure demand.

Managing compute is therefore becoming as important as managing model quality. The goal is not simply to use the most capable model available, but to understand the trade-off between performance and efficiency.

This repository demonstrates one practical approach to managing GenAI compute through context compression. By reducing the amount of information sent to a model before inference, it is possible to lower token consumption while measuring the effect on application performance.

The repository contains a reproducible benchmark that evaluates how context compression and model selection influence accuracy, token usage, latency, and estimated inference cost using a long-context question-answering task.

Questions explored

The benchmark investigates three practical questions:

1. How much context can be removed before model performance starts to degrade?
2. How much compute can be saved through prompt compression?
3. How does model selection compare with context compression as a strategy for improving efficiency?

Benchmark design

A Q&A GenAI application was benchmarked to illustrate the performance-cost trade-off. The evaluation measures answer accuracy and normalized compute cost across:

• three model sizes: gpt-5, gpt-5-mini, and gpt-5-nano;
• four context settings: full context, 20% reduction, 40% reduction, and 60% reduction;
• 300 questions from the QuALITY long-context question-answering dataset.

Only the article context is compressed. The question and answer options remain unchanged for every run.

Accuracy versus normalized compute cost across model and context configurations. Cost is normalized to gpt-5 with full context and is based on measured token usage and relative model price factors.

Main findings

• Compute cost is controllable. gpt-5 maintains its baseline performance at 20% context reduction and declines moderately at 40% and 60%. Smaller models trade accuracy more quickly as context is reduced.
• Model choice dominates cost. gpt-5-mini achieves accuracy close to gpt-5 at roughly one-fifth of the normalized cost, while gpt-5 is approximately 25 times more expensive than gpt-5-nano in this comparison.
• Small accuracy gains can require disproportionately higher cost. Selecting a model and context configuration is therefore a strategic decision, not only a technical one.

The benchmark does not aim to identify one universally optimal configuration. It provides a repeatable way to find an acceptable operating point for a specific application, based on its required accuracy, cost, and latency.

How it works

The workflow has four main stages:

1. Prepare the dataset. Validate the QuALITY input and confirm that each question has four answer options and a valid gold answer.
2. Compress the context. Run LLMLingua-2 locally to generate context variants with keep-rates of 0.8, 0.6, and 0.4.
3. Run the benchmark. Evaluate the original and compressed contexts with each Azure OpenAI deployment.
4. Analyze the results. Aggregate accuracy, token usage, latency, compression statistics, and estimated cost into summary tables and plots.

Compression rates in this repository are described as context reduction:

Context setting	LLMLingua-2 keep-rate	Approximate context reduction
Full	1.0	0%
Compression 20%	0.8	20%
Compression 40%	0.6	40%
Compression 60%	0.4	60%

Repository structure

artifacts/              Publishable figures
scripts/
  analyze.py            Aggregate results and generate plots
  benchmark.py          Run Azure OpenAI question answering
  compress.py           Generate compressed dataset variants
  validate_dataset.py   Validate the QuALITY dataset
results/
  eval.sqlite           Cached benchmark results
  analysis/             Generated summaries and plots
data/
  quality/              Original QuALITY files
  quality_comp_*/       Compressed dataset variants

Requirements

• Python 3.10 or newer
• Azure OpenAI deployments for the models being evaluated
• Internet access for the first LLMLingua-2 model download
• Local disk space for the dataset, model cache, and benchmark results

Dataset validation and context compression run locally. Benchmark evaluation requires Azure OpenAI credentials and incurs API usage charges.

Setup

Run the following commands from the repository root.

1. Create and activate an environment

python -m venv .venv
.venv\Scripts\Activate.ps1

If an existing environment is located in the parent folder, activate it instead:

..\.venv\Scripts\Activate.ps1

2. Install the project

python -m pip install --upgrade pip
python -m pip install -e ".[dev]"

3. Configure Azure OpenAI

Copy-Item .env.example .env

Edit .env with the Azure OpenAI credentials and deployment names available in your environment. Do not commit .env.

LLMLingua-2 setup

Context compression uses the following Hugging Face model:

microsoft/llmlingua-2-xlm-roberta-large-meetingbank

No separate download command is required. The first run of scripts/compress.py downloads the model and stores it in the local Hugging Face cache. Later runs reuse the cached files.

The current implementation uses:

• LLMLingua-2 mode with use_llmlingua2=True;
• CPU execution with device_map="cpu";
• paragraph-based chunking with a maximum of 1,400 characters per chunk;
• keep-rates supplied through --rates;
• punctuation and line breaks preserved as forced or chunk-ending tokens.

The first run requires internet access and may take longer while the model is downloaded and initialized. Compression itself is CPU-intensive.

Dataset

The benchmark uses QuALITY: Question Answering with Long Input Texts, introduced by Pang et al. at ACL 2022.

• QuALITY repository
• QuALITY v1.0.1 data

Download the QuALITY v1.0.1 files and place them under data/quality/:

data/
└── quality/
    ├── QuALITY.v1.0.1.dev
    ├── QuALITY.v1.0.1.train
    ├── QuALITY.v1.0.1.test
    ├── QuALITY.v1.0.1.htmlstripped.dev
    ├── QuALITY.v1.0.1.htmlstripped.train
    └── QuALITY.v1.0.1.htmlstripped.test

The files can be copied from a local clone of the official repository:

git clone https://github.com/nyu-mll/quality.git
cp quality/data/v1.0.1/QuALITY.v1.0.1* data/quality/

The benchmark described in this repository uses:

data/quality/QuALITY.v1.0.1.htmlstripped.train

Run the benchmark

Run the workflow one step at a time. Start with small validation and benchmark runs before launching the full evaluation.

1. Validate the dataset

This command checks that the dataset exists, can be read, contains questions with four options, and has valid gold answers. It runs offline.

python scripts/validate_dataset.py

For a quick validation of the first 20 questions:

python scripts/validate_dataset.py --limit 20

The command reports the number of questions validated in the local file.

2. Test context compression

Start by compressing three articles at a single keep-rate:

python scripts/compress.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --rates 0.8 `
    --limit 3

The compressed output is written to:

data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train

Once the test succeeds, generate all compressed variants:

python scripts/compress.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --rates 0.8,0.6,0.4

3. Run a small Azure OpenAI evaluation

Evaluate three questions with gpt-5-nano before running the complete benchmark:

python scripts/benchmark.py `
    data/quality/QuALITY.v1.0.1.htmlstripped.train `
    --deployment gpt-5-nano `
    --db results/eval.sqlite `
    --n 3

Then evaluate the compressed version:

python scripts/benchmark.py `
    data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train `
    --deployment gpt-5-nano `
    --db results/eval.sqlite `
    --n 3

Results are cached in results/eval.sqlite. Re-running the same dataset, deployment, and prompt version skips completed questions and avoids duplicate API calls.

4. Run the full evaluation

The published benchmark uses the first 300 questions for each model and context configuration.

Full context

python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

20% context reduction

python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.80/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

40% context reduction

python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.60/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

60% context reduction

python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5 --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-mini --db results/eval.sqlite --n 300
python scripts/benchmark.py data/quality_comp_0.40/QuALITY.v1.0.1.htmlstripped.train --deployment gpt-5-nano --db results/eval.sqlite --n 300

5. Analyze the results

python scripts/analyze.py --db results/eval.sqlite --out results/analysis

The analysis step writes summary tables, JSON output, and generated plots to:

results/analysis/

The publishable figure is written to:

artifacts/accuracy_vs_normalized_cost.png

Measurements

For each question and configuration, the benchmark records:

• answer correctness;
• input, output, and total tokens;
• request latency;
• response parsing status;
• compression ratio;
• local compression time.

Accuracy is calculated against the QuALITY gold answer. Monetary cost is derived separately from measured token usage using documented model prices or relative price factors.

Normalized cost is calculated relative to the baseline configuration:

normalized cost = configuration cost / gpt-5 full-context cost

Token savings and latency are directly measured. CO2 should be treated as a proxy unless an explicit energy and emissions methodology is added for both local compression and cloud inference.

Interpreting the results

The observed results are specific to this dataset, sample, prompt, model deployments, and compression configuration. They should not be treated as universal performance guarantees.

The main value of the repository is the evaluation pattern:

1. define the required application quality;
2. benchmark multiple model and context configurations;
3. measure the full cost-performance trade-off;
4. select the least compute-intensive configuration that satisfies the application requirements.

This same approach can be adapted to document Q&A, retrieval-augmented generation, enterprise search, meeting analysis, and other long-context applications.

References

• LLMLingua-2 project page
• LLMLingua-2 paper
• Microsoft LLMLingua repository
• LLMLingua-2 model
• QuALITY dataset repository