Add vLLM offline backend with micro-batching support

[maryamtahhan]

Add vLLM Offline Backend with Shared Base Class

This PR implements offline/batch inference support for vLLM using a clean, extensible architecture that eliminates code duplication between vLLM backends.

Summary

Adds VLLMOfflineBackend for batch processing and refactors existing vLLM code into a shared VLLMBackendBase class. This reduces code duplication by ~360 lines while adding new offline inference capabilities optimized for benchmarking scenarios.

New Components

VLLMBackendBase (base.py)

Shared base class for all vLLM backends containing ~400 lines of common functionality:

• Chat template resolution (plain, default-template, custom Jinja2)
• Multimodal data handling (image/audio columns)
• Request formatting and prompt resolution
• Sampling parameter creation
• Abstract _get_tokenizer() method for subclass implementation

VLLMOfflineBackend (offline.py)

New backend for offline batch processing using vLLM's LLM class:

• Micro-batching with configurable batch_size (default: 32)
• Buffers requests until batch is full, then processes with LLM.generate()
• Auto-flushes remaining requests on shutdown
• Single-process execution for batch coordination
• Ideal for offline benchmarking and dataset evaluation

Refactored VLLMPythonBackend (vllm.py)

• Now extends VLLMBackendBase instead of Backend directly
• Removed ~360 lines of duplicate code
• Implements _get_tokenizer() for AsyncLLMEngine
• No breaking changes to public API

Key Benefits

• ✅ Code Reuse: ~400 lines shared between backends
• ✅ Reduced Duplication: ~360 lines eliminated from VLLMPythonBackend
• ✅ Extensibility: Easy to add new vLLM-based backends (e.g., vLLM server)
• ✅ No Breaking Changes: VLLMPythonBackend API unchanged
• ✅ Clean Architecture: Clear separation of concerns with shared base

Documentation

• New guide: docs/guides/vllm-offline-backend.md
  • Usage examples and configuration options
  • Performance tuning (batch size, vLLM EngineArgs)
  • Comparison with other backends
  • Troubleshooting guide
• Updated: docs/guides/backends.md with offline backend section

Usage Example

guidellm benchmark run \
  --backend vllm_offline \
  --model "Qwen/Qwen3-0.6B" \
  --backend-kwargs '{"batch_size": 64, "vllm_config": {"tensor_parallel_size": 2}}' \
  --data "prompt_tokens=256,output_tokens=128" \
  --max-requests 1000

Test Plan

Unit Tests (✅ Passing)

• 2296 unit tests passing (all existing + new tests)
• New test coverage:
  • VLLMOfflineBackend lifecycle (startup, shutdown, validate)
  • Batch processing logic and request buffering
  • VLLMBackendBase request resolution and formatting
  • Chat template handling (plain, default, custom)
  • Multimodal data processing (audio/image)
  • Sampling parameter creation
  • Backend registration and creation

Integration Tests (✅ Verified)

• Backend registration in Backend registry
• Args creation and validation (VLLMOfflineBackendArgs)
• Backend creation via Backend.create()
• Request resolution with chat templates
• Batch size configuration (8-128+)
• vLLM config passthrough (tensor_parallel_size, gpu_memory_utilization, etc.)
• Backend info property exposure

Manual Testing

• Validated functionality on local environment

Details

• Add VLLMBackendBase shared base class in src/guidellm/backends/vllm_python/base.py
• Add VLLMOfflineBackend and VLLMOfflineBackendArgs in src/guidellm/backends/vllm_python/offline.py
• Refactor VLLMPythonBackend to extend VLLMBackendBase (eliminate duplication)
• Re-export test helpers (_ResolvedRequest, _has_jinja2_markers) from base for backward compatibility
• Add optional dependency handling for audio/vision extras (catch RuntimeError from torchcodec/PIL)
• Add comprehensive test coverage in tests/unit/backends/vllm_python/test_vllm.py
• Add new guide docs/guides/vllm-offline-backend.md
• Update docs/guides/backends.md with offline backend documentation
• Register vllm_offline backend type in Backend registry
• Update test_backend.py with offline backend registration test

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes code generated or substantially modified by an AI agent
• Includes tests generated or substantially modified by an AI agent

All commits include appropriate Co-Authored-By trailers as described in DEVELOPING.md.

---

git log

commit ff7f125
Author: Maryam Tahhan [email protected]
Date: Fri Jul 17 14:30:31 2026 +0100

Add vLLM Offline Backend for batch processing

Introduces VLLMOfflineBackend inheriting from VLLMPythonBackend,
using vLLM's synchronous LLM engine for micro-batched inference.
Wires RequestStateStats metrics into response timings. Includes
registration test, docs, and backends.md update.

Signed-off-by: Maryam Tahhan <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Maryam Tahhan <[email protected]>

commit fd96103
Author: Maryam Tahhan [email protected]
Date: Fri Jul 17 15:33:26 2026 +0100

Fix tokenizer access path for vLLM V1 LLM engine

Use llm.get_tokenizer() instead of llm.llm_engine.tokenizer.tokenizer
which doesn't exist in the V1 engine.

Signed-off-by: Maryam Tahhan <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Maryam Tahhan <[email protected]>

commit 253b08f
Author: Maryam Tahhan [email protected]
Date: Fri Jul 17 15:41:25 2026 +0100

Fix formatting, metrics, and mdformat issues

- Apply ruff format to offline.py
- Fix _wire_vllm_metrics: only use token counts, not monotonic
  timestamps that produce garbage when mixed with wall-clock times
- Apply mdformat to markdown docs

Signed-off-by: Maryam Tahhan <[email protected]>
Co-Authored-By: Claude Opus 4.6 <[email protected]>
Signed-off-by: Maryam Tahhan <[email protected]>

---

Co-Authored-By: Claude Opus 4.6 [email protected]
Signed-off-by: Maryam Tahhan [email protected]
Signed-off-by: Maryam Tahhan [email protected]

[mergify]

Hi @maryamtahhan, the DCO check has failed. Please click on DCO in the Checks section for instructions on how to resolve this.

[maryamtahhan]

@sjmonson @jaredoconnell this PR has been rebased and is green again

[sjmonson]

Don't pass modules as arguments. Just move from guidellm.extras import vllm out of the TYPE_CHECKING block. guidellm.extras builds lazy stubs so it won't actually import vLLM until you call something on the module.

[sjmonson]

Its funny this is not a linting error.

Suggested change

	if TYPE_CHECKING:
	from guidellm.extras import vllm
	else:
	from guidellm.extras import vllm
	from guidellm.extras import vllm

[sjmonson]

So poking around the vLLM docs a bit it looks like vLLM will tell us a lot about the request from what it returns. Maybe take a look at batched_req.result.metrics which is a RequestStateStats and wire all of the latencies to the correct metrics. You can get pretty much all of the latency metrics we normally track. Start/end time I would probably keep how you currently have them though since that is what the "user" sees as start/end.

[sjmonson]

I didn't check too closely but it seems like even with common.py the normal and offline backends share a lot of code. Would it make more sense if the base class was the other vLLM backend? E.g.

Suggested change

	@Backend.register("vllm_offline")
	class VLLMOfflineBackend(Backend):
	@Backend.register("vllm_offline")
	class VLLMOfflineBackend(VLLMPythonBackend):

Either way I think you should keep common.py as it makes the classes easier to read.

[dbutenhof]

I think the code is mostly OK (except it should be updated to match the current backend init conventions). The documentation needs to be updated to 0.7 CLI conventions.