Skip to content

vllm: restore ParoQuant compatibility with vLLM 0.25.1#54

Open
lesj0610 wants to merge 2 commits into
z-lab:mainfrom
lesj0610:agent/vllm-0.25.1-compat
Open

vllm: restore ParoQuant compatibility with vLLM 0.25.1#54
lesj0610 wants to merge 2 commits into
z-lab:mainfrom
lesj0610:agent/vllm-0.25.1-compat

Conversation

@lesj0610

@lesj0610 lesj0610 commented Jul 16, 2026

Copy link
Copy Markdown

Problem

ParoQuant models cannot be served with vLLM 0.25.1. Plugin discovery fails while importing the removed AWQMarlinLinearMethod, so the paroquant quantization method is never registered. Registration also imports the generator and CUDA extension before a ParoQuant linear layer is created.

Root cause

The backend still targets the vLLM 0.19.x AWQ-Marlin API and dependency range. Its fused-projection conversion also manages Marlin state outside the current weight-loader and layerwise-reload contracts, which can replace tensor storage referenced by torch.compile or a captured CUDA graph.

Fix

  • Update the optional dependency to vllm>=0.25.1,<0.26 and align the README, generated model-card instructions, and Docker chat/serve build pins with vLLM 0.25.1.
  • Keep general-plugin discovery lazy; load the CUDA extension only when ParoQuantLinearMethod is instantiated.
  • Port the backend to AutoAWQMarlinLinearMethod, weight-loader v2, and current Marlin tile-padding and input-dtype selection.
  • Process each fused logical projection with an independent current Marlin kernel, preserve partition order, and apply the fused bias once.
  • Store transformed multipart tensors on the owning linear layer and preserve their pointers across layerwise reload.
  • Detect unquantized checkpoint leaves from exact *.qweight metadata before applying vLLM's model mapper.
  • Reject unsupported tensor-parallel grouping and multipart model-weight offloading instead of silently selecting an invalid fallback.

No vLLM source, package, or environment changes are required.

Verification

  • python -m pytest -q tests/test_vllm_plugin.py tests/test_vllm_plugin_gpu.py: 19 passed.
  • GPU coverage includes single and 2/3/4-part fused Marlin output, bias handling, two layerwise reloads, stable parameter/buffer/kernel/workspace pointers, no torch.compile recompilation, and replay of a CUDA graph captured before reload.
  • Compatibility was validated against the vLLM 0.25.1 interfaces and the current integration tree.
  • An end-to-end ParoQuant checkpoint load and chat-completion smoke test returned HTTP 200.
  • ruff format --check, ruff check, python -m compileall -q, and git diff --check passed for the changed files.

@lesj0610
lesj0610 marked this pull request as ready for review July 16, 2026 05:51
@lesj0610
lesj0610 marked this pull request as draft July 16, 2026 05:54
@lesj0610
lesj0610 marked this pull request as ready for review July 16, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant