ParVL

中文说明

ParVL scales the vision encoder and language model independently inside a multimodal large language model. Each branch reuses the same backbone weights and is distinguished by learnable key/value prefixes; a lightweight token-wise aggregator merges the parallel outputs.

ParVL independently scales P_v shared visual branches and P_l shared language branches. Vector PDF

This repository is a focused research release built on InternVL3.5. It contains the ParVL model implementation, supervised fine-tuning pipeline, local/S3 data loading, DeepSpeed configurations, an inference example, small CPU tests, the complete 1B allocation grid, release-ready 2B/8B model packages, and checkpoint-matched structured training logs. It excludes private datasets, optimizer states, internal cluster launchers, raw cluster/TensorBoard logs, and unrelated InternVL tasks.

Highlights

• Independently configurable vision and language branch counts (P_v:P_l).
• Shared backbone parameters with branch-specific KV prefixes.
• Mean or MLP branch aggregation.
• Full-parameter fine-tuning, frozen-backbone tuning, and Qwen3 LoRA options.
• Optional dense-branch diversity and prefix-orthogonality losses.
• Experimental sparse path-router components retained for follow-up research.

The primary supported path is an InternVL3.5 checkpoint with a Qwen3 language model. The GPT-OSS attention extensions are experimental and require the optional FlashAttention dependencies.

Repository layout

configs/deepspeed/       DeepSpeed ZeRO-1 and ZeRO-3 examples
examples/data/           Dataset metadata and JSONL examples
examples/evaluation/     VLMEvalKit configuration example
internvl/model/          InternVL base and ParVL model implementations
internvl/train/          ParVL SFT entry point and data pipeline
scripts/train_parvl.sh   Portable torchrun launcher
scripts/train_parvl_*b.sh 1B, 2B, and 8B reference presets
scripts/chat.py          Single-image inference example
scripts/validate_release_models.py  Model-package integrity checker
models/                  1B allocation suite plus 2B/8B Hugging Face packages
training_logs/           Public histories for all 11 released checkpoints
tests/                   Tiny CPU forward tests

Installation

Python 3.12, PyTorch 2.6, Transformers 4.55.0, and DeepSpeed 0.17.4 were used for the reference experiments.

python -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt

For packed-sequence training or the experimental custom attention paths:

pip install -r requirements-flash.txt

FlashAttention is compiled against the local CUDA/PyTorch installation. If its build isolation cannot see PyTorch, install it with pip install --no-build-isolation flash-attn.

S3-compatible storage is optional:

pip install -r requirements-storage.txt
export PARVL_STORAGE_CONFIG=/path/to/storage.conf
export CEPH_CLIENT=boto3

Local image and video files are the default and need no storage configuration.

Data format

--meta_path points to a JSON object whose values describe one or more JSONL datasets:

{
  "demo": {
    "annotation": "examples/data/train.jsonl",
    "media_root": "examples/data/images",
    "sample_ratio": 1.0,
    "data_augment": true,
    "max_dynamic_patch": 12
  }
}

Each annotation line contains an image path and an InternVL-style conversation:

{"id": 0, "image": "example.jpg", "conversations": [{"from": "human", "value": "<image>\nDescribe this image."}, {"from": "gpt", "value": "A concise description."}]}

Replace the placeholder annotation and add the referenced images before training. Paths in the supplied launcher are resolved from the repository root.

Training

Use one of the scale presets for the reference branch layout:

META_PATH=examples/data/meta.json \
GPUS_PER_NODE=8 \
GLOBAL_BATCH_SIZE=512 \
bash scripts/train_parvl_1b.sh

META_PATH=examples/data/meta.json \
GPUS_PER_NODE=8 \
GLOBAL_BATCH_SIZE=512 \
bash scripts/train_parvl_2b.sh

META_PATH=examples/data/meta.json \
GPUS_PER_NODE=8 \
GLOBAL_BATCH_SIZE=512 \
bash scripts/train_parvl_8b.sh

Preset	Base checkpoint	P_v:P_l	Vision/LLM prefix	DeepSpeed	Model grad checkpoint
1B	OpenGVLab/InternVL3_5-1B-Pretrained	4:4	96 / 48	ZeRO-1	off
2B	OpenGVLab/InternVL3_5-2B-Pretrained	2:2	96 / 48	ZeRO-1	off
8B	OpenGVLab/InternVL3_5-8B-Pretrained	2:2	96 / 48	ZeRO-3	on

The presets enable FlashAttention by default. Install requirements-flash.txt, or set USE_FLASH_ATTN=false to use eager attention. Override MODEL_PATH to use a local checkpoint.

For multi-node training, additionally set NODE_RANK, MASTER_ADDR, and MASTER_PORT. GLOBAL_BATCH_SIZE must be divisible by NNODES * GPUS_PER_NODE * PER_DEVICE_BATCH_SIZE.

All presets share the paper recipe: one epoch, 448-pixel inputs, up to 12 dynamic tiles plus a thumbnail, cosine decay, 3% warmup, weight decay 0.05, BF16, learning rate 8e-5 for new prefix/aggregator parameters, and 8e-6 for backbone parameters.

For custom branch layouts, call scripts/train_parvl.sh directly. It uses environment variables for machine-specific values and passes additional command-line arguments through to the trainer:

MODEL_PATH=/path/to/checkpoint \
META_PATH=/path/to/meta.json \
VISION_BRANCHES=4 \
LLM_BRANCHES=2 \
bash scripts/train_parvl.sh

The main scale controls are:

• VISION_BRANCHES: number of parallel vision branches.
• LLM_BRANCHES: number of parallel language branches.
• VISION_PREFIX_TOKENS: vision KV-prefix length, default 96.
• LLM_PREFIX_TOKENS: language KV-prefix length, default 48.
• DEEPSPEED_CONFIG: ZeRO configuration path.
• GRAD_CHECKPOINT: model-level gradient checkpointing toggle.

Values 1, 2, and 4 are the configurations used by the research code. Set a side to 0 to use the original non-ParVL implementation for that side.

Training logs

training_logs/ contains the exact Trainer history through step 7000 for every released 1B, 2B, and 8B checkpoint: 497,000 source records in total. Each model has a loss/LR/gradient-norm CSV, a path-free allowlisted training configuration, checksums, and the complete trainer_state.json encoded as deterministic gzip. Runs that recorded branch diagnostics also include a convenient per-step mean CSV; the unaveraged microbatch values remain in the compressed source JSON.

The published checkpoint was selected at approximately 0.78 epoch from a one-epoch schedule. All checkpoint-producing runs recorded ZeRO-1; the public 8B launcher uses ZeRO-3 as a more memory-conservative reference preset. See the log documentation for exact per-model settings and the exclusion rationale, then validate the complete bundle with:

python scripts/validate_training_logs.py

Inference

Use one of the packaged release models or point the example at another saved checkpoint:

python scripts/chat.py \
  --model models/ParVL-1B \
  --image /path/to/image.jpg \
  --question "What is unusual in this image?"

Evaluation

The reported results were produced with VLMEvalKit using deterministic generation. The paper evaluates nine image benchmarks:

Group	VLMEvalKit dataset keys
General	MMMU_DEV_VAL
Math	MathVista_MINI, MathVision_MINI, LogicVista, WeMath
OCR	ChartQA_TEST, TextVQA_VAL, DocVQA_VAL, OCRBench

Math and OCR are the equal-weight means of their four constituent benchmarks, and All is the equal-weight mean of all nine benchmarks. The paper first uses deterministic answer extraction; unresolved MMMU, MathVista, MathVision, and WeMath responses, together with all LogicVista responses, use gpt-4o as a fixed answer extractor rather than as a response-quality judge. The four OCR-group benchmarks use deterministic scoring only.

The release includes a ready-to-edit VLMEvalKit config. Run the following commands from the ParVL repository root so its relative model path resolves correctly:

git clone https://github.com/open-compass/VLMEvalKit.git /path/to/VLMEvalKit
pip install -e /path/to/VLMEvalKit

export OPENAI_API_KEY=your_key
python /path/to/VLMEvalKit/run.py \
  --config examples/evaluation/vlmevalkit_parvl_1b.json \
  --judge gpt-4o \
  --work-dir eval_outputs

Change the config's model name and model_path to evaluate another released 1B allocation or the 2B/8B packages. For multi-GPU data-parallel inference, launch the same command with torchrun --nproc-per-node=N. Pin the VLMEvalKit revision when reporting results because dataset post-processing can change between toolkit versions.

Tests

The tests instantiate tiny Qwen3-ParVL and InternViT-ParVL models and run CPU forward passes; they do not download checkpoints.

pip install -r requirements-dev.txt
pytest -q

Released models

The models/ directory contains 11 independently uploadable Hugging Face model folders: every point in the paper's 1B allocation grid and the 2B/8B main-table models.

1B model	P_v:P_l	Vision/LLM prefix	Weights	All
ParVL-1B-Pv1-Pl1	1:1	n/a / n/a	2.12 GB	49.6
ParVL-1B-Pv1-Pl2	1:2	n/a / 48	2.14 GB	49.9
ParVL-1B-Pv1-Pl4	1:4	n/a / 48	2.15 GB	50.3
ParVL-1B-Pv2-Pl1	2:1	96 / n/a	2.14 GB	49.8
ParVL-1B-Pv2-Pl2	2:2	96 / 48	2.16 GB	49.4
ParVL-1B-Pv2-Pl4	2:4	96 / 48	2.18 GB	49.9
ParVL-1B-Pv4-Pl1	4:1	96 / n/a	2.17 GB	50.1
ParVL-1B-Pv4-Pl2	4:2	96 / 48	2.18 GB	49.3
ParVL-1B	4:4	96 / 48	2.20 GB	50.5

ParVL-1B is the canonical 4:4 model. The code denotes an original single branch with 0, while public names use the paper's P=1 notation.

Larger model	P_v:P_l	Vision/LLM prefix	Weight layout	All
ParVL-2B	2:2	96 / 48	1 shard, 4.75 GB	54.7
ParVL-8B	2:2	96 / 48	4 shards, 17.16 GB	63.0

Each package contains BF16 SafeTensors weights, tokenizer assets, the matching Transformers remote code, an Apache-2.0 license, a model card, and checksums. Resume-only state such as DeepSpeed optimizer shards, RNG snapshots, and the pickled training-arguments cache is deliberately excluded from the model folders. Checkpoint-matched Trainer histories and sanitized training settings are published separately in training_logs/. See models/README.md for model inventory and upload guidance.

Validate all packages without allocating the tensors:

python scripts/validate_release_models.py
python scripts/validate_release_models.py --check-hashes

Scope and reproducibility

This release does not redistribute training data or benchmark evaluation suites. Results depend on global batch construction, hardware count, FlashAttention/CUDA versions, and the exact data mixture.

License and attribution

The ParVL source release is provided under the MIT license in LICENSE. The model packages and their InternVL3.5/Qwen3 components are provided under Apache-2.0; each package includes its own LICENSE. Vendored Qwen3 configuration/modeling files retain their Apache-2.0 headers, with a copy in licenses/Apache-2.0.txt. The project is derived from OpenGVLab InternVL; attribution details are recorded in NOTICE. Third-party datasets and dependencies remain subject to their own licenses.