Multimodal Concept Bottleneck Models

• This is our official repo of Multi-Modal Concept Bottleneck Models (MM-CBMs) presented at NeurIPS 2025 Mechanistic Interpretability Workshop (non-archival).
• Please see our Project Page for a quick overview of our work.

Setup

1. Install Python (3.10) and PyTorch (1.13).
2. Install dependencies by running pip install -r requirements.txt
3. Download and process CUB dataset by running bash download_cub.sh
4. Download and process ImageNet dataset by running bash download_imagenet.sh (Replace the download link)

Evaluate pretrained models

Run MMCBM_evaluation_finetuned.ipynb and MMCBM_evaluation_zero-shot.ipynb to evaluate the pretrained models. Provides both overall accuracy and per-class accuracy. Generates multimodal explanations in textual form for the model’s predictions. Weights can be downloaded from https://drive.google.com/file/d/1myxyqthTE1L4YgEZzjMRabbWjo9sokCQ/view?usp=sharing.

Train your own model (Optional)

1. Extract candidate concepts with GPT and clean concept set: GPT_initial_concepts.ipynb, GPT_conceptset_processor.ipynb (This step will incur expenses.)
2. Text augmentation with LLaMA: label_expand.ipynb
3. Annotate images with object detection: python image_annotation.py --dataset {dataset_name_subset}
4. Train the Concept Bottleneck Model (CBM): python train_cbm.py --dataset {dataset_name} --weight 0.2

Overview

Results

1. Compared with other CBMs

MM-CBM achieves performance comparable to the strongest baseline, VLG-CBM, and surpasses others by over 10% accuracy on ImageNet.

2. Compared with black-box CLIP

Across seven datasets, MM-CBM attains performance comparable to CLIP’s linear-probe and zero-shot results.

3. Interpretable results

We compared two variants of MM-CBM (zero-shot and fine-tuned) against SpLiCE. For each method, participants were shown 100 randomly sampled ImageNet images with the correct label.

Reference

1. Radford et al., Learning transferable visual models from natural language supervision, ICML 2021
2. Oikarinen et al., Label-free Concept Bottleneck Models, ICLR 2023
3. Bhalla et al., Interpreting clip with sparse linear concept embeddings (splice), NeurIPS 2024
4. Srivastava et al., VLG-CBM: Training concept bottleneck models with vision-language guidance, NeurIPS 2024
5. Koh et al., Concept bottleneck models, ICML 2020
6. Liu et al., Grounding dino: Marrying dino with grounded pre-training for open-set object detection, ECCV 2024
7. Song et al., Mpnet: Masked and permuted pre-training for language understanding, NeurIPS 2020.
8. Dubey et al., The llama 3 herd of models, arXiv preprint 2024.
9. Liu et al., Visual instruction tuning, NeurIPS 2023
10. Yan, et al., Learning concise and descriptive attributes for visual recognition, ICCV 2023
11. Yang, et al., Language in a bottle: Language model guided concept bottlenecks for interpretable image classification, CVPR 2023

Cite this work

T. Shi, G. Yan, T. Oikarinen, and T.-W. Weng, Multimodal concept bottleneck models, Preprint 2025.

@misc{shi2025multimodal,
      title={Multimodal Concept Bottleneck Models},
      author={Shi, Tongqing and Yan, Ge and Oikarinen, Tuomas and Weng, Tsui-Wei},
      booktitle={Preprint 2025}
    }