Triple Cross-Modal Interaction for Infrared–Visible Abnormal Behavior Recognition
-orange){kind=icon}
Dual-stream visible–thermal video recognition on the MIHAB benchmark. TriCM-Net fuses infrared and visible streams through a lightweight TriCMI block (CCM + CSA + AGF), reaching 87.41% Top-1 with only +0.44 M extra parameters over a single-stream I3D.
Under low light the visible stream collapses to near-zero luminance while the thermal stream lacks fine-grained appearance cues — neither modality alone yields a confident prediction. TriCMI lets the two streams reciprocally condition on each other through channel modulation, spatial attention, and adaptive gating, producing a decisive fused softmax.
TriCMI consists of three complementary sub-modules:
- CCM (Cross-modal Channel Modulation) — FiLM-style affine conditioning drawn from the other modality's global descriptor
- CSA (Cross-modal Spatial Attention) — 3D convolution on a cross-modal mean/max spatial map
- AGF (Adaptive Gated Fusion) — per-voxel soft gate that merges the two conditioned streams
The full TriCMI block adds only 0.44 M parameters (+3.5%) on top of a shared I3D backbone.
| Method | Modality | Top-1 (%) | +Params | Config |
|---|---|---|---|---|
| Visible-only I3D | V | 76.38 | — | configs/baseline_v2/baseline_visible.yaml |
| Thermal-only I3D | T | 74.51 | — | configs/baseline_v2/baseline_thermal.yaml |
| Late Fusion (score avg) | V+T | 80.12 | — | tools/eval/late_fusion.py (post-hoc) |
| Early Stack + MAM | V+T | 80.23 | 0.15 M | configs/baseline_v2/baseline_mam.yaml |
| CMX (T-ITS 2023) | V+T | 74.30 | 1.83 M | configs/baselines_sota/cmx.yaml |
| CAINet (TMM 2024) | V+T | 77.70 | 5.76 M | configs/baselines_sota/cainet.yaml |
| TriCM-Net (Ours) | V+T | 87.41 | 0.44 M | configs/tricm/tricm_final.yaml |
Across seeds {42, 1337, 2024}: 87.13 ± 0.23%.
- Python 3.8+, PyTorch 2.0+, CUDA 11.8+, GPU >= 12 GB
pip install -r requirements.txtThe model imports pytorch_i3d.py from the official I3D repository, so its source code (which also bundles models/rgb_imagenet.pt) must live under pretrained/pytorch-i3d/:
git clone https://github.com/piergiaj/pytorch-i3d.git pretrained/pytorch-i3dSee docs/DOWNLOAD.md for dataset and checkpoint instructions.
Place the MIHAB dataset at ./MIHAB_dataset/datasets_all/ (or update dataset_root in the config).
Extract frames and optical flow before training:
python data/extract_frames.py --dataset_root ./MIHAB_dataset/datasets_all
python data/extract_flow_crop.py --dataset_root ./MIHAB_dataset/datasets_all# TriCM-Net (full model, Focal Loss)
python train.py --config configs/tricm/tricm_final.yaml
# Ablation variants
python train.py --config configs/tricm/tricm_none.yaml # no interaction
python train.py --config configs/tricm/tricm_channel.yaml # CCM only
python train.py --config configs/tricm/tricm_spatial.yaml # CSA only
python train.py --config configs/tricm/tricm_gate.yaml # AGF onlyThe current open-source flow is train from scratch: the commands above produce checkpoints/<name>/best_model.pth, and train.py already prints per-epoch test Top-1 on the held-out 30% video split.
Once a run finishes, tools/eval/eval_ablation.py re-evaluates a saved checkpoint with the test-time loader; tools/eval/compile_results.py aggregates the registry of trained checkpoints into a Markdown table:
python tools/eval/eval_ablation.py --gpu 0 --out ablation_verified.json
python tools/eval/compile_results.pyBoth scripts require local checkpoints. Pretrained checkpoints used in the paper will be released after acceptance (see docs/DOWNLOAD.md); during the anonymous review period, the headline numbers can be reproduced from scratch with the configs above.
TriCM-Net/
├── configs/
│ ├── tricm/ # TriCM-Net & ablation configs
│ ├── baseline_v2/ # single-stream & naive fusion baselines
│ └── baselines_sota/ # CMX, CAINet reproduction configs
├── models/
│ ├── tricm_net.py # TriCM-Net + TriCMI module (CCM + CSA + AGF)
│ ├── single_stream_i3d.py # single-stream I3D baseline
│ ├── stfn_enhanced_stack.py # early-stack baseline (MAM)
│ └── baselines_sota/ # CMX, CAINet reproductions
├── losses/ # Focal Loss, CE variants
├── data/ # MIHAB dataset & preprocessing
├── tools/ # evaluation scripts
├── docs/ # DOWNLOAD.md, images
└── train.py
@inproceedings{tricmnet2026,
title = {TriCM-Net: Triple Cross-Modal Interaction for
Infrared--Visible Abnormal Behavior Recognition},
author = {Anonymous},
booktitle = {PRCV},
year = {2026}
}This project is released under the MIT License.