Calibration

a python package for calibrating the deep learning models

Install

pip install calibrator

Use case

from calibrator import LocalCalibrator
import torch

val_logits = torch.randn(1000, 10)
val_labels = torch.randint(0, 10, (1000,))
test_logits = torch.randn(1000, 10)

calibrator = LocalCalibrator()
eps_opt = calibrator.fit(val_logits, val_labels)
calibrated_probability = calibrator.calibrate(test_logits)

Benchmarking

Datasets	Method	Method Description	ECE	ACE	MCE	CECE	PIECE
CIFAR-10
CIFAR-100
ImageNet-1K

Package Information

Post-Hoc Calibration methods

Calibration methods	Description	Paper	Source Code	Status
Temperature Scaling (TS)	ICML 2017	paper	code	✅ Implemented
Parameterized Temperature Scaling (PTS)	ECCV 2022	paper	code	🔜 Pending
Ensemble Temperature Scaling (ETS)	ICML 2020	paper	code	🔜 Pending
Class-based Temperature Scaling (CTS)	EUSIPCO 2021	paper	unavailable	🔜 Pending
Group Calibration with Temperature Scaling (GCTS)	NeurIPS 2023	paper	code	🔜 Pending
Proximity-informed Calibration (PROCAL)	NeurIPS 2023	paper	code	🔜 Pending
Isotonic Regression
Histogram Binning
Platt Scaling
Bayesian Binning into Quantiles (BBQ)	AAAI 2015	paper		🔜 Pending
BetaCal
Scaling-Binning Calibrator	NeuIPS 2019	paper	code	🔜 Pending
Dirichlet calibration	NeurIPS 2019	paper	code	🔜 Pending

Train-time Calibration Methods

Calibration Losses	Description	Paper	Source Code	Status
Focal Loss
Dual Focal Loss
Adaptive Focal Loss

Metrics

Metrics	Description	Paper	Source Code	Status
Expected Calibration Error (ECE)	AAAI 2015	paper	code	✅ Implemented
Maximum Calibration Error (MCE)	AAAI 2015	paper	code	🔜 Pending
Adaptive Calibration Error (ACE)	CVPRW 2019	paper	code	✅ Implemented
Classwise Expected Calibration Error (CECE)	NeurIPS 2019	paper	code	✅ Implemented
Negative Log Likelihood (NLL)
Accuracy
Proximity-informed Expected Calibration Error (PIECE)	NeurIPS 2023	paper	code	🔜 Pending

Pre-trained Model Weights

Datasets	Description	Paper	Source Code	Status
CIFAR-10
CIFAR-100
ImageNet
ImageNet-100
ImageNet-1000

Pre-calculated Logits

Datasets	Description	Paper	Source Code	Status
CIFAR-10
CIFAR-100
ImageNet
ImageNet-100
ImageNet-1000