ML-Reproducibility-2020-DUQ

[RE] Uncertainty Estimation Using a Single Deep Deterministic Neural Network

This repository is the reproduction of paper "Uncertainty Estimation Using a Single Deep Deterministic Neural Network" by Joost van Amersfoort, Lewis Smith, Yee Whye Teh, Yarin Gal.

All codes for training and experiments are provided. They are as colab notebooks. Additionally, py files are provided for training and testing. Training and models are based on codes provided by the author

Requirements

Colab notebooks

All requirements will be self-installed

py files

To install requirements:

pip install -r requirements.txt

All datasets will be downloaded from torchvision except for notMNIST
Download it from "http://yaroslavvb.com/upload/notMNIST/notMNIST_small.mat" and place it in "data" folder:

mkdir -p data && cd data && curl -O "http://yaroslavvb.com/upload/notMNIST/notMNIST_small.mat"

Running Experiments and Training, Testing

Colab notebooks

• Open a notebook
• Place the utils folder (only requirement)
• Place the trained models, specify the location of trained model in code (for experiments)
• Run the notebook

py files

• Create a virtual env
• Download the requirements
• Change hyper parameters and Run the python files

Training: will be done on FMnist or CIFAR-10
Testing: Accuracy will be calculated on testset of FMnist or CIFAR-10
Testing: Auroc-ood will be calculated on FMnist+Mnist-ood or CIFAR10+SVHN-ood

Trained models are required to execute the experiment codes
Experiment codes are self explanatory (includes uncertainty Histograms, rejection plots, ROC curve, aleatoric plots, uncertainty maps)

Pre-trained Models

You can download pretrained models here:

• DUQ and DE trained on FashionMNIST, CIFAR10 with paper's default parameters

Results

Performance of the model with our implementation at a glance:

FashionMNIST, Mnist-ood

Model name	Accuracy on FM	Auroc-ood(M)	Train, Test Time*
DUQ with gp	92.13%	0.947	23s and 1s
DUQ + our work	92.35%	0.964	23s and 1s
DE	93.30%	0.889	9x5s and 2.3s

CIFAR10, SVHN-ood

Model name	Accuracy on CIFAR10	Auroc-ood(M)	Train, Test Time*
DUQ with gp	93.45%	0.931	210s and 4s
DE	94.44%	0.949	60x5s and 14s

Reference

@misc{vanamersfoort2020uncertainty,
      title={Uncertainty Estimation Using a Single Deep Deterministic Neural Network}, 
      author={Joost van Amersfoort and Lewis Smith and Yee Whye Teh and Yarin Gal},
      year={2020},
      eprint={2003.02037},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}