Learning to Defer to Multiple Experts

Code for Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles [AISTATS'23]

Installation

The installation is straightforward using the following instruction, that creates a conda virtual environment named multi_l2d using the provided file environment.yml:

conda env create -f environment.yml

Folder Structure

• CIFAR-10: This folder contains the experiments and results for CIFAR-10 dataset. With respect to the AISTATS paper, it also contains experiments for confifence calibration [7.2] and conformal ensembles [7.3] (Figures 2 and 3).
• Galaxy-Zoo: This folder contains the experiments for Galaxy-Zoo dataset. With respect to the AISTATS paper, it also contains experiments for overall system accuracy [7.1] (Figure 1).
• ham10000: This folder contains the experiments for HAM10000 dataset. With respect to the AISTATS paper, it also contains experiments for overall system accuracy [7.1] (Figure 1).
• Hatespeech: This folder contains the experiments for Hatespeech dataset. With respect to the AISTATS paper, it also contains experiments for overall system accuracy [7.1] (Figure 1).
• HMCat_ICML22: This folder contains the results presented in the ICML 2022 Workshop on Human-Machine Collaboration and Teaming.
• lib: This folder contains the shared code among al the experiments, such as the surrogate losses, the conformal methods and other utils.

Citation

Please, if you use this code, include the following citation:

@inproceedings{multil2d,
  title = {Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles},
  author = {Verma, Rajeev and Barrejon, Daniel and Nalisnick, Eric},
  booktitle = {International Conference on Artificial Intelligence and Statistics (AISTATS)},
  pages = {11415--11434},
  year = {2023},
  series = {Proceedings of Machine Learning Research},
  month = {25--27 Apr},
  publisher = {PMLR},
  url = {https://proceedings.mlr.press/v206/verma23a.html},
}

Contact

Please, do not hesitate to contact: [email protected] and [email protected]