Skip to content

Multi-Score Paradigms #947

@davisethan

Description

@davisethan

Problem: MOABB assumes paradigms use one score for evaluations. However, besides discrimination (e.g. accuracy, AUROC) measuring calibration (e.g. NLL, Brier score, ECE) is also useful.

Proposed Solution: Allows custom paradigms to use multiples scorers. The parameter additional_columns from evaluations is used for multiple scores in results. The column score in results is kept and used as a register for statistical analysis and plotting.

Google Colab: https://colab.research.google.com/drive/1gFDBIfdbAWl2UdvslxVnLFhV7qws6_qm?usp=sharing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions