DiceScore yields 1.0 scores when using average="none" for samples where class is not present

[nkaenzig]

🐛 Bug

The current implementation of DiceScore yields scores of 1.0 for samples that don't contain a particular class when calculating class-wise metrics via average="none".

This leads to very high dice scores, particularly for rare classes, as samples where the class is not present will push the metric towards 1.0. It also makes dice scores among classes incomparable, unless the val/test dataset is balanced.

To Reproduce

The following code snipped shows an example where for 1000 samples, the first class is only present in the first sample. Even though the prediction on that first sample is wrong, it yields a close to perfect class dice score of 99.9%, because for the 999 samples where the class is absent, a score of 1.0 is used.

Code sample

import torch
from torchmetrics.segmentation import DiceScore

N_SAMPLES = 1000
N_CLASSES = 3

target = torch.full((N_SAMPLES, N_CLASSES, 128, 128), 0, dtype=torch.int8)
preds = torch.full((N_SAMPLES, N_CLASSES, 128, 128), 0, dtype=torch.int8)

# The first class is only present in the first sample, but prediction is wrong
target[0, 0] = 1 

dice = DiceScore(num_classes=3, average='none', include_background=True)

print(dice(preds, target))
# prints: tensor([0.9990, 1.0000, 1.0000])

Expected behavior

The above code snipped should return [0.0, nan, nan] (nan for second & third classes because they are not present in any of the provided samples, therefore no meaningful score can be calculated).

Environment

• TorchMetrics version (if build from source, add commit SHA): 1.6.0
• Python & PyTorch Version (e.g., 1.0): Python 3.11.10 & torch==2.3.0
• Any other relevant information such as OS (e.g., Linux): macOS 15.1.1 (24B91)

Additional context

• Very similar to issue GeneralizedDiceScore yields 0 scores when using per_class=True for samples where class is not present #2846 for GeneralizedDiceScore, but there 0.0 scores for samples where class is missing are used.
• The MONAI library addresses this through the ignore_empty keyword argument, which is set to True by default (see monai.metrics.DiceMetric)