`ModelCheckpoint` not saving best model

[sravan953]

Bug description

In DDP, the ModelCheckpooint (configuration below) does not save the best model despite lower validation losses being achieved in later epochs. Configuration:

checkpoint_callback = ModelCheckpoint(
        dirpath=path_save_model,
        filename="best_loss",
        monitor="val_loss",
        mode="min",
        every_n_epochs=1,
        verbose=True,
    )

Here is a snippet of the output log:

Epoch 292: 100% 40/40 [00:49<00:00,  1.24s/it, v_num=0, l1=0.00589, msssim=0.00441, lpips=0.0184, loss=0.0287, lr=0.000128, val_l1=0.00746, val_msssim=0.0122, val_lpips=0.0299, val_loss=0.0495]Epoch 292, global step 11720: 'val_loss' reached 0.04955 (best 0.04955), saving model to '/qumulo/sravan/Projects/pmc_lit/experiments/e3_uwsyn_multi_noprompt/250318_2110_30k_uw50pc_ghosting/best_loss.ckpt' as top 1

Epoch 297: 100% 40/40 [00:49<00:00,  1.24s/it, v_num=0, l1=0.0141, msssim=0.00869, lpips=0.0113, loss=0.0341, lr=0.000124, val_l1=0.00779, val_msssim=0.0121, val_lpips=0.0266, val_loss=0.0466]Epoch 297, global step 11920: 'val_loss' was not in top 1

Clearly, epoch 297 should be saved to disk

What version are you seeing the problem on?

v2.5

How to reproduce the bug

Error messages and logs

Epoch 292: 100% 40/40 [00:49<00:00,  1.24s/it, v_num=0, l1=0.00589, msssim=0.00441, lpips=0.0184, loss=0.0287, lr=0.000128, val_l1=0.00746, val_msssim=0.0122, val_lpips=0.0299, val_loss=0.0495]Epoch 292, global step 11720: 'val_loss' reached 0.04955 (best 0.04955), saving model to '250318_2110_30k_uw50pc/best_loss.ckpt' as top 1

Epoch 297: 100% 40/40 [00:49<00:00,  1.24s/it, v_num=0, l1=0.0141, msssim=0.00869, lpips=0.0113, loss=0.0341, lr=0.000124, val_l1=0.00779, val_msssim=0.0121, val_lpips=0.0266, val_loss=0.0466]Epoch 297, global step 11920: 'val_loss' was not in top 1

Environment

Current environment

#- PyTorch Lightning Version (e.g., 2.5.0): 2.5.0
#- PyTorch Version (e.g., 2.5): 2.3.1+cu121
#- Python version (e.g., 3.12): 3.10.12
#- OS (e.g., Linux): Ubuntu 22.04.3 LTS
#- CUDA/cuDNN version: 
#- GPU models and configuration: 
#- How you installed Lightning(`conda`, `pip`, source): pip

More info

No response

[Borda]

@sravan953 could you please share a full example to reproduce?

[sravan953]

I am unable to reproduce this issue, though I can swear it was an issue for a couple of days. Thank you for the quick response, closing this now!

[sravan953]

Hello. I have been facing this issue on and off over many runs on multiple private datasets when training U-Nets. For example:

Epoch 64: 100% 131/131 [01:58<00:00,  1.11it/s, v_num=0, l1=0.00829, msssim=0.00589, lpips=0.0279, loss=0.0421, lr=0.0002, val_l1=0.00504, val_mse=0.000264, val_msssim=0.00295, val_lpips=0.00748, val_loss=0.016]
Epoch 64, global step 8515: 'val_loss' reached 0.01598 (best 0.01598), saving model to 'best_loss.ckpt' as top 1
.
.
.
Epoch 240: 100% 131/131 [01:58<00:00,  1.11it/s, v_num=0, l1=0.00897, msssim=0.0023, lpips=0.0136, loss=0.0249, lr=1.72e-5, val_l1=0.653, val_mse=3.34e+3, val_msssim=0.00215, val_lpips=0.0119, val_loss=0.0107]
Epoch 240, global step 31571: 'val_loss' was not in top 1

Note that val_mse is only logged but not summed with other losses to obtain the final val_loss. Any ideas as to where I should look? Clearly the epoch 240 val_loss < epoch 64 val_loss.