Can provide reference wandb training log?

[DylanLIiii]

Currently I use default setting training the model on a 8xH20 GPUs. But the training log is strange. In the first eval, my model had an accuracy close to 0.3. Then it dropped to 0 in the next round and gradually increased. Can anyone provide a reference training log so that I can better understand what wrong is?

[Nagi-ovo]

Hi, Did you use torch.compile ?

[DylanLIiii]

Hi, Did you use torch.compile ?

No, I didn't use torch.compile. I kept the default configuration from the repo, only adjusting the batch size to 32 per GPU to maintain a global batch size of 256 (guess it is global one).

[Jiajun-Chu]

Hi, I'm encountering the same issue. The accuracy started dropping from the beginning.

In my first two trials, I used the default settings, but I noticed that both batch_loss and val_loss increased unexpectedly. To address this, I applied gradient clipping in the next two trials, which smoothed out the loss curve. However, it didn't improve the accuracy.

[pdh930105]

I have a similar issue.

I tested 3 trial. (8 RTX3090 GPU 5 epoch (total BS=64x8), 8 RTX3090 15 epoch (total BS=64x8), 1 RTX3090 GPU 15 epoch (total BS=64))

In my assumption, DDP has breaked accuracy and overfitting.

[lusxvr]

You are right, we are also seeing that the accuracy is quite sensitive to individual runs. This might be because the current evaluations if more 'hacky' than production-ready. In general I would advise to only compare runs with the same global batch size, and focus more on the loss curves (train/val), to gauge how the performance changes. If you use the provided config of the model (e.g. #75), with the same learning rate and global batch size, you should hopefully see similar accuracies.