Why does the loss fluctuate so drastically?

[Crnake]

Here are the training parameters I used： --lr "7.048e-4" \ --aug-cfg scale='(0.8, 1.0)' color_jitter='(0.32, 0.32, 0.32, 0.08)' color_jitter_prob=0.8 gray_scale_prob=0.2 \ --warmup 1000 \ --wd 0.2 \ --batch-size 512\ --epochs=5 \ --workers=2 \ --local-loss \ --gather-with-grad \ --force-image-size 224 \ --log-every-n-steps 10 \ --seed 0 \ --lock-image \ --train-num-samples 12100000 \ --report-to 'tensorboard' \ --imagenet-val $imagenet_val \ --grad-clip-norm 1.0 \ Global BatchSize is 2048, I'm curious why the loss is so volatile. Is it because the learning rate is too high? The optimizer I use is AdamW.c Additionally, as a newbie, I'm quite curious whether the training epoch here should be set a bit higher, like 20. And As far as I understand, a logit scale reaching 100 generally indicates complete convergence, but my logit scale only goes up to 80 and doesn't increase further. Is this due to model capacity or data quality?

[rwightman]

Could be any number of things, though usually it is data quality and/or learning rate. It's a small dataset for contrastive training, comparable to training CC12m so yeah, 32-64 epochs, probably some increased image augmentations would benefit. Going to move to discussion as isn't a bug to address.