Recommended fine-tuning recipe for 2b model

[zhuqiangLu]

For those who have successfully fine-tune the 2b model, could you please share your fine-tuning recipe?

I was trying fine-tune the 2b model with 20k images with --pn 0.06M and the result is not satisfying. Also, I noticed the learning rate in the training script is set to 0.006, I was wondering perhaps this value is way too high for fine-tuning.

[znavidi]

@zhuqiangLu
Have you seen any improvements when fine-tuning the 2B model? Were there any configuration settings that particularly helped during fine-tuning? Would appreciate if you could share.

[zhuqiangLu]

Unfortunately, no.
But there are some tricks that may help.

The first thing is disabling data augmentation. Strange to me, the official data loader doesn't come with augmentation. So I added a simple augmentation as my custom dataset is a bit small. Turns out the accs can reach 80.x without augmentation.

Tuning the model with 0.25M and 1M can converge much faster. Honestly I am not sure about this one. However, tuning the model with the official toy dataset and set pn to 0.06M do not get you a good result. Even the acct/accm get to 90, the synthesized image is still terrible (using the text prompt the toy example)

[znavidi]

@zhuqiangLu Thanks for sharing :)