When is fine-tuning recommended or necessary?

[tankylz]

Hi devs and community, I've followed the example files on fine-tuning and tinkered with some of the hyperparameters, but my observation is that fine-tuning on ~2k materials science data (or a subset of it for cross-validation) led to comparable performance with the pre-trained models. I am wondering generally, is this expected, or are there some key parameters I am supposed to explore? Does it also makes sense to fine-tune and then perform ICL on the same dataset when using it later? Appreciate your advice on the best practices for fine-tuning. Many thanks in advanced!