Question about fine-tuning

[jimmyparadm]

Hello. I am trying to fine tune the model. I am testing it with using dataset containing images of 2 car models, and 2 labels that are the model numbers. In the train command, I locked the image tower, as I do not want the model to change how it embeds images.

The issue I have is I am not seeing any improvement in training. The performance in Epoch 1 is roughly the same as Epoch 10. I wonder if I set any parameter wrong in the train command. Any suggestions would be greatly appreciated.

Part of the training dataset (roughly 300 pairs, label are either sg2023 or ql2024):

label	path
sg2023	img1.jpg
ql2024	img2.jpg
sg2023	img3.jpg
ql2024	img4.jpg

Command:

python -m open_clip_train.main 
--epochs=100 
--save-frequency 20 
--lock-image
--warmup 5 --batch-size=32 --lr=5e-5 --wd=0.2 
--beta1 0.9 --beta2 0.98 --eps=1e-6 
--model xlm-roberta-large-ViT-H-14 
--pretrained frozen_laion5b_s13b_b90k 
--train-data=".\train_data.csv" --val-data=".\val_data.csv"
--csv-img-key path --csv-caption-key label --csv-separator ","

Result:

2024-09-25,18:27:30 | INFO | Eval Epoch: 1 image_to_text_mean_rank: 19.7143     image_to_text_median_rank: 21.0000      image_to_text_R@1: 0.0143       image_to_text_R@5: 0.1429       image_to_text_R@10: 0.2714       text_to_image_mean_rank: 18.5857        text_to_image_median_rank: 18.0000      text_to_image_R@1: 0.0286       text_to_image_R@5: 0.1429       text_to_image_R@10: 0.2857      clip_val_loss: 2.8442   epoch: 1.0000    num_samples: 70.0000
2024-09-25,20:37:58 | INFO | Eval Epoch: 10 image_to_text_mean_rank: 18.9286    image_to_text_median_rank: 19.0000      image_to_text_R@1: 0.0000       image_to_text_R@5: 0.1286       image_to_text_R@10: 0.2429       text_to_image_mean_rank: 18.1286        text_to_image_median_rank: 18.0000      text_to_image_R@1: 0.0286       text_to_image_R@5: 0.1429       text_to_image_R@10: 0.2857      clip_val_loss: 2.7193   epoch: 10.0000   num_samples: 70.0000