Fine-Tuning UForm models

[wematan]

Hi,

Thank you for your excellent work on UForm. Are there plans to release fine-tuning scripts for the models in this repository? Such resources would be immensely helpful for adapting the models to specific tasks.

Thank you!

[alexisdrakopoulos]

Any update on this? Would be fantastic to be able to fine-tune these.

[dame-cell]

@wematan @alexisdrakopoulos hey guys I hope I'm not late when you mention fine=tuning do you mean like the vision language models or the tiny embeddings models

[alexisdrakopoulos]

Hi @dame-cell , I mean the vision models. I'm looking to build an image to image / multimodal retrieval system for a large custom dataset of images. I have several million examples to tune on. My use case is quite custom, I'm investigating just using CLIP but since you guys claimed such high retrieval performance with less data I had hoped to use a similar approach.

[dame-cell]

Yea this is actually possible , but since you need it only for the image to image and no text needed , you can actually only finetune the vision part of the model on your own custom dataset without finetuning the entire model.

I can provide a notebook but probably not today maybe Tomorrow.

let me know if this make sense.

[alexisdrakopoulos]

Wow that would be fantastic! I'd love to experiment with it. I was being a little vague but let me explain. I have a large dataset (2 - 4 million) images with associated textual descriptions. These are images of objects. I want to search this database for objects by providing an image of another object. I can also write a textual descriptions of the object if it helps improve retrieval performance.

…

On Tue, May 6, 2025, 17:10 cell-dame ***@***.***> wrote: *dame-cell* left a comment (unum-cloud/uform#89) <#89 (comment)> Yea this is actually possible , but since you need it only for the image to image and no text needed , you can actually only finetune the vision part of the model on your own custom dataset without finetuning the entire model. I can provide a notebook but probably not today maybe Tomorrow. let me know if this make sense. — Reply to this email directly, view it on GitHub <#89 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH4X76MSNSM7PHZPAW6VIRL25DGGJAVCNFSM6AAAAABLTXPYNKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQNJUHEZDKNZXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[dame-cell]

hey there @alexisdrakopoulos I was able to code up the stuff you wanted but its better to also include textual description of the object .
I did this in the span of 2 hours so there might be some mistake but I think it should work , I might look into it again.

https://colab.research.google.com/drive/1tEmvG7haoVtYKp8L-uMbS0G2lJpJSRM0?usp=sharing