how to finetuning a bi-encoder embedding model of multimodel input

[fancyerii]

I want to cluster ecommerce products by bi-encoder. For each product, it has a name(text) and an image. Can I use sentence-transfomer to finetune a bi-encoder model? The training dataset contains product clusters, like:

product1_name, product1_img, cluster_id1
product2_name, product2_img, cluster_id1
product3_name, product3_img, cluster_id2

productm_name,productm_img, cluster_idn

I want to try first to define it as a classification problem(cluster_id1,...cluster_idn) and use arcface loss. But If there are other suitable losses, it's also fine.

Is sentence transformer suitable for my use case? I find siglip(something like clip) is good at embedding. Its training data is image/text pair, but my data is not the same as it.

[tomaarsen]

Hello!

Sentence Transformers has some multimodality support, currently only supporting CLIP models, although more will be added quite soon with #3554. You can use any clip-tagged model on Hugging Face like so:

from sentence_transformers import SentenceTransformer
from sentence_transformers.models import CLIPModel

clip_model = CLIPModel("openai/clip-vit-large-patch14")
model = SentenceTransformer(modules=[clip_model])

text = "Stylish summer dress with floral patterns" # Can also be a list of texts
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw) # Can also be a list of images

text_embedding = model.encode(text)
image_embedding = model.encode(image)

# Simple cosine similarity between text and image embeddings
similarity = model.similarity(text_embedding, image_embedding)

Apart from loading the model (CLIPModel -> SentenceTransformer), the training approach is identical to training with texts, and this documentation should be very helpful in that regard: https://sbert.net/docs/sentence_transformer/training_overview.html
Alternatively, examples such as this recently proposed one might be useful to get you started: #3595 with https://github.com/aardoiz/sentence-transformers/blob/main/examples/sentence_transformer/training/clip/training_clip_flickr8k_mlflow.py

Sentence Transformers doesn't implement an ArcFace loss out of the box. The most commonly used loss is MultipleNegativesRankingLoss, also known as InfoNCE loss in the literature. Given a dataset with 2 columns: product texts and product images, the loss will first embed all data. Then, for each product text (the 'anchor'), the loss rewards a high similarity between product text $i$ and product image $i$ (the positive), while punishing high similarities between product text $i$ and product image $j$ with $j$ != $i$ (the in-batch negatives). In short, the other images in the batch are assumed to be unrelated & low similarity.

• Tom Aarsen