[question] how does one use the SFTTrainer with VLMs for prompt completion task?

[osaidr]

I have a visual question answering task that I want to train a VLM for using SFT. I want to train the VLM only on the completions and not on the prompt itself.

1. How do I use the SFTTrainer for that? For text only tasks, I can use the prompt-completion dataset type and offload everything to the SFTTrainer. Is that possible for multimodal datasets and VLMs? I went through the source code and I believe it should work fine but wondering if that's not the case.
2. Is it possible to avoid writing the collator? I believe this is a huge can of worms that I want to avoid as much as possible.

[qgallouedec]

You can use the SFTTrainer exactly as you do for text only task :)

[qgallouedec]

A toy example:

from trl import SFTTrainer
from datasets import load_dataset

dataset = load_dataset("trl-internal-testing/zen-image", "conversational_prompt_completion", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-VL-3B-Instruct",
    train_dataset=dataset,
)
trainer.train()

[osaidr]

makes sense, thank you! just to clarify the data should have the columns: prompt, completion and images. images should be a list of PIL Image and prompt, completion are in conversation format right?

the confusion I have is here it's mentioned:

The "content" field in messages must be a list of dictionaries, where each dictionary specifies the type of data: "image" or "text"

but in the trl-internal-testing/zen-image dataset it seems like content is just a text and not a dictionary. does both work?

[qgallouedec]

Actually both are supported. If you pass

{
    "image": [<PILImage>],
    "prompt": [
        {"role": "user", "content": "What color is the sky?"},
    ],
    "completion": [
        {"role": "user", "content": "It is blue."},
    ],
}

it will be converted to:

{
    "image": [<PILImage>],
    "prompt": [
        {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What color is the sky?"}]}
    ],
    "completion": [
        {"role": "user", "content": "It is blue."}
    ],
}

but you can pass directly the later if you want more control on where the image is in the prompt

[osaidr]

That's perfect, thank you!