Adding an additional trainable module for sft

[ymliu529]

Reminder

• I have read the above rules and searched the existing issues.

System Info

May I ask if it is possible to add a trainable module to the multimodal large model, such as internvl3, during the fine-tuning of Lora, so that it can be trained together with Lora? If possible, where should I modify which part of the code to implement it?

Reproduction

Put your message here.

Others

No response

[Kuangdd01]

1. Overwrite the class InternVLForConditionGeneration with your custom code and make sure this model class is imported correctly, and the parameters are loaded properly.
2. Assume that you have trained this model, making this additional module adapted, and then you want to use LORA to finetune this custom model further. Set lora_target: all do this finetune.
3. Hope that you do not modify structures of llm and vit. If you did, I suggest you use the LLAVA repo to achieve this goal.