Support for Qwen3-VL models

[zhaohan-alan]

🚀 The feature, motivation and pitch

I’d like to use Liger-Kernel with Qwen3-VL models (e.g. Qwen3-VL-30B-A3B-Instruct). Currently, Qwen3-VL cannot initialize with Liger-Kernel.

It would be very appreciated if Qwen3-VL could be supported, which could significantly reduce GPU memory!

Alternatives

No response

Additional context

No response

[mayankagarwals]

@Tcc0403 I would like to give it a shot!

[John1231983]

Any update @mayankagarwals @Tcc0403. It looks like it works, but it hasn’t been merged into the main branch yet.
I tested it but slows 3 times than off flag :(

[mayankagarwals]

I tested it but slows 3 times than off flag :(

Hi @John1231983 , I've started efforts here. Will keep the PR updated on progress.

For qwen3, FLCE is integrated and all convergence tests are passing + it's a patch of existing kernel. Since its a tradeoff of memory for speed, it is possible that this is expected but i'll need to confirm it once.

[Tcc0403]

@John1231983 would you like to share your training recipe? Or maybe a screenshot of profiling trace? With liger kernel, you should be able to train with larger batch size/longer seqlen.

[John1231983]

@Tcc0403 Sure
First add apply_liger_kernel_to_qwen3_vl to the line
https://github.com/2U1/Qwen-VL-Series-Finetune/blob/master/src/train/train_sft.py#L19

Then, change the line https://github.com/2U1/Qwen-VL-Series-Finetune/blob/master/src/train/train_sft.py#L110 to apply_liger_kernel_to_qwen3_vl()

Finally, change flag to True
https://github.com/2U1/Qwen-VL-Series-Finetune/blob/master/scripts/finetune.sh#L20

You can setup more batch size in the case. Good luck

[Tcc0403]

@John1231983 quick question: what's your environment and config?

The following:

MODEL_NAME="Qwen/Qwen3-VL-4B-Instruct"
GLOBAL_BATCH_SIZE=128
BATCH_PER_DEVICE=4
NUM_DEVICES=8 

with same config in zero3_offload.json?

BTW, have you tried training without offload? with liger it might be possible.

[Tcc0403]

@mayankagarwals Feel free to ping me if you got any problems with the PR

[John1231983]

@Tcc0403 @mayankagarwals any update to support Qwen3VL? Thanks

[mayankagarwals]

Hi @John1231983 Qwen3VL is supported (for FLCE) through this branch . I'm working on mrope which might take some time but you can use it in it's current state and should see benefits. CC @Tcc0403

[VietDunghacker]

When I fine-tune Qwen3-VL with the latest ms-swift framework and transformers==4.57.0, if I turn on liger kernel, the training process will be frozen indefinitely. Can you check it?