Need Help!! qwen2.5vl7b lora sft with deepspeed zero3

[Luffy966]

Reminder

• I have read the above rules and searched the existing issues.

System Info

[2025-04-03 06:02:20,328] [INFO] [real_accelerator.py:239:get_accelerator] Setting ds_accelerator to cuda (auto detect)
INFO 04-03 06:02:21 [init.py:256] Automatically detected platform cuda.

• llamafactory version: 0.9.3.dev0
• Platform: Linux-5.4.0-162-generic-x86_64-with-glibc2.31
• Python version: 3.12.9
• PyTorch version: 2.6.0+cu118 (GPU)
• Transformers version: 4.50.0
• Datasets version: 3.4.1
• Accelerate version: 1.5.2
• PEFT version: 0.15.0
• TRL version: 0.9.6
• GPU type: NVIDIA GeForce RTX 4090
• GPU number: 8
• GPU memory: 23.65GB
• DeepSpeed version: 0.16.5
• vLLM version: 0.8.1

Reproduction

在使用deepspeed=examples/deepspeed/ds_z3_config.json用4张4090对qwen2.5vl7b进行lora微调时出现如下问题

Traceback (most recent call last):
  File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/bin/torchrun", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl                                                            
[rank1]:     return self._call_impl(*args, **kwargs)                                                                                                                                                                                        
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                        
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl                                                                                               
[rank1]:     return inner()                                                                                                                                                                                                                 
[rank1]:            ^^^^^^^                                                                                                                                                                                                                 
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1793, in inner                                                                                                    
[rank1]:     result = forward_call(*args, **kwargs)                                                                                                                                                                                         
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                         
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/peft/tuners/tuners_utils.py", line 193, in forward                        
[rank1]:     return self.model.forward(*args, **kwargs)
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1875, in forward
[rank1]:     logits = self.lm_head(hidden_states)
[rank1]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                           
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank1]:     return self._call_impl(*args, **kwargs)                                                                                                                                                                                        
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                  
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1845, in _call_impl
[rank1]:     return inner()                                                                                                                                                                                                                 
[rank1]:            ^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1782, in inner
[rank1]:     args_result = hook(self, args)                                                                           
[rank1]:                   ^^^^^^^^^^^^^^^^                                                                           
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 372, in _post_backward_module_hook
[rank1]:     return apply_to_tensors_only(module.post_bwd_fn.apply,
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/utils.py", line 133, in apply_to_tensors_only
[rank1]:     touched_output = apply_to_tensors_only(function, elem)
[rank1]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/utils.py", line 149, in apply_to_tensors_only
[rank1]:     touched_output = function(value)
[rank1]:                      ^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/autograd/function.py", line 575, in apply
[rank1]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                    
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 440, in forward
[rank1]:     module.ds_grads_remaining += 1                                                                           
[rank1]:     ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1928, in __getattr__
[rank1]:     raise AttributeError(                 
[rank1]: AttributeError: 'Linear' object has no attribute 'ds_grads_remaining'

修改为ds_z2_config时出现oom

此外，在使用ds_z3_config时，添加如下配置：

enable_liger_kernel: True
use_unsloth_gc: True

出现了如下新的问题：

[rank1]: Traceback (most recent call last):                                                                                                                                                                                     
[rank1]:   File "/home/mas-wang.zhenyu/download/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>                                                                                                                           
[rank1]:     launch()                                                                                                                                                                                                                       
[rank1]:   File "/home/mas-wang.zhenyu/download/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch                                                                                                                             
[rank1]:     run_exp()                                                                                                                                                                                                                      
[rank1]:   File "/home/mas-wang.zhenyu/download/LLaMA-Factory/src/llamafactory/train/tuner.py", line 107, in run_exp                                                                                                                        
[rank1]:     _training_function(config={"args": args, "callbacks": callbacks})                                                                                                                                                              
[rank1]:   File "/home/mas-wang.zhenyu/download/LLaMA-Factory/src/llamafactory/train/tuner.py", line 69, in _training_function                                                                                                              
[rank1]:     run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)                                                                                                                                     
[rank1]:   File "/home/mas-wang.zhenyu/download/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 102, in run_sft                                                                                                                 
[rank1]:     train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/trainer.py", line 2245, in train
[rank1]:     return inner_training_loop(
[rank1]:            ^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/trainer.py", line 2556, in _inner_training_loop
[rank1]:     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/transformers/trainer.py", line 3764, in training_step
[rank1]:     self.accelerator.backward(loss, **kwargs)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/accelerate/accelerator.py", line 2351, in backward
[rank1]:     self.deepspeed_engine_wrapped.backward(loss, **kwargs)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/accelerate/utils/deepspeed.py", line 266, in backward
[rank1]:     self.engine.backward(loss, **kwargs)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank1]:     ret_val = func(*args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 2187, in backward
[rank1]:     self._do_optimizer_backward(loss, retain_graph)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 2133, in _do_optimizer_backward
[rank1]:     self.optimizer.backward(loss, retain_graph=retain_graph)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank1]:     ret_val = func(*args, **kwargs)
[rank1]:               ^^^^^^^^^^^^^^^^^^^^^
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/zero/stage3.py", line 2284, in backward
[rank1]:     self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
[rank1]:     scaled_loss.backward(retain_graph=retain_graph)
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/_tensor.py", line 626, in backward
[rank1]:     torch.autograd.backward(
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/autograd/__init__.py", line 347, in backward
[rank1]:     _engine_run_backward(
[rank1]:   File "/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
[rank1]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank1]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: RuntimeError: The size of tensor a (0) must match the size of tensor b (3584) at non-singleton dimension 1

请问在不修改cutoff_len和max_pixels的前提下能否通过修改其他配置解决问题？求助！！

Others

No response

[Luffy966]

yaml配置文件如下：

### model
model_name_or_path: "/148Dataset/data-wang.zhenyu/models/{{BASE_MODEL_SUBDIR}}/{{BASE_MODEL_NAME}}"
trust_remote_code: true
new_special_tokens: "{{NEW_SPECIAL_TOKENS}}"
skip_special_tokens: false
# flash_attn: False

### 降低分辨率
image_max_pixels: 23520 # 原262144，200704=448*448, 31360=40*28*28, 23520=30*28*28
video_max_pixels: 47040 # 原16384(128*128)，65536=256*256
video_fps: 2.0 # 默认2.0
video_maxlen: 48 # 视频输入最大总帧数 # 原版是针对单视频的，后修改源码改成了多视频时按视频时长分配maxlen

### method
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 8 # 数据多就多给点
lora_alpha: 8 # 数据多就多给点
lora_target: all
deepspeed: "{{BASE_PATH}}/LLaMA-Factory/examples/deepspeed/ds_z3_config.json"  # 降低显存的办法 # ds_z3_config # ds_z3_offload_config
# enable_liger_kernel: True
# use_unsloth_gc: True

### dataset
dataset_dir: "{{BASE_PATH}}/LLaMA-Factory/data/" # dataset_info.json所在目录
dataset: "CPED_RE_{{TASK}}"  # video_annotation, audio_chat, audio_annotation
template: qwen2_vl
cutoff_len: 2048 # 2048
max_samples: 320 # for DEBUG
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: "/148Dataset/data-wang.zhenyu/models/checkpoints/{{BASE_MODEL_NAME}}_train_lora_{{TASK}}_{{CURRENT_DATE}}"
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false

### train
per_device_train_batch_size: 1 # 每个设备（如 GPU）上的训练批次大小，适合显存有限的设备。
gradient_accumulation_steps: 4 # 梯度累积步数为n，这意味着每n个小批次的梯度将累积后再进行一次更新，等效于增加了有效批次大小。4卡时bs变成16
learning_rate: 1.0e-5
num_train_epochs: 2.0 # 原3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000 # DDP（分布式数据并行）超时设置，确保在分布式环境下不会因为超时导致训练中断。
# resume_from_checkpoint: null # 指定checkpoint路径即可断点续训

### eval
val_size: 0.1 # 验证集占数据集的比例为 0.1，表示使用 10% 的数据集进行验证。
per_device_eval_batch_size: 1 # 每个设备上评估的批次大小为 1，与训练的批次大小一致。
eval_strategy: steps # 评估策略为按步评估，意味着每隔一定步数进行一次评估。
eval_steps: 500 # 每 500 步进行一次评估，确保训练期间可以监控模型性能。

# swanlab
use_swanlab: true
swanlab_project: llamafactory
swanlab_run_name: "{{BASE_MODEL_NAME}}_train_lora_{{TASK}}_{{CURRENT_DATE}}"

[Luffy966]

我刚把cutoff_len和两个max_pixels降下来，然后尝试用ds_z2_config，结果也遇到了AttributeError: 'Linear' object has no attribute 'ds_grads_remaining'
@hiyouga 大佬有什么头绪吗，在线等急

[Luffy966]

用fsdp可以顺利运行，但是模型并行方面显然仍有较大提升空间（这样我就不需要降低cutoff_len和两个max_pixels了），强烈希望大佬能看看如何让deepspeed也正常工作！！
@hiyouga

[Luffy966]

现在fsdp训完之后不知道为什么又卡住了

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 36/36 [26:57<00:00, 44.62s/it]
/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set
_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .  
  warnings.warn(                                           
/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs
, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial:
 https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .                                           
  warnings.warn(                                           
/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs
, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial:
 https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .                                           
  warnings.warn(                                           
/home/mas-wang.zhenyu/anaconda3/envs/vllm/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs
, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial:
 https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .                                           
  warnings.warn(                                           
[INFO|trainer.py:3966] 2025-04-03 08:13:10,827 >> Saving model checkpoint to /148Dataset/data-wang.zhenyu/models/checkpoints/Qwen2.5-VL-7B-Instruct_train_lora_video_chat_2025-04-03-07-44-59/checkpoint-36                                 
[INFO|configuration_utils.py:697] 2025-04-03 08:13:11,331 >> loading configuration file /148Dataset/data-wang.zhenyu/models/ModelScope_download/Qwen2.5-VL-7B-Instruct/config.json
[INFO|configuration_utils.py:771] 2025-04-03 08:13:11,333 >> Model config Qwen2_5_VLConfig {
  "architectures": [                                       
    "Qwen2_5_VLForConditionalGeneration"                   
  ],                                                       
  "attention_dropout": 0.0,                                
  "bos_token_id": 151643,                                  
  "eos_token_id": 151645,                                  
  "hidden_act": "silu",                                    
  "hidden_size": 3584,                                     
  "image_token_id": 151655,                                
  "initializer_range": 0.02,                               
  "intermediate_size": 18944,                              
  "max_position_embeddings": 128000,                       
  "max_window_layers": 28,                                 
  "model_type": "qwen2_5_vl",                              
  "num_attention_heads": 28,                               
  "num_hidden_layers": 28,                                 
  "num_key_value_heads": 4,                                
  "rms_norm_eps": 1e-06,                                   
  "rope_scaling": {                                        
    "mrope_section": [                                     
      16,                                                  
      24,
      24                                                   
    ],
    "rope_type": "default", 
    "type": "default"
  },
  "rope_theta": 1000000.0,
  "sliding_window": 32768,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.50.0",
  "use_cache": true,
  "use_sliding_window": false,
  "video_token_id": 151656,
  "vision_config": {
    "depth": 32,
    "fullatt_block_indexes": [
      7,
      15,
      23,
      31    ],    "hidden_act": "silu",
    "hidden_size": 1280,
    "in_channels": 3,
    "in_chans": 3,
    "intermediate_size": 3420,
    "model_type": "qwen2_5_vl",
    "num_heads": 16,
    "out_hidden_size": 3584,
    "patch_size": 14,
    "spatial_merge_size": 2,
    "spatial_patch_size": 14,
    "temporal_patch_size": 2,
    "tokens_per_second": 2,
    "window_size": 112
  },
  "vision_end_token_id": 151653,
  "vision_start_token_id": 151652,
  "vision_token_id": 151654,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2510] 2025-04-03 08:13:59,174 >> tokenizer config file saved in /148Dataset/data-wang.zhenyu/models/checkpoints/Qwen2.5-VL-7B-Instruct_train_lora_video_chat_2025-04-03-07-44-59/checkpoint-36/tokenizer_config.json
[INFO|tokenization_utils_base.py:2519] 2025-04-03 08:13:59,207 >> Special tokens file saved in /148Dataset/data-wang.zhenyu/models/checkpoints/Qwen2.5-VL-7B-Instruct_train_lora_video_chat_2025-04-03-07-44-59/checkpoint-36/special_tokens_map.json
WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'RemoteDisconnected('Remote end closed connection without response')': /api/house/metrics

以上是最新日志

[Luffy966]

刚修改了ds版本，降到0.16.4和0.15.0都不会出现AttributeError: 'Linear' object has no attribute 'ds_grads_remaining'的问题了，而且也能成功训练，我想这是ds的问题，因此在ds那边也发了issue，deepspeedai/DeepSpeed#7203 (comment)
正在测试liger_kernel的问题RuntimeError: The size of tensor a (0) must match the size of tensor b (3584) at non-singleton dimension 1是否仍然存在，同样我也在liger-kernel处找到了类似的issue：linkedin/Liger-Kernel#480 (comment)

[Luffy966]

问题已解决：
前面使用liger_kernel的报错RuntimeError: The size of tensor a (0) must match the size of tensor b (3584) at non-singleton dimension 1同样在降低版本deepspeed==0.16.4后不再出现，0.15.0的ds没有尝试

最后的问题就是fsdp训练完后save或eval的过程中卡住的问题，与本issue无关，也不再使用fsdp，搁置

[Dengsi]

你好，你用4张4090就能微调qwen2.5-vl-7b吗？微调出来效果如何？没有搞过多模态大模型的微调，不太清楚需要用多少GPU资源。

[Luffy966]

你好，你用4张4090就能微调qwen2.5-vl-7b吗？微调出来效果如何？没有搞过多模态大模型的微调，不太清楚需要用多少GPU资源。

应该可以，没记错的话qwen2.5-vl-7b官方repo上有显存占用表，部署float16精度的qwen2.5-vl-7b理论最小显存占用是13.17GB，实际使用的话一般是1.2倍，训练的话还要考虑上数据集最大样本大小，一般两卡够用，怕oom也可以四卡，btw用llamafactory微调时出现oom可以参考#4614