单机多卡报错求助

[2792715512]

这是我的命令

CUDA_VISIBLE_DEVICES=0,1 NCCL_P2P_LEVEL=NVL llamafactory-cli train \
    --stage sft \
    --do_train True \
    --model_name_or_path /data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct \
    --preprocessing_num_workers 16 \
    --finetuning_type lora \
    --template default \
    --flash_attn auto \
    --dataset_dir /data/aimanager/upFiles/tmp_1745395139326 \
    --dataset self_cognition_1914880990308855810 \
    --cutoff_len 2048 \
    --learning_rate 5e-05 \
    --num_train_epochs 3.0 \
    --max_samples 100000 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --lr_scheduler_type cosine \
    --max_grad_norm 1.0 \
    --logging_steps 5 \
    --save_steps 100 \
    --warmup_steps 0 \
    --packing False \
    --report_to none \
    --output_dir /data/aimanager/upFiles/tmp_1745395139326/train \
    --bf16 True \
    --plot_loss True \
    --trust_remote_code True \
    --ddp_timeout 180000000 \
    --include_num_input_tokens_seen True \
    --optim adamw_torch \
    --lora_rank 8 \
    --lora_alpha 16 \
    --lora_dropout 0 \
    --lora_target all

这是我的所有日志信息

[2025-04-24 18:34:00,559] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|2025-04-24 18:34:04] llamafactory.cli:143 >> Initializing 2 distributed tasks at: 127.0.0.1:53445
W0424 18:34:06.047000 3773 site-packages/torch/distributed/run.py:792] 
W0424 18:34:06.047000 3773 site-packages/torch/distributed/run.py:792] *****************************************
W0424 18:34:06.047000 3773 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0424 18:34:06.047000 3773 site-packages/torch/distributed/run.py:792] *****************************************
[2025-04-24 18:34:10,503] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-24 18:34:10,693] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|2025-04-24 18:34:12] llamafactory.hparams.parser:401 >> Process rank: 1, world size: 2, device: cuda:1, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-24 18:34:13] llamafactory.hparams.parser:143 >> Set `ddp_find_unused_parameters` to False in DDP training since LoRA is enabled.
[INFO|2025-04-24 18:34:13] llamafactory.hparams.parser:401 >> Process rank: 0, world size: 2, device: cuda:0, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,097 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2323] 2025-04-24 18:34:13,432 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:691] 2025-04-24 18:34:13,432 >> loading configuration file /data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct/config.json
[INFO|configuration_utils.py:765] 2025-04-24 18:34:13,434 >> Model config Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": 131072,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-24 18:34:13,435 >> loading file chat_template.jinja
[rank1]:[W424 18:34:13.085814696 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[INFO|tokenization_utils_base.py:2323] 2025-04-24 18:34:13,784 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|2025-04-24 18:34:13] llamafactory.data.loader:143 >> Loading dataset self_cognition_1914880990308855810.json...
[rank0]:[W424 18:34:14.322123916 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
E0424 18:34:15.475000 3773 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -11) local_rank: 0 (pid: 3850) of binary: /home/gpu/anaconda3/envs/llama_factory_env_new/bin/python
Traceback (most recent call last):
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/data/aimanager/backup/LLaMA-Factory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2025-04-24_18:34:15
  host      : gpu-vm
  rank      : 1 (local_rank: 1)
  exitcode  : -11 (pid: 3851)
  error_file: <N/A>
  traceback : Signal 11 (SIGSEGV) received by PID 3851
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-24_18:34:15
  host      : gpu-vm
  rank      : 0 (local_rank: 0)
  exitcode  : -11 (pid: 3850)
  error_file: <N/A>
  traceback : Signal 11 (SIGSEGV) received by PID 3850
============================================================
Traceback (most recent call last):
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
  File "/data/aimanager/backup/LLaMA-Factory/src/llamafactory/cli.py", line 95, in main
    process = subprocess.run(
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['torchrun', '--nnodes', '1', '--node_rank', '0', '--nproc_per_node', '2', '--master_addr', '127.0.0.1', '--master_port', '53445', '/data/aimanager/backup/LLaMA-Factory/src/llamafactory/launcher.py', '--stage', 'sft', '--do_train', 'True', '--model_name_or_path', '/data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct', '--preprocessing_num_workers', '16', '--finetuning_type', 'lora', '--template', 'default', '--flash_attn', 'auto', '--dataset_dir', '/data/aimanager/upFiles/tmp_1745395139326', '--dataset', 'self_cognition_1914880990308855810', '--cutoff_len', '2048', '--learning_rate', '5e-05', '--num_train_epochs', '3.0', '--max_samples', '100000', '--per_device_train_batch_size', '2', '--gradient_accumulation_steps', '8', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--logging_steps', '5', '--save_steps', '100', '--warmup_steps', '0', '--packing', 'False', '--report_to', 'none', '--output_dir', '/data/aimanager/upFiles/tmp_1745395139326/train', '--bf16', 'True', '--plot_loss', 'True', '--trust_remote_code', 'True', '--ddp_timeout', '180000000', '--include_num_input_tokens_seen', 'True', '--optim', 'adamw_torch', '--lora_rank', '8', '--lora_alpha', '16', '--lora_dropout', '0', '--lora_target', 'all']' returned non-zero exit status 1.

nvcc版本

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Feb_27_16:19:38_PST_2024
Cuda compilation tools, release 12.4, V12.4.99
Build cuda_12.4.r12.4/compiler.33961263_0

显卡

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.107.02             Driver Version: 550.107.02     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090        Off |   00000000:03:00.0 Off |                  Off |
| 44%   33C    P8              5W /  450W |     517MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 4090        Off |   00000000:04:00.0 Off |                  Off |
| 45%   34C    P8              7W /  450W |      18MiB /  24564MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A      2492      G   /usr/lib/xorg/Xorg                              4MiB |
|    0   N/A  N/A      2902      C   python3.10                                    494MiB |
|    1   N/A  N/A      2492      G   /usr/lib/xorg/Xorg                              4MiB |
+-----------------------------------------------------------------------------------------+

torch版本

python -c "import torch; print(torch.__version__)"
2.6.0+cu124

单卡可以完美运行，多卡就是一直报错找了很多文档都没有用，求助！！！

[hiyouga]

try removing preprocessing_num_workers

[2792715512]

updated Command

CUDA_VISIBLE_DEVICES=0,1 NCCL_P2P_LEVEL=NVL llamafactory-cli train \
    --model_name_or_path /data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct \
    --stage sft \
    --do_train True \
    --finetuning_type lora \
    --lora_target all \
    --dataset_dir /data/aimanager/upFiles/tmp_1745395139326 \
    --dataset self_cognition_1914880990308855810 \
    --template default \
    --cutoff_len 2048 \
    --max_samples 100000 \
    --overwrite_cache True \
    --output_dir /data/aimanager/upFiles/tmp_1745395139326/train \
    --logging_steps 5 \
    --save_steps 100 \
    --plot_loss True \
    --overwrite_output_dir True \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 8 \
    --learning_rate 5e-05 \
    --num_train_epochs 3.0 \
    --lr_scheduler_type cosine \
    --warmup_ratio 0.1 \
    --bf16 True \
    --ddp_timeout 180000000 \
    --val_size 0.1 \
    --per_device_eval_batch_size 1 \
    --eval_strategy steps \
    --eval_steps 500

logs

[2025-04-25 09:47:25,731] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|2025-04-25 09:47:29] llamafactory.cli:143 >> Initializing 2 distributed tasks at: 127.0.0.1:33517
W0425 09:47:30.931000 39281 site-packages/torch/distributed/run.py:792] 
W0425 09:47:30.931000 39281 site-packages/torch/distributed/run.py:792] *****************************************
W0425 09:47:30.931000 39281 site-packages/torch/distributed/run.py:792] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0425 09:47:30.931000 39281 site-packages/torch/distributed/run.py:792] *****************************************
[2025-04-25 09:47:35,405] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2025-04-25 09:47:35,645] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[INFO|2025-04-25 09:47:37] llamafactory.hparams.parser:401 >> Process rank: 1, world size: 2, device: cuda:1, distributed training: True, compute dtype: torch.bfloat16
[INFO|2025-04-25 09:47:38] llamafactory.hparams.parser:143 >> Set `ddp_find_unused_parameters` to False in DDP training since LoRA is enabled.
[INFO|2025-04-25 09:47:38] llamafactory.hparams.parser:401 >> Process rank: 0, world size: 2, device: cuda:0, distributed training: True, compute dtype: torch.bfloat16
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,012 >> loading file chat_template.jinja
[INFO|tokenization_utils_base.py:2323] 2025-04-25 09:47:38,352 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:691] 2025-04-25 09:47:38,353 >> loading configuration file /data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct/config.json
[INFO|configuration_utils.py:765] 2025-04-25 09:47:38,354 >> Model config Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": 131072,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.51.3",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 152064
}

[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2058] 2025-04-25 09:47:38,355 >> loading file chat_template.jinja
[rank1]:[W425 09:47:38.998116076 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 1]  using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
[INFO|tokenization_utils_base.py:2323] 2025-04-25 09:47:38,708 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|2025-04-25 09:47:38] llamafactory.data.loader:143 >> Loading dataset self_cognition_1914880990308855810.json...
Converting format of dataset: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 189/189 [00:00<00:00, 6381.15 examples/s]
[rank0]:[W425 09:47:39.180107967 ProcessGroupNCCL.cpp:4561] [PG ID 0 PG GUID 0 Rank 0]  using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id.
E0425 09:47:40.259000 39281 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -11) local_rank: 0 (pid: 39349) of binary: /home/gpu/anaconda3/envs/llama_factory_env_new/bin/python
Traceback (most recent call last):
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/run.py", line 918, in main
    run(args)
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/data/aimanager/backup/LLaMA-Factory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2025-04-25_09:47:40
  host      : gpu-vm
  rank      : 1 (local_rank: 1)
  exitcode  : -11 (pid: 39350)
  error_file: <N/A>
  traceback : Signal 11 (SIGSEGV) received by PID 39350
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-04-25_09:47:40
  host      : gpu-vm
  rank      : 0 (local_rank: 0)
  exitcode  : -11 (pid: 39349)
  error_file: <N/A>
  traceback : Signal 11 (SIGSEGV) received by PID 39349
============================================================
Traceback (most recent call last):
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
  File "/data/aimanager/backup/LLaMA-Factory/src/llamafactory/cli.py", line 95, in main
    process = subprocess.run(
  File "/home/gpu/anaconda3/envs/llama_factory_env_new/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['torchrun', '--nnodes', '1', '--node_rank', '0', '--nproc_per_node', '2', '--master_addr', '127.0.0.1', '--master_port', '33517', '/data/aimanager/backup/LLaMA-Factory/src/llamafactory/launcher.py', '--model_name_or_path', '/data/aimanager/upFiles/local_models/Qwen2.5-7B-Instruct', '--stage', 'sft', '--do_train', 'True', '--finetuning_type', 'lora', '--lora_target', 'all', '--dataset_dir', '/data/aimanager/upFiles/tmp_1745395139326', '--dataset', 'self_cognition_1914880990308855810', '--template', 'default', '--cutoff_len', '2048', '--max_samples', '100000', '--overwrite_cache', 'True', '--output_dir', '/data/aimanager/upFiles/tmp_1745395139326/train', '--logging_steps', '5', '--save_steps', '100', '--plot_loss', 'True', '--overwrite_output_dir', 'True', '--per_device_train_batch_size', '2', '--gradient_accumulation_steps', '8', '--learning_rate', '5e-05', '--num_train_epochs', '3.0', '--lr_scheduler_type', 'cosine', '--warmup_ratio', '0.1', '--bf16', 'True', '--ddp_timeout', '180000000', '--val_size', '0.1', '--per_device_eval_batch_size', '1', '--eval_strategy', 'steps', '--eval_steps', '500']' returned non-zero exit status 1.

不知道是不是我这台机器的环境问题，我已经对照过很多次了，仍然无法进行单机多卡微调