Some NCCL operations have failed or timed out.

[qiangtang2017]

[rank2]:[E220 01:25:08.728702337 ProcessGroupNCCL.cpp:681] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.

[qiangtang2017]

when I run first step , I meet this error at 416/962 steps。The same error occure when I run this “torchrun --nproc_per_node=8 --nnodes=1 …… ”

[Shajiu]

当我运行第一步时，我在 416/962 步遇到此错误。当我运行这个“torchrun --nproc_per_node=8 --nnodes=1 …… ”时出现同样的错误

大佬解决了这个问题么？跪求交流？18810979033 微信