训练过程中时常报错RuntimeError: MUSA error: unknown error

[zhutao1221]

训练偶尔可以跑完所有轮次，但大部分时候，都会在训练过程中报错。报错信息如下：
Traceback (most recent call last):
File "/home/zhutao/workspace/StockFormer-main/code/Transformer/run_short.py", line 109, in
exp.train(setting)
File "/home/zhutao/workspace/StockFormer-main/code/Transformer/exp/exp_pred.py", line 160, in train
loss.backward()
File "/home/zhutao/programs/conda3/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward
torch.autograd.backward(
File "/home/zhutao/programs/conda3/lib/python3.10/site-packages/torch/autograd/init.py", line 266, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: MUSA error: unknown error