I met a problem when using coco2017 as dataset #5
Description
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:276: operator(): block: [1742,0,0], thread: [37,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:276: operator(): block: [1742,0,0], thread: [38,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
/opt/conda/conda-bld/pytorch_1634272168290/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:276: operator(): block: [1742,0,0], thread: [29,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds" failed.
File "/public/home/jw12138/anaconda3/envs/lstenv/lib/python3.7/site-packages/detectron2/modeling/meta_arch/dense_detector.py", line 101, in forward
return self.forward_training(images, features, predictions, gt_instances)
File "/public/home/jw12138/anaconda3/envs/lstenv/lib/python3.7/site-packages/detectron2/modeling/meta_arch/retinanet.py", line 158, in forward_training
return self.losses(anchors, pred_logits, gt_labels, pred_anchor_deltas, gt_boxes)
File "/public/home/jw12138/anaconda3/envs/lstenv/lib/python3.7/site-packages/detectron2/modeling/meta_arch/retinanet.py", line 191, in losses
cat(pred_logits, dim=1)[valid_mask],
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Please someone who konws help me! Thank u!