Very large loss of "FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_conf_loss" when fine tuning

[bowieshi]

Hi. Thanks for your great work! I am trying to fine tune Map-anything on my customized dataset. After several epoch, the training failed with the following info. I found the loss of "FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_conf_loss" becomes very large and the training failed the assertion for loss < 1000. May I ask what is the potential reason for the very large loss and how can I fix it? Thank you!

[2026-01-09 01:19:05,258][__main__][INFO] - [01:19:05.257972]                                                                                                                             
[2026-01-09 01:19:05,361][__main__][INFO] - Epoch: [3]  [540/745]  eta: 0:21:53  lr: 0.000074  lr_encoder: 0.000004  epoch: 3.7114 (3.3624)  loss: -0.2796 (-0.0543)  FactoredGeometryScal
eRegr3DPlusNormalGMLoss_pts3d_conf_loss_avg: -0.2961 (-0.2075)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss_avg: 0.0023 (0.0052)  FactoredGeometryScaleRegr3DPlusNor
malGMLoss_depth_along_ray_bot95%_loss_avg: 0.0018 (0.0047)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_avg: 0.0804 (0.1372)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_
avg: 0.0080 (0.0137)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_avg: 0.0074 (0.0131)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_avg: 0.0007 (0.0018)  F
actoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_avg: 0.0000 (0.0000)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_avg: 0.0000 (0.0000)  NonAmbiguousMaskLoss_mask_avg: 0.
0475 (0.0494)
 time: 5.9609  data: 0.0032  max mem: 39620      
[2026-01-09 01:19:36,745][__main__][INFO] - [01:19:36.744978]                                                                                                                             
[2026-01-09 01:19:36,745][__main__][INFO] - Error in datasetWAI.__getitem__ for scene_idx=30528:                                                                                          
[2026-01-09 01:19:36,745][__main__][INFO] - [01:19:36.745607]                                                                                                                             
[2026-01-09 01:19:36,745][__main__][INFO] - Retrying with scene_idx=14533 (1 of 5)                                                       
[2026-01-09 01:20:47,119][__main__][INFO] - [01:20:47.119676]                                                                            
[2026-01-09 01:20:47,176][__main__][INFO] - Loss is 5927.96533203125, stopping training                                                  
[2026-01-09 01:20:47,176][__main__][INFO] - [01:20:47.176251]                                                                            
[2026-01-09 01:20:47,176][__main__][INFO] - Loss Details: {'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_conf_loss_view1': 2665.58984375, 'FactoredGeometryScaleRegr3DPlusNormalGMLos
s_pts3d_conf_loss_avg': 2963.632568359375, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_conf_loss_view2': 3261.67529296875, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_b
ot95%_loss_view1': 0.09395701438188553, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss_avg': 0.09373242780566216, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts
3d_bot95%_loss_view2': 0.09350784122943878, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_view1': 0.09339836239814758, 'FactoredGeometryScaleRegr3DPlusNormalGM
Loss_depth_along_ray_bot95%_loss_avg': 0.09323638305068016, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_view2': 0.09307440370321274, 'FactoredGeometryScaleRe
gr3DPlusNormalGMLoss_pts3d_view1': 0.988899827003479, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_view2': 0.9848102927207947, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_av
g': 0.9868550598621368, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_view1': 0.09890376776456833, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_view2': 0.0984916910529
1367, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_avg': 0.098697729408741, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_view1': 0.09831757843494415, 'FactoredG
eometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_view2': 0.09802951663732529, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_avg': 0.09817354753613472, 'FactoredGeometryS
caleRegr3DPlusNormalGMLoss_ray_directions_view1': 0.001488085137680173, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_view2': 0.0012360082473605871, 'FactoredGeometryScaleR
egr3DPlusNormalGMLoss_ray_directions_avg': 0.00136204669252038, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_view1': 1.5497208494252845e-07, 'FactoredGeometryScaleRegr3DPlusNo
rmalGMLoss_pose_quats_view2': 1.6093255794658035e-07, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_avg': 1.579523214445544e-07, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_po
se_trans_view1': 2.920627935054654e-07, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_view2': 3.0398371109185973e-07, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_av
g': 2.9802325229866256e-07, 'NonAmbiguousMaskLoss_mask_view1': 0.07778102159500122, 'NonAmbiguousMaskLoss_mask_view2': 0.08098114281892776, 'NonAmbiguousMaskLoss_mask_avg': 0.07938108220
696449, 
ll-l40-0:980357:980980 [3] NCCL INFO [Service thread] Connection closed by localRank 3                                                                                                    
ll-l40-0:980357:988679 [3] NCCL INFO comm 0x3a749720 rank 3 nranks 8 cudaDev 3 busId 61000 - Abort COMPLETE                                                                               
W0109 01:21:52.090000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980354 closing signal SIGTERM                                            
W0109 01:21:52.094000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980355 closing signal SIGTERM                                            
W0109 01:21:52.097000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980356 closing signal SIGTERM                                            
W0109 01:21:52.099000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980358 closing signal SIGTERM                                            
W0109 01:21:52.101000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980359 closing signal SIGTERM                                            
W0109 01:21:52.104000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980360 closing signal SIGTERM                                            
W0109 01:21:52.106000 950999 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 980361 closing signal SIGTERM

[Nik-V9]

Hi, Thanks for flagging this!

This is an issue with the pre-trained MapAnything V1.1 checkpoint where the confidence channel is not trained in the second stage finetuning. I will be update the finetuning bash scripts to reflect the latest config.

Please use this loss function config:

map-anything/bash_scripts/train/main/mapa_curri_24v_13d_36ipg_64g.sh

Line 44 in b84b28f

loss=overall_loss_highpm_plus_rel_pose_no_conf \

[bowieshi]

Hi, Thanks for flagging this!

This is an issue with the pre-trained MapAnything V1.1 checkpoint where the confidence channel is not trained in the second stage finetuning. I will be update the finetuning bash scripts to reflect the latest config.

Please use this loss function config:

map-anything/bash_scripts/train/main/mapa_curri_24v_13d_36ipg_64g.sh

Line 44 in b84b28f

loss=overall_loss_highpm_plus_rel_pose_no_conf \

Thank you for your quick reply! I used the new loss function to fine tune the model. However, the training break down again after 7 epoch with Loss is nan, stopping training. The following is the log for the break down. Do you have any suggestion for this issue? Thank you for your help!

[2026-01-12 00:59:21,358][__main__][INFO] - Epoch: [7]  [320/745]  eta: 0:40:35  lr: 0.000149  lr_encoder: 0.000007  epoch: 7.4161 (7.2[1626/1987]
 0.0651 (0.0851)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_bot95%_loss_avg: 0.0258 (0.0343)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_
cam_pts3d_bot95%_loss_avg: 0.0025 (0.0034)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_avg: 0.0017 (0.0031)  Factored
GeometryScaleRegr3DPlusNormalGMLoss_pts3d_avg: 0.0884 (0.1341)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_avg: 0.0089 (0.0134)  Factor
edGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_avg: 0.0087 (0.0130)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_avg: 0.0008
 (0.0010)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_avg: 0.0000 (0.0000)  FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_avg
: 0.0000 (0.0001)  NonAmbiguousMaskLoss_mask_avg: 0.0254 (0.0242)  time: 5.1015  data: 0.0007  max mem: 36703                                     
[2026-01-12 00:59:31,196][__main__][INFO] - [00:59:31.196889]                                                                                     
[2026-01-12 00:59:31,197][__main__][INFO] - Loss is nan, stopping training                                                                        
[2026-01-12 00:59:31,197][__main__][INFO] - [00:59:31.197463]                                                                                     
[2026-01-12 00:59:31,197][__main__][INFO] - Loss Details: {'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_bot95%_loss_view1': 0.02155563421547
413, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_bot95%_loss_avg': 0.02266706945374608, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_
bot95%_loss_view2': 0.025079762563109398, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_bot95%_loss_view3': 0.024039963260293007, 'FactoredGe
ometryScaleRegr3DPlusNormalGMLoss_pts3d_bot95%_loss_view4': 0.019992917776107788, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_lo
ss_view1': 0.0021483523305505514, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss_avg': 0.002192512562032789, 'FactoredGeometry
ScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss_view2': 0.002457397524267435, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss
_view3': 0.002303954679518938, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_bot95%_loss_view4': 0.0018603457137942314, 'FactoredGeometry
ScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_view1': 0.0019646859727799892, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ra
y_bot95%_loss_avg': 0.002022430271608755, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_view2': 0.002284932415932417, '
FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_bot95%_loss_view3': 0.0021442309953272343, 'FactoredGeometryScaleRegr3DPlusNormalGMLos
s_depth_along_ray_bot95%_loss_view4': 0.0016958717023953795, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_view1': 0.05865059047937393, 'Fact
oredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_view2': 0.06658760458230972, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_view3': 0.0621923618
0186272, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_view4': 0.060079146176576614, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pts3d_avg':
 0.061877425760030746, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_view1': 0.0058586024679243565, 'FactoredGeometryScaleRegr3DPlusNorma
lGMLoss_cam_pts3d_view2': 0.00661842804402113, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_view3': 0.006139894016087055, 'FactoredGeome
tryScaleRegr3DPlusNormalGMLoss_cam_pts3d_view4': 0.005899397190660238, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_cam_pts3d_avg': 0.006129080429
673195, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_view1': 0.005666662938892841, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_de
pth_along_ray_view2': 0.006439320743083954, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_view3': 0.005972475279122591, 'FactoredGe
ometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_view4': 0.005727598909288645, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_depth_along_ray_avg':
 0.005951514467597008, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_view1': 0.0005937067326158285, 'FactoredGeometryScaleRegr3DPlus
NormalGMLoss_ray_directions_view2': 0.0005998766864649951, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_view3': 0.00055084604537114
5, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_directions_view4': 0.0005630161613225937, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_ray_dir
ections_avg': 0.0005768614064436406, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_view1': 2.3841860041784457e-08, 'FactoredGeometryScal
eRegr3DPlusNormalGMLoss_pose_quats_view2': 2.6822091214739885e-08, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_view3': 2.6822091214739
885e-08, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_quats_view4': 2.086162709247219e-08, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_
quats_avg': 2.4586917390934104e-08, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_view1': 7.27623701095581e-05, 'FactoredGeometryScaleRe
gr3DPlusNormalGMLoss_pose_trans_view2': 3.7053228879813105e-05, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_view3': 5.335510286386125e
-05, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_trans_view4': 0.00012083948240615427, 'FactoredGeometryScaleRegr3DPlusNormalGMLoss_pose_tra
ns_avg': 7.100254606484668e-05, 'NonAmbiguousMaskLoss_mask_view1': 0.028883634135127068, 'NonAmbiguousMaskLoss_mask_view2': 0.042588699609041214, 
'NonAmbiguousMaskLoss_mask_view3': 0.03011639602482319, 'NonAmbiguousMaskLoss_mask_view4': 0.03358123451471329, 'NonAmbiguousMaskLoss_mask_avg': 0
.03379249107092619}