error in sequence smoothing when using modified metro code

[boukhayma]

hello, thanks for the work.
I tried running metro with the modifications from this repo. it worked with the coarse and fine alignments but at the stage of smoothing. I get the following error:

 --- Sequence Smoothing --- 
 --- Sequence Smoothing --- 
samples/hand/sample/metro_mano
dict_keys(['joints', 'verts', 'rot', 'pose', 'shape', 'trans', 'cam', 'joints_metro', 'verts_metro'])
Traceback (most recent call last):
  File "./metro/tools/end2end_inference_handmesh.py", line 544, in <module>
    main(args)
  File "./metro/tools/end2end_inference_handmesh.py", line 540, in main
    run_inference(args, image_list, _metro_network, mano_model, renderer, mesh_sampler)    
  File "./metro/tools/end2end_inference_handmesh.py", line 313, in run_inference
    smooth_sequence(renderer, mano_arm_layer, mano_fit_output_dir, mano_fit_smooth_dir, use_smplx_arm=True, img_res=RESOLUTION)
  File "/test_container/metro/hand_utils/hand_utils.py", line 706, in smooth_sequence
    params_out = optimize_smooth_seq(params, mano_layer, use_smplx_arm=use_smplx_arm, img_res=img_res)
  File "/test_container/metro/hand_utils/hand_utils.py", line 588, in optimize_smooth_seq
    betas=params['shape'], global_orient=params['rot'], transl=params['trans'], right_hand_pose=params['pose'], return_type='mano')
  File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/test_container/hand_models/smplx/smplx/body_models.py", line 2367, in forward
    lmk_bary_coords)
  File "/test_container/hand_models/smplx/smplx/lbs.py", line 152, in vertices2landmarks
    landmarks = torch.einsum('blfi,blf->bli', [lmk_vertices, lmk_bary_coords])
  File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/functional.py", line 241, in einsum
    return torch._C._VariableFunctions.einsum(equation, operands)

when running without fit_arm option I get the following error instead:

2023-07-20 21:24:46,013 METRO Inference INFO: Run inference
After coarse alignment: 218.062958
After fine alignment: 2.566075
save to  samples/hand/sample/metro_image/0001_metro_pred.jpg
After coarse alignment: 209.102310
After fine alignment: 2.593198
save to  samples/hand/sample/metro_image/0002_metro_pred.jpg
 --- Sequence Smoothing --- 
 --- Sequence Smoothing --- 
samples/hand/sample/metro_mano
dict_keys(['joints', 'verts', 'rot', 'pose', 'shape', 'trans', 'cam', 'joints_metro', 'verts_metro'])
Traceback (most recent call last):
  File "./metro/tools/end2end_inference_handmesh.py", line 544, in <module>
    main(args)
  File "./metro/tools/end2end_inference_handmesh.py", line 540, in main
    run_inference(args, image_list, _metro_network, mano_model, renderer, mesh_sampler)    
  File "./metro/tools/end2end_inference_handmesh.py", line 313, in run_inference
    smooth_sequence(renderer, mano_arm_layer, mano_fit_output_dir, mano_fit_smooth_dir, use_smplx_arm=False, img_res=RESOLUTION)
  File "/test_container/metro/hand_utils/hand_utils.py", line 706, in smooth_sequence
    params_out = optimize_smooth_seq(params, mano_layer, use_smplx_arm=use_smplx_arm, img_res=img_res)
  File "/test_container/metro/hand_utils/hand_utils.py", line 590, in optimize_smooth_seq
    hand_verts, hand_joints = mano_layer(torch.cat((params['rot'], params['pose']), 1), params['shape'], params['trans'])
  File "/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/test_container/hand_models/smplx/smplx/body_models.py", line 2304, in forward
    full_pose = torch.cat([global_orient.reshape(-1, 1, 3),
RuntimeError: shape '[-1, 1, 3]' is invalid for input of size 50

do you have an idea of what could be the reason behind this.
Thanks

[korrawe]

Hi,
I cannot reproduce these errors. But as this happens in both cases it is probably related to the shape of params[...].
Could you check the shape of params['rot'], params['pose'], params['shape'], params['trans'] ? They should all be of size (N, feat_size).
Also maybe try with N != 3 because I remember in some pytorch version, there is a weird broadcasting behavior when selecting which dimension to broadcast.
Best,

[anonymous-pusher]

I also had this error and it's probably because einsum has a different behavior for broadcasting in newer versions of pytorch. the function vertices2landmarks in lbs.py works fine in the first stage of per-image metro fitting but in the smoothing stage, the batch size is not 1 but equal to the number of used images.
Expanding the dimension of lmk_bary_coords worked for me.
I added these line to the vertices2landmarks function:

try:
    landmarks = torch.einsum('blfi,blf->bli', [lmk_vertices, lmk_bary_coords])
except:
    landmarks = torch.einsum('blfi,blf->bli', [lmk_vertices, lmk_bary_coords.expand(batch_size, -1, -1)])