Empty images in "metro_image" and "mano_fit_image"

[pyxploiter]

Thank you for open-sourcing your wonderful work.

I configured the setup for my custom videos. But before processing my own videos, I tried to test it on the given preprocessed_data to confirm that everything is configured correctly and working fine.

Data: I choose "preprocessed_data/dim_light/3" sequence for initial testing as I have groundtruth available. I removed all other folders except "unscreen", "unscreen_cropped" and "mask" for inference

./data/dim_light/3
|-- unscreen
|-- unscreen_cropped
|-- mask

I ran the following command after following all the given instructions in the README:

python ./metro/tools/end2end_inference_handmesh.py --resume_checkpoint ./models/metro_release/metro_hand_state_dict.bin --image_file_or_path data --do_crop

Terminal log looks like:

2023-12-22 13:21:05,927 METRO Inference INFO: Inference: Loading from checkpoint ./models/metro_release/metro_hand_state_dict.bin
2023-12-22 13:21:05,927 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter hidden_size: 768 -> 1024
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 4096
2023-12-22 13:21:07,769 METRO Inference INFO: Init model from scratch.
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter hidden_size: 768 -> 256
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 1024
2023-12-22 13:21:08,027 METRO Inference INFO: Init model from scratch.
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter hidden_size: 768 -> 64
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 256
2023-12-22 13:21:08,086 METRO Inference INFO: Init model from scratch.
=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2023-12-22 13:21:10,246 METRO Inference INFO: => loading hrnet-v2-w64 model
2023-12-22 13:21:10,247 METRO Inference INFO: Transformers total parameters: 101182022
2023-12-22 13:21:10,252 METRO Inference INFO: Backbone total parameters: 128059944
2023-12-22 13:21:10,254 METRO Inference INFO: Loading state dict from checkpoint ./models/metro_release/metro_hand_state_dict.bin
2023-12-22 13:21:11,231 METRO Inference INFO: Run inference
- Skip data/dim_light/3/unscreen. Images already cropped
After coarse alignment: 284.209717
After fine alignment: 2.175960
save to  data/dim_light/3/metro_image/0001_metro_pred.jpg
After coarse alignment: 277.326782
After fine alignment: 2.160460
save to  data/dim_light/3/metro_image/0002_metro_pred.jpg
After coarse alignment: 279.830750
After fine alignment: 2.158962
save to  data/dim_light/3/metro_image/0003_metro_pred.jpg
After coarse alignment: 271.266846
After fine alignment: 2.142965

Which looks fine to me as the error after alignment looks good. But "metro_image" and "mano_fit_image" folders have white images in it like this:

Further, I checked the "metro_mano" and printed the output of mano["joints"] from "metro_mano/0001_mano.pkl" and compared with the already given mano files in preprocessed_data and they were almost equal:

# my prediction
new_pred/dim_light/3/metro_mano/0001_mano.pkl : [[[   8.495154    17.06903     -9.26395  ]
  [ -41.61213    -12.395572     8.721587 ]
  [ -75.39449    -19.07165     12.3267145]
  [ -99.15905    -35.592865     7.109699 ]
  [-134.94511    -44.250084     6.7244143]
  [ -55.110466   -82.35045      9.510782 ]
  [ -67.26341   -114.763336    17.313774 ]
  [ -75.669304  -136.09357     27.320322 ]
  [ -85.184204  -161.39336     35.048496 ]
  [ -33.910183   -98.4381      17.329912 ]
  [ -39.628586  -131.31053     24.955    ]
  [ -43.71412   -154.00847     37.94211  ]
  [ -48.900585  -184.4718      41.905487 ]
  [  -6.6895456  -92.741684    27.531963 ]
  [ -14.318776  -123.943405    31.026226 ]
  [ -15.8594885 -149.02838     39.92393  ]
  [ -21.418255  -177.08452     40.96007  ]
  [  13.468429   -82.36368     37.21435  ]
  [  16.056091  -104.14632     42.349155 ]
  [  20.082771  -124.0588      49.40557  ]
  [  22.207354  -149.0515      47.382576 ]]]
# groundtruth already provided
../data/preprocessed_data/dim_light/3/metro_mano/0001_mano.pkl : [[[   8.087268    16.605965    -9.308287 ]
  [ -41.77881    -12.450466     8.813671 ]
  [ -75.50443    -18.959173    12.084147 ]
  [ -99.14457    -35.56908      7.2946463]
  [-134.7546     -44.14023      6.619013 ]
  [ -55.1098     -82.33965      9.5261545]
  [ -67.22784   -114.70128     17.272465 ]
  [ -75.6818    -135.92662     27.256243 ]
  [ -85.07761   -161.1897      34.914143 ]
  [ -33.834896   -98.423485    17.284163 ]
  [ -39.522892  -131.12558     25.191135 ]
  [ -43.885292  -153.96214     37.649086 ]
  [ -48.765457  -184.25931     41.956997 ]
  [  -6.548176   -92.76689     27.53105  ]
  [ -14.17833   -123.8556      31.163118 ]
  [ -16.051815  -148.91757     39.79642  ]
  [ -21.312515  -176.93044     40.97242  ]
  [  13.623191   -82.3927      37.285084 ]
  [  16.057844  -104.11501     42.403454 ]
  [  20.027308  -124.03084     49.31449  ]
  [  22.230108  -148.9662      47.409035 ]]]

Why output images are blank but the mano output looks correct?