Description
Dear authors, thank you for your great work!
I'm trying to run the inference for planning tasks, so I downloaded the pretrained model and split files from your google-drive and adjusted the advqa_t5_elm.yaml config file by changing the following lines:
images: storage: 'data/nuscenes' annotations: train: storage: 'data/drivelm/planning_train.json' test: storage: 'data/drivelm/planning_val.json' val: storage: 'data/drivelm/planning_val.json'
And changed the resume_ckpt_path to load the pretrained model:
resume_ckpt_path: "lavis/output/elm_checkpoint_pretrain.pth"
Finally I enabled the inference as you referred in the README by setting
evaluate: True
All paths are given as absolute paths in my file just for here I changed them to relative paths and as a far as I can tell, there were no issues in loading the split files or the checkpoint file.
When the script now tries to load the checkpoint-file, two errors occur:
RuntimeError: Error(s) in loading state_dict for PeftModel: Missing key(s)in state_dict: "base_model.model.extra_query_tokens", "base_model.model.t5_model.encoder.block.0.layer.0.SelfAttention.q.base_layer.weight", "base_model.model.t5_model.encoder.block.0.layer.0.SelfAttention.v.base_layer.weight",...
and later:
Unexpected key(s) in state_dict: "base_model.model.t5_model.encoder.block.0.layer.0.SelfAttention.q.weight", "base_model.model.t5_model.encoder.block.0.layer.0.SelfAttention.v.weight", "base_model.model.t5_model.encoder.block.1.layer.0.SelfAttention.q.weight",...
I also tried the other checkpoint files (ckpt_elm_drivelm, ckpt_elm_box.pth) and each time those errors occur, only the missing keys and the unexpected keys differ for each of those checkpoints.
Can you help me with this issue? Thank you very much in advance!