About performance

[Zbozhou]

Hello,

I followed the official configuration files and conducted the training process on the CALVIN dataset, including:

• Stage 1: Training the Latent Motion Tokenizer (20 epochs)

• Stage 2: Pre-training Moto-GPT (20 epochs)

• Stage 3: Fine-tuning Moto-GPT (20 epochs)

However, the final evaluation metric I obtained is only 1.8, which is significantly lower than the reported 3.1 from the provided checkpoint.

I would like to ask if there might be any missing steps or overlooked details that could have caused the performance drop.

Thank you!

[ChenYi99]

Hello,

We initialize fine-tuning from the pretrained checkpoint at 10 epochs, so please try fine-tuning starting from that checkpoint. Regarding the tokenizer, we have restructured the code, and the actual training steps are detailed in our paper. Additionally, we provide checkpoints for each stage—you can use these to continue training or compare results to identify which stage might be causing the performance drop.

[Zbozhou]

Thanks for your reply. So you mean, Stage 1 for 20 epochs, Stage 2 for 10 epochs, Stage 3 for 20 epochs, is that right?

[ChenYi99]

The detailed training steps for each stage can be found in our paper.

[srijandas07]

@ChenYi99 Unfortunately, we are also facing the same problem in reproducing the results. I guess, a clarification on github might help the users to reproduce the results which clearly cannot be done by referring the paper. Thank you for the nice documentation btw and congos for ICCV oral!

[ChenYi99]

Thank you for your interest in our work.

For better code readability, we refactored the code before release and did not strictly verify the training results at each stage based on the refactored version. We will take time to compare the released code with the previous version to identify possible causes.