how to implement "Moto w/ Motion Token"

[nanamma]

thx for your hard work

Is there a configuration setting corresponding to the last two rows of Table 8 in the paper? My training results are close to those of "Moto w/o Motion Token." How can I implement the configuration for "Moto w/ Motion Token"? The code for the attention-mask design seems quite complex.

[nanamma]

Also, I found that the attention mask of part of latent motion in src/models/moto_gpt.py MotoGPT.forward() is not casual, which conflicts with Next Latent Motion Token Prediction.

[nanamma]

Regarding the training process, could you show the detailed training configuration (yaml config file) for each stage?
I really want to reproduce your training process. But my current reproduction results are not as well as yours.

[ChenYi99]

Also, I found that the attention mask of part of latent motion in src/models/moto_gpt.py MotoGPT.forward() is not casual, which conflicts with Next Latent Motion Token Prediction.

The causal attention is implemented in the GPT2 code: src/models/trajectory_gpt2.py#L340.
The causal attention mask is added to our attention mask in src/models/trajectory_gpt2.py#L346

[ChenYi99]

Regarding the training process, could you show the detailed training configuration (yaml config file) for each stage? I really want to reproduce your training process. But my current reproduction results are not as well as yours.

I have provided the detailed training configuration for each stage in README. Did you strictly follow them to reproduce the results? Moreover, could you provide more details about your implementation?

[nanamma]

Regarding the training process, could you show the detailed training configuration (yaml config file) for each stage? I really want to reproduce your training process. But my current reproduction results are not as well as yours.

I have provided the detailed training configuration for each stage in README. Did you strictly follow them to reproduce the results? Moreover, could you provide more details about your implementation?

I follow this #8 (comment) and your paper to adjust the hyper parameters.

maybe can you provide the log of each stage? especially the loss. So that i can check whether my training of each stage is enough. thanks a lot

[ChenYi99]

pretrain_moto_gpt_on_calvin.log
finetune_moto_gpt_on_calvin.log
train_latent_motion_tokenizer_on_calvin.log

[feng-tianli]

I followed the default configurations to train the model, but the results I obtained are far from those reported in the paper. As mentioned above, my results are also closer to those of "Moto w/o Motion Token." Did you manage to resolve this issue later? @nanamma

In addition, I noticed that the number of active codes during my training does not reach the levels mentioned in the training logs. The number of active codebook entries remains around 80–90 throughout training.

Could you please provide any insights or suggestions regarding these issues?

[nanamma]

I followed the default configurations to train the model, but the results I obtained are far from those reported in the paper. As mentioned above, my results are also closer to those of "Moto w/o Motion Token." Did you manage to resolve this issue later? @nanamma

In addition, I noticed that the number of active codes during my training does not reach the levels mentioned in the training logs. The number of active codebook entries remains around 80–90 throughout training.

Could you please provide any insights or suggestions regarding these issues?

My reproduction results for the first stage are quite good. The number of activated codes is close to what was reported in the original paper. And the losses are close, too.
It's best not to modify the training configuration, especially parameters like batch_size and gradient_accumulation_step. Although they can theoretically be adjusted proportionally, doing so may lead to some differences in results.