Unable to Reproduce MOTO Results on CALVIN Benchmark

[ManishGovind]

Hello @yxgeee, @ChenYi99, @yeliudev

Thank you for your great work! The codebase is well-structured and easy to follow.

I’ve been trying to reproduce the MOTO results on the CALVIN benchmark by training all three stages as described in the paper. However, the results I’m obtaining do not match those reported in the paper. I’ve followed the training setup exactly—using the same number of epochs, batch size, and gradient accumulation steps. To verify, I also compared the training losses with the logs you’ve provided here in this issue #10.
I’ve attached my logs and included comparison plots below for reference.

pretrain_moto_gpt_on_calvin.log
finetune_moto_gpt_on_calvin.log
train_latent_motion_tokenizer_on_calvin.log

Originally posted by @ChenYi99 in #10

Training hyper parameters that i follow to reproduce results :

Stage 1(Latent motion Tokenizer) : 150K steps(Epochs 11) ; Effective Batch size - 256

Stage 2(Pre-training Moto-GPT) : 10 Epochs ; Effective Batch Size - 512

Stage 3(Finetuning Moto-GPT) : 18 Epochs ; Effective Batch Size - 512

Results Reproduced :

Average successful sequence length: 1.821
Success rates for 5 tasks completed in a row on 1000 chains:

T1: 73.4%
T2: 47.6%
T3: 30.1%
T4: 19.7%
T5: 11.3%

Here are the training logs for my 3 stages of training and plots of reproduced logs comparing with provided log for three stages :

Reproduced Training logs*

reproduce-finetune-moto-gpt-calvin-from-lmt-150k.log
reproduce-latent-motion-tokenizer-calvin.log
reproduce-pretrain-moto-gpt-calvin.log

Comparison of Training losses

Debugging with provided checkpoints

Interestingly, I observed that when I use the provided Stage 2 checkpoint and proceed with Stage 3 finetuning, the results are close to what is reported in the paper. However, when I re-train Stage 2 myself, the downstream performance drops drastically.

Here’s a snapshot of the experiments I conducted to narrow down the issue:

Could you kindly clarify:

• Are there any undocumented details or changes related to Stage 2 training?
• Were there any differences in training environments, seeds, or data processing that might impact reproducibility?
• Are there known issues with the current Stage 2 training code?

Also, has there been any recent update to the codebase that might affect training behavior or results?

I’d really appreciate your help in clarifying this. I’m happy to share additional logs or experiment details if needed.

Thanks again for your excellent work and for open-sourcing this project

[ChenYi99]

Hi,

Thanks a lot for your interest and for sharing the detailed logs and comparisons!

We did a big refactor of the code before releasing it to make it easier to read, but we didn’t fully re-check all training stages after that. We’ll take some time to review the code and results to see if anything might be causing the differences you found.

Really appreciate your feedback and patience. We’ll keep you updated!

Best,

[ManishGovind]

Thank you for the quick response and for taking time to look into this! That makes sense regarding the refactor. I really appreciate your willingness to review the code and investigate further.

Looking forward to your response and any further updates!

[dominickrei]

Hello @yxgeee, @ChenYi99, @yeliudev,

Thank you for the open-sourced work! Since it has been 1 month I wanted to follow up on this issue. I am working with @ManishGovind and have also verified that the results we obtain using Moto are far from what is reported in the paper. Additionally, after following the details in the paper we have been unable to successfully train Moto on real-world robot tasks we have collected, but we have been able to train other VLAs using the same data. As we have been unable to reproduce any of the results we think some issues might have been introduced during the code refactoring your team has mentioned.

Has your team had the chance to review the refactored code and confirm it can be used to reproduce the results? If not, maybe you could temporarily release the working version of the code to enable reproducibility and follow up works?

Thank you!

[ChenYi99]

Hi, thanks for the follow-up. For CALVIN, our tokenizer was trained on the original .npz RGB data, while pretrain/finetune used compressed JPEG images, which may have caused some performance drop—you could try pretraining also with .npz. At evaluation time, it may also help to check multiple epoch checkpoints. Another possible factor is environment/library updates—I didn’t save the environment after finishing the experiments last year and have left the company, so I can’t recover the exact setup. It might help to check whether any library updates after last October could be affecting performance. I’ve attached my original tokenizer and Moto-GPT pretrain logs for reference (https://drive.google.com/drive/folders/1SSuXpBQ6x8bnwkjQhthiC52_taAfBzmX?usp=sharing).

For the real-robot experiments, could you share more details about your setup and training procedure?

[ManishGovind]

Hi @ChenYi99,

Thanks again for the update regarding the CALVIN results. As suggested, I experimented with different Stage 3 checkpoints and observed slight improvements in performance.

I had a few follow-up questions regarding both the dataset and pretraining / finetuning implementation :

1. Dataset Creation

While inspecting the dataset code, I noticed that if the extract_actions or extract_frames methods fail to retrieve future datapoints, the code defaults to using dummy image tensors and dummy actions. https://github.com/TencentARC/Moto/blob/63244f4e25d2c32d3b4b38d96b524ac301dadb5c/common/data/datasets.py#L182
https://github.com/TencentARC/Moto/blob/63244f4e25d2c32d3b4b38d96b524ac301dadb5c/common/data/datasets.py#L190

• Is this the intended behavior?
• Wouldn’t this introduce noisy supervision during training, especially for chunks near the end of episodes?

2. Moto-GPT Pretraining/ Finetuning Implementation in Training loop

Shouldn't this line in moto_gpt_trainer.py be Train=False in eval mode ? https://github.com/TencentARC/Moto/blob/63244f4e25d2c32d3b4b38d96b524ac301dadb5c/moto_gpt/src/trainers/moto_gpt_trainer.py#L172

Real-Robot Experimental Settings and Training procedure

We are currently adapting MOTO for real-world experiments using the XARM7 robot. We collected 30 trajectories per task and followed the fine-tuning procedure outlined in the paper. Here's a summary of our setup:

Latent Motion Tokenizer(stage 1)

• Pretrained on the OXE dataset
• Further finetuned for *20 epochs using our 60 real-world trajectories (30 per task)

Moto-GPT Pretraining (Stage 2)

• Initialized from OXE-pretrained Moto-GPT checkpoint
• Trained further for 5 epochs on a combined dataset of OXE (excluding RT-1 and BridgeData) + our 60 real-world trajectories

Moto-GPT Finetuning (Stage 3)

• Finetuned using only 30 trajectories of Task 1 (Pick-and-Place) for 10 epochs
• Resumed from the above Stage 2 checkpoint
• Arm and gripper actions were normalized and clipped to $-1, 1$

Issue During Real-World Inference

Despite following this setup, during real-world inference on Task 1, the model outputs nearly identical actions across time steps, and Interestingly, when I repeat Stage 3 finetuning using the RT-1–finetuned checkpoint as the base (instead of our combined OXE + real data checkpoint), the robot starts to move and even performs successful object placement but fails to pick up the object. This suggests 30 trajectories per task are not enough to finetune Moto-GPT

Few Questions Related to Real world experiments

1. How sensitive is Stage 3 finetuning to the source of the pretrained checkpoint( on OXE + real world for Latent motion prediction) vs RT-1 finetuned checkpoint( latent motion and action token prediction)?
2. Roughly how many trajectories per task are needed for decent real-world performance?
3. Do you recommend combining tasks or keeping task-specific finetuning for better generalization ?

we’d really appreciate if you could clarify our doubts or share any insights you might have regarding this. Thanks again for all your help!!

[ChenYi99]

Thank you for your detailed questions and for sharing your experimental setup. Let me address each of your concerns:

1. Dataset Creation:
   • We use latent_mask and mask to ignore these dummy values during loss computation, so they don't contribute to noisy supervision.

• The masks ensure that training focuses only on valid future datapoints.

2. Moto-GPT Training:
   • The train=True in eval mode is correct - it only controls whether image augmentation is applied during evaluation, not the model's training behavior.

Regarding your real-robot experiments:

Checkpoint Sources:
• Could you clarify if the "RT-1-pretrained checkpoint" refers to our provided OXE-pretrained checkpoint?

• Your observation suggests possible overfitting when fine-tuning tokenizer and Moto-GPT on small real-world data for extended periods.

Stage 2 Pretraining Questions:
• I'm curious why you excluded RT-1 and BridgeData from OXE? These are among the most diverse and large-scale datasets in OXE, and their exclusion might significantly impact pretraining quality.

Stage 3 Fine-tuning Recommendations:

1. With limited data (30 trajectories), I suggest:
   • Extending fine-tuning to 20 epochs

   • Reducing print_step to 1 in training config (as learning rate updates occur at print steps)

2. For single-task finetuning, 30-50 trajectories should suffice if:
   • Camera views clearly show x-y-z movements (similar to Bridge/RT-1 views)

   • Task distribution aligns reasonably with pretraining data

   • Consider concatenating gripper-view with main-view to a single RGB image if available

Additional Suggestions:
• I can share a pretrained checkpoint based on a larger dataset (using the DINO-v2 encoder) that might perform better for your stage-3 fine-tuning on real-world data (https://www.modelscope.cn/models/MilkWhite/Moto_trained_on_oxe_ssv2).

• For training the tokenizer from scratch, setting do_random_resized_crop=True may improve its generalization capability.

[ManishGovind]

Hi @ChenYi99,

Thank you for the detailed clarifications on the CALVIN dataset and Moto-GPT training implementation.

Regarding the RT-1–pretrained checkpoint, I was referring to the Stage 3–finetuned Moto-GPT model trained using RT-1 action-paired data.

As for excluding RT-1 and BridgeData from Stage 2: this was due to time constraints, and we had initially assumed that the OXE checkpoint already incorporated those datasets. Your explanation about the potential forgetting effect from training on only ~20% of OXE (excluding RT-1 and Bridge) makes sense — this could indeed be the main reason behind the performance drop. We’ll update our pretraining pipeline to include these datasets.

Thank you also for the helpful suggestions on real-world finetuning and for sharing a stronger pretrained checkpoint (with DINOv2 encoder) which is trained on larger dataset ,that would be greatly appreciated and will definitely help us improve our results.

[ManishGovind]

Hi, thanks for the follow-up. For CALVIN, our tokenizer was trained on the original .npz RGB data, while pretrain/finetune used compressed JPEG images, which may have caused some performance drop—you could try pretraining also with .npz. At evaluation time, it may also help to check multiple epoch checkpoints. Another possible factor is environment/library updates—I didn’t save the environment after finishing the experiments last year and have left the company, so I can’t recover the exact setup. It might help to check whether any library updates after last October could be affecting performance. I’ve attached my original tokenizer and Moto-GPT pretrain logs for reference (https://drive.google.com/drive/folders/1SSuXpBQ6x8bnwkjQhthiC52_taAfBzmX?usp=sharing).

For the real-robot experiments, could you share more details about your setup and training procedure?

I had a couple of quick follow-up questions after going through the log files you shared:

In the paper and logs, I noticed some discrepancy in the number of epochs used for Stage 2 (Moto-GPT pretraining). Could you clarify whether it should be trained for 10 epochs or 20 epochs?

Train vs. Test Chunk Size:
I observed that the test chunk size is set to 8, while train chunk size is 5 in the provided scripts.

Could this mismatch affect evaluation, especially if temporal context length impacts prediction quality?

[ChenYi99]

Regarding the number of epochs for Stage 2 pretraining: Due to resource constraints, we primarily used the 10-epoch pretrained checkpoint as our default in the experiments. Although the model was later fully pretrained for 20 epochs, we did not extensively evaluate checkpoints from other epoch intervals.

About the chunk size mismatch: The train chunk size of 5 refers to the number of action steps between two consecutive frames. Since our model predicts two consecutive frames at a time, it effectively predicts 10 action steps per forward pass. During inference, we also generate 10 steps at once but only execute the first 8. This ensures consistency in the temporal context and should not affect evaluation quality.

[ManishGovind]

Hi @ChenYi99,

Thanks for clarifying the number of pretraining epochs and the chunk size mismatch.

I wanted to share a quick update:
After including RT-1 and BridgeData in Stage 2 pretraining, the real-robot performance improved — the robot now moves toward the object correctly. However, there are still some issues with picking and placing failures that we are trying to resolve.

While adapting the evaluation pipeline (evaluation_calvin.py and evaluation_simpler.py) to our robot, I noticed one thing-

MLMP Probability during inference :
The mask_latent_motion_probability is set to 1.0, meaning the latent motion tokens are always masked during inference. Shouldn't this be set to 0.0 so that the model can actually use the latent tokens?

Moto vs Moto W/o Latent Motion Tokens :
We also compared full Moto (with latent motion tokens) vs. Moto without latent tokens (i.e., Stage 3 only, trained from scratch). Surprisingly, Moto without latent tokens performed better in calvin experiments.

Only changes we made:

Stage 1: Set legacy = False (previously True)

Stage 2 & 3: Increased effective batch size from 512 → 1024

Since we have sufficient compute, we experimented with these changes , but found that latent motion tokens aren't helping, which is unexpected.

Could you please clarify how sensitive are these hyperparameters and MLMP probability issue during inference ?

Thanks!!

[ChenYi99]

Hi, thank you for sharing the detailed observations.

• Tokenizer legacy: We reused an existing VQ codebase where the default was legacy=True. We didn’t specifically ablate this option; it was exposed in the config simply because we noticed this flag when organizing the code. One tip: when training the tokenizer, setting do_random_resize_crop=True can sometimes improve generalization.

• Batch size: We haven’t systematically ablated batch size. For Stage-3 fine-tuning, especially when real-robot data is limited, using a smaller batch size (e.g., 32) is more reasonable. You may also try directly fine-tuning Stage-3 on top of the two pretrained checkpoints we provided.

• Real-robot setup: In our paper, the real-robot experiments followed the FANUC dataset setting, where each action dimension is discretized (x, y, z, roll, yaw, pitch ∈ {−0.02, 0, 0.02}, gripper ∈ {0,1}). This discretization may have made action prediction easier in our setting compared to continuous actions.

• Pretraining data: Another possible factor is embodiment mismatch. The current pretraining corpus may contain too few demonstrations with robots similar to your setup. You could check the Open-X-Embodiment table and try mixing in more similar-embodiment datasets with higher sampling weight during pretraining.

• Input design: Concatenating the side-view and gripper-view images into a single RGB input sometimes may help improve grasp precision.

• MLMP probability: During evaluation, we set it to 0.0 so inference is faster. In fine-tuning, when the dataset is small, using 1.0 can sometimes improve policy performance, so you might also try enabling it.

Hope this helps.

[ManishGovind]

@ChenYi99 Thanks again for the insights on the real-robot setup and the helpful suggestions to improve performance.

Just to clarify — the results I shared comparing MOTO vs. MOTO without motion tokens were actually on the CALVIN Benchmark, not from real-robot experiments.

I had a quick follow-up question on Calvin experiments:
Have you performed any ablations with different batch sizes during Stage 2 or Stage 3 training? Or were the results reported in the paper based solely on an effective batch size of 512?

In our case, the results I shared were based on the following changes:

Tokenizer: Set legacy = False to align with the actual VQ-VAE codebook and commit loss behavior.

Effective Batch Size: Increased from 512 to 1024 using gradient accumulation.

Curious if you’ve experimented with larger batch sizes and whether that impacted performance in your setup.

Hi @ChenYi99,

Thanks for clarifying the number of pretraining epochs and the chunk size mismatch.

I wanted to share a quick update: After including RT-1 and BridgeData in Stage 2 pretraining, the real-robot performance improved — the robot now moves toward the object correctly. However, there are still some issues with picking and placing failures that we are trying to resolve.

While adapting the evaluation pipeline (evaluation_calvin.py and evaluation_simpler.py) to our robot, I noticed one thing-

MLMP Probability during inference : The mask_latent_motion_probability is set to 1.0, meaning the latent motion tokens are always masked during inference. Shouldn't this be set to 0.0 so that the model can actually use the latent tokens?

Moto vs Moto W/o Latent Motion Tokens : We also compared full Moto (with latent motion tokens) vs. Moto without latent tokens (i.e., Stage 3 only, trained from scratch). Surprisingly, Moto without latent tokens performed better in calvin experiments.

Only changes we made:

Stage 1: Set legacy = False (previously True)

Stage 2 & 3: Increased effective batch size from 512 → 1024

Since we have sufficient compute, we experimented with these changes , but found that latent motion tokens aren't helping, which is unexpected.

Could you please clarify how sensitive are these hyperparameters and MLMP probability issue during inference ?

Thanks!!

[ChenYi99]

Due to limited training resources, I haven’t conducted ablation studies on training batch size. Additionally, I recently reran the Stage-2 pretraining on the CALVIN benchmark and was able to reproduce the results. I plan to release the updated code on GitHub within the next two weeks.

One thing worth noting: using unique positional embeddings for each latent motion token and latent motion query token is helpful to improve pretraining stability.

[ManishGovind]

Hi @ChenYi99,

Thank you for the update on MOTO pretraining — great to hear that you were able to reproduce the results! Looking forward to the updated code. In the meantime, I’ll experiment with using positional embeddings for latent motion tokens.

I also noticed that the use_timestep_embedding flag doesn’t seem to be used during fine-tuning with action data.
Do you think enabling timestep embeddings during fine-tuning could lead to a significant performance improvement?

[ManishGovind]

Hi @ChenYi99 ,

Hope you're doing well!

Just checking in to see if there’s any update on the release of the updated code that you mentioned earlier.
Also, I experimented with incorporating positional embeddings for the latent motion tokens and observed improvement in performance. Thanks for that suggestion and I really appreciate your time for looking into this matter.

-Manish