output the thought or explain

[vectorNN]

Thank you for providing such an excellent thesis. I have two questions:

1. When conducting testing and inference, by adding prompt words, the model can output a simple explanation. I would like to ask whether the training data includes the thinking process and explanations. Does the model support reasoning? In the dataset I only saw question-answer pairs, and there were no "explain" or "thought" fields.
2. How about the inference time? For the task of desktop organization, if the machine calls the model to infer the next step for each operation, will it seem rather laggy? I tested it on my machine with a 4090 graphics card, and each step takes approximately 3 to 4 seconds. How long does it take on your end? And how can we solve the lag problem during the step-by-step process?
   I'm looking forward to your reply.

[Yuheng2000]

1. We included over 600k math/reasoning samples in our training dataset, such as Geo170k, GeoQA+, Geometry3k, and Super-Clevr. These data are sourced from LLaVA-OneVision, primarily presented in the form of questions with only answers provided. However, a small amount of data also includes the thought processes, though not explicitly labeled with explain/thought fields, such as in mathqa and mavis_math. For more information, please refer to Table 16 in the LLaVA-OneVision paper: LLaVA-OneVision. Consequently, RoboBrain V1.0 may not excel in reasoning but can demonstrate some reasoning capabilities under suitable prompts. Our other work discusses visual reasoning here, and we plan to implement stronger embodied reasoning capabilities in RoboBrain V2.0. We welcome your interest!

2. For reference, we previously deployed RoboBrain on a machine with an RTX 4090, utilizing vllm to accelerate inference. With a single image and 1k tokens as input, the average output token count was 32, with a total of 705 samples processed in approximately 10 minutes, resulting in an average response time of under 1 second. In practical use, a 1-second response time is still quite acceptable. Have you employed any inference acceleration techniques to enhance the model's response time, such as vllm?