Dataset creation

[apbose]

Hi,
Thanks for opensourcing this amazing work.
I am trying to deploy a demo of the model using TRT and TRT-LLM on Nvidia ORIN Drive platform. However running CARLA simulator on DRIVE platforms does not seem to be a feasible option.
I was wondering if you could point me to the tools you used for the dataset creation for creating the model checkpoints. Or if you could point me to the dataset. You could also point me to some other alternatives in case I am missing something. Thanks!

[Jieqianyu]

Thank you for your interest in our work! Due to the large size of the dataset, we have provided a subset of the CarlaSim test dataset accessible at https://huggingface.co/datasets/Jianbiao/CarlaSim. We hope this will be helpful to you.

[apbose]

Thanks for the pointer to the dataset above. The above dataset looks to be for the Qwen-VL model. I have couple of follow up questions:

1. The Qwen-VL and Qwen1.5 models provided, are they the SFT pretrained checkpoints? My scores generated while running with LeapAD/leaderboard/data/routes_town05_short.xml does not seem to be correct
2. Also, what is the data used for pretraining of the Qwen1.5 model? Is it the response generated by Qwen-VL and the memory bank generated by the heuristic process during training? And which route data did you use for the fine tuning of the Qwen-VL? The above link contains the town05 data, but it seems to be a subset of the whole data.

[Jieqianyu]

1. The provided checkpoints are pre-trained; you may need to uncomment the routes in routes_town05_short.xml to obtain the average score on the Carla short benchmark.

[Jieqianyu]

2. The data used for training the Qwen1.5 is the response generated by Qwen-VL and the analytical process. We train qwen-vl in town01,04,06,07,10, and test on town05. Due to the large size of the dataset, we only release part of the test set.

[apbose]

In the above table I see that in the first set of rows, you have results for town 5, with a comparative study on how Memory Few- shot and SFT improves the result. Is my understanding correct when I say that in Memory Few-shot, you get the result by feeding in the current sample Dc, along with the topK Di of the "similar" samples in the memory database. How is the reasoning and decisions taken into account in finding the similarity? (section 3.4 in paper)

Also utilizing the memory bank of the Town05 data makes sense to me, but I do not understand as to why are we doing a SFT on town05 data when we are testing on it? (which seems to be the last row). I think I am missing something in my understanding.

So when you say that we have the 9K samples in the memory bank (without Town5) with pretraining on town01,04,06,07,10, is it the row02 in the first set of rows with results on test Town05 when I run the evaluation script (DS- 75.73, RC- 92.10, IS- 82.66) ? I think I am getting confused with pretraining mentioned in the comments above and SFT in the paper. Are they the same? If not, could you please help me understand the difference between the two?

Thanks in advance for the clarification!