Inquiry About SFT Warm-Up Data Ratio and Its Impact on RL Performance

[starHu123]

Dear Authors,

First, thank you for your excellent work! In the paper, you mentioned using a small amount of data for supervised fine-tuning (SFT) warm-up before the RL phase. Could you kindly clarify:

1. Data Scale: What is the specific data size used for the SFT warm-up phase?
2. Proportion to RL Data: What percentage does this warm-up data account for relative to the total data used in the RL training stage?
3. Impact Analysis: How does the ratio of SFT-to-RL data (whether higher or lower) affect the final RL evaluation metrics? Have you observed any notable patterns or thresholds in your experiments?

This clarification would greatly help readers understand the relationship between warm-up strategies and RL optimization.

Thank you for your time and insights!

[rb93dett]

Hi, thank you for your interest in AlphaDrive, to answer your questions,

1. As mentioned in paper (Sec 3.1), we use 30k SFT data for warm-up, and 80k data for RL (without planning reasoning annotation), the total training data size is 110k.
2. The proportion of SFT data to the total training data is 27.28% (30k / 110k).
3. Even a small amount of warm-up data (~10k) can effectively enhance planning accuracy after RL. With the current total amount of data, allocating around 20% to warm-up data is an effective ratio, and further increases bring only marginal gains. We believe the model's potential has not yet been fully realized, and that incorporating more high-quality warm-up and RL data will lead to further improvements.