Clarification on default top_k sampling parameter

[MostHumble]

Hi 🤗 team,

I'm writing to inquire about the design choice to set the default top_k sampling parameter to 50 in the transformers library.

transformers/src/transformers/generation/configuration_utils.py

Line 431 in f4fc422

self.top_k = kwargs.pop("top_k", 50)

It appears top_k is the only sampling parameter with an opinionated default value, as others like top_p are typically set to a neutral value (e.g., 1.0). For consistency and to allow for more flexible default behavior (i.e., no top_k filtering by default), I would personally advocate for a default value of -1, similar to how vLLM handles its sampling parameters (vLLM SamplingParams documentation).

Could you please clarify the reasoning behind this specific default?

Thank you for your time and consideration.

[zucchini-nlp]

Hey @MostHumble !

Thanks for raising this issue. I agree with you that having a not None default value isn't the best design choice, and that is something we have due to legacy reasons. We are aware of the shortcomings and the future plan will be to gradually transition to setting config values to None by default

cc @gante as he's the generation code owner and can share more on this