Ques regarding deploying api server, help

[ychel]

After i deployed api connection, i was using completions mode. I constantly receive same result when I ask the same prompt for different times. Does any one know what issue may cause this problem?

[airaria]

Did you mean that you expect different responses with the same input? You can set the temperature to a higher value as described here

[ychel]

<Did you mean that you expect different responses with the same input?> Yes. I tried to set temperature and top_p higher, but it does not work.

[sunyuhan19981208]

generation_config = GenerationConfig(
    temperature=temperature,
    top_p=top_p,
    top_k=top_k,
    num_beams=num_beams,
    do_sample=True,
    **kwargs,
 )

@ychel You can add do_sample=True at openai_api_server.py's predict function, I will fix this insufficient problem ASAP.

[ychel]

much appreciate it