Add Logits to OpenAI ChatCompletions model

[haileyschoelkopf]

OpenAI added logits back to their ChatCompletions API.

This means we can re-add support for all tasks to this LM type! See OpenAICompletionsLM for an example of how to do this.

Contributions on this would be very welcomed if people have the bandwidth, as I may not get to this super soon but it's high priority.

[haileyschoelkopf]

I've been informed that returning the logprobs of the prompt / input which we would want to be able to support logprobs is not included.

However, supporting logprobs for ChatCompletions-mirroring APIs that do return logprobs including for echo=True would still be very desirable.

[gmottajr]

Hi Hailey,
I'm interested in contributing to this GitHub issue, but I would appreciate some additional context to better understand the task. Could you please provide more details? Specifically, it would be helpful if you could instruct me on which file(s) are affected and what specific changes are needed.
Actually, overall, I need more clarification.
Thank you very much for your attention to this matter,
looking forward to hear from you!
Cheers!

[haileyschoelkopf]

Language models return output in the form of "logits" / "logprobs" (log probabilities) over their vocabulary of possible next tokens. You can use these to sample text probabilistically or deterministically from, or, these allow you to get the estimated (log) probability of a string by the language model.

However, many closed model providers such as OpenAI stopped providing this info to users, and just gave generated text out. They've re-added that feature for their chat models now. This will allow us to run the models on loglikelihood-based multiple choice QA tasks like hellaswag which are currently implemented by taking an LM, running it on each (input + answer) and asking it to return the log probs on not just the answer, but also the input (echo=True allows this in OpenAI's Completions API), then taking only the logprobs from the answer portion. We then compare the logprobability of each multiple choice answer and say the model chose the one it thinks is most likely. Currently, because logprobs weren't available in openAI's ChatCompletions, we couldn't evaluate on the tasks we defined this way, and would need to write a new task that scores via just asking the model to generate text and checking if it matches the right answer.

The steps to complete this feature would be:

• Check out a provider that mirrors OpenAI's ChatCompletions API (e.g. Together, who also provide free credits upon signup): https://docs.together.ai/docs/openai-api-compatibility . Does this (and does OpenAI) allow for echo=True as a parameter for their chat models, allowing us to get log probabilities on a target string by feeding in (input + target) and subsetting the logprobabilities to only the target part?

• If yes, you can get echo=True from Together or other providers who can be used with OpenAI's Python SDK, then port over https://github.com/EleutherAI/lm-evaluation-harness/blob/b69ca72ec3a0294638382e0f90cf32f90d761b44/lm_eval/models/openai_completions.py#L174C2-L226C1 from OpenAICompletionsLM into OpenAIChatCompletionsLM, making changes as appropriate.

• If yes, test this on an open-source model and compare to our Huggingface implementation.

• If no, then investigate whether we can feed in just (input) to these APIs and still measure the logprobability of a target string from them.

[veekaybee]

Hi all, happy to help with this since I'm more familiar with the OpenaiChatCompletionsLM class after working on it a bit. Let me know if you'd like support or scaffolding for this!

I started by checking to see if OpenAI exposes the echo command for their chat completion models:

from openai import OpenAI

client = OpenAI()

stream = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Say this is a test"}],
    stream=True,
    echo=True
)
for chunk in stream:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

and received

File "pyenv/versions/3.10.0/lib/python3.10/site-packages/openai/_utils/_utils.py", line 272, in wrapper
    return func(*args, **kwargs)
TypeError: Completions.create() got an unexpected keyword argument 'echo'`

which makes sense since the CLIChatCompletionCreateArgs doesn't allow for it

[veekaybee]

Something I'm missing in understanding: If in the task we only use the logprobs for the response for the task evaluation, what does computing the logprobs of the input string give us in the case of the evaluation, Do we also use this as part of the calculation?

taking an LM, running it on each (input + answer) and asking it to return the log probs on not just the answer, but also the input (echo=True allows this in OpenAI's Completions API), then taking only the logprobs from the answer portion.

[haileyschoelkopf]

Something I'm missing in understanding: If in the task we only use the logprobs for the response for the task evaluation, what does computing the logprobs of the input string give us in the case of the evaluation, Do we also use this as part of the calculation?

For multi-token continuations/targets, to get the loglikelihood of the whole target, we need to get the logprob of token 0 conditioned on the input/context, the logprob of token 1 of the target conditioned on (context + token 0 of target), and so on.

With echo=True, we feed in (context + continuation) and get out all the logprobs which we can subset to the target string’s loglikelihood.

If we don’t have echo=True, then we can only feed in the inputs. Then if the model does not output token 0, the logprob of token 1 at target position 1 does not correspond to the right thing and we can’t accurately compute the loglikelihood in a (single) API call for multi-token continuations

[veekaybee]

Thanks, this is super helpful. I also just tested on Together and am getting the same issue, because it's OpenAICompletions compatible:

from openai import OpenAI

system_content = "You are a travel agent. Be descriptive and helpful."
user_content = "Tell me about San Francisco"

client = OpenAI(api_key="TOGETHER_API_KEY",
    base_url="https://api.together.xyz/v1",)

chat_completion = client.chat.completions.create(
    model="mistralai/Mixtral-8x7B-Instruct-v0.1",
    messages=[
        {"role": "system", "content": system_content},
        {"role": "user", "content": user_content},
    ],
    temperature=0.7,
    max_tokens=1024,
    echo=True
)

response = chat_completion.choices[0].message.content
print("Together response:\n", response)

TypeError: Completions.create() got an unexpected keyword argument 'echo'

It sounds like, based on this, the next step is: investigate whether we can feed in just (input) to these APIs and still measure the logprobability of a target string from them.

Where would we want this output to feed into?

It looks like this would be a good place to start to see how to get them in the completions call? And then we'd want to see how to pull them in here? , the same way we do here

[haileyschoelkopf]

Darn, that makes sense! Thanks for checking on this.

It sounds like, based on this, the next step is: investigate whether we can feed in just (input) to these APIs and still measure the logprobability of a target string from them.
Where would we want this output to feed into?

We could conceivably achieve measurement of target string loglikelihood in O(target len) API calls but this is quite expensive.

Further, since OpenAI only gives up to 5 token logprobs I think, we are limited by this—if the logprobs for our next desired target token do not appear in the top 5 we are out of luck.

Given this, one option may be to add Chat Templating support for local OpenAI Completions support once local Completions support is enabled, and get echo=True that way. Would you be willing to take on local-completions as a next step?

Orthogonal to this, we should think about how we want to support greedy-gen exact match vs. multichoice loglikelihood for multiple choice tasks to have both variants in some unified way.

[veekaybee]

Further, since OpenAI only gives up to 5 token logprobs I think, we are limited by this—if the logprobs for our next desired target token do not appear in the top 5 we are out of luck.

Ah, this is good to know, thanks!

Chat Templating support for local OpenAI Completions support once local Completions support is enabled, and get echo=True that way. Would you be willing to take on local-completions as a next step?

Yep, local completions sounds good: want to make sure I'm thinking the right way:

1. Add case statements in here for branching -if it's a local model, set echo=True and test.

For this piece:

we should think about how we want to support greedy-gen exact match vs. multichoice loglikelihood for multiple choice tasks to have both variants in some unified way.

Sounds like possibly a separate issue to file and link to this one?

[haileyschoelkopf]

Yep, local completions sounds good: want to make sure I'm thinking the right way:

Add case statements in here for branching -if it's a local model, set echo=True and test

we should always need to use echo=True for remote/local models! the additions needed for local-completions would be ~ just the same as that of local-chat-completions (tokenizer backend selectable between HF/openai and exact tokenizer can be specified by user, plus base_url configurable)

Sounds like possibly a separate issue to file and link to this one?

Definitely, will open it for tracking! Just mentioning aloud that our best solution here is (unfortunately) to forgo loglikelihood entirely perhaps.

[veekaybee]

Ah ok so these changes would be to enable local-completions, which is not the same as local-chat-completions, but do have the ability to run echo=True (for legacy reasons), to also run locally (just clarifying for myself).

That makes sense, can start on those changes!