You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, the way gemini multimodal function calling is being done has one flaw that prevents the function calling result to be acknowledged by Gemini.
It is the fact that when the FunctionCallResultFrame is processed by OpenAIAssistantContextAggregator in services/openai.py file, you guys forgot to add 'name' field of the function call result into the Context:
In line 575, what you guys did is:
self._context.add_message(
{
"role": "tool",
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
Therefore when you call 'name = tool_result_message.get("name")' in '_tool_result' function of gemini.py, it returns None and you send function call results to Gemini live api without the function names, as such the results are not utilised.
=> The fix is very simple:
self._context.add_message(
{
"role": "tool",
"name": frame.function_name,
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
In line 575 of openai.py file
The text was updated successfully, but these errors were encountered:
Description
This is a flaw report as well as a fix proposed.
Issue description
Currently, the way gemini multimodal function calling is being done has one flaw that prevents the function calling result to be acknowledged by Gemini.
It is the fact that when the FunctionCallResultFrame is processed by OpenAIAssistantContextAggregator in services/openai.py file, you guys forgot to add 'name' field of the function call result into the Context:
In line 575, what you guys did is:
self._context.add_message(
{
"role": "tool",
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
Therefore when you call 'name = tool_result_message.get("name")' in '_tool_result' function of gemini.py, it returns None and you send function call results to Gemini live api without the function names, as such the results are not utilised.
=> The fix is very simple:
self._context.add_message(
{
"role": "tool",
"name": frame.function_name,
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
In line 575 of openai.py file
The text was updated successfully, but these errors were encountered: