Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function calling fix for pipecat gemini multimodal function calling #908

Open
phamtrung0633 opened this issue Dec 23, 2024 · 0 comments
Open

Comments

@phamtrung0633
Copy link

Description

This is a flaw report as well as a fix proposed.

Issue description

Currently, the way gemini multimodal function calling is being done has one flaw that prevents the function calling result to be acknowledged by Gemini.
It is the fact that when the FunctionCallResultFrame is processed by OpenAIAssistantContextAggregator in services/openai.py file, you guys forgot to add 'name' field of the function call result into the Context:
In line 575, what you guys did is:
self._context.add_message(
{
"role": "tool",
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
Therefore when you call 'name = tool_result_message.get("name")' in '_tool_result' function of gemini.py, it returns None and you send function call results to Gemini live api without the function names, as such the results are not utilised.

=> The fix is very simple:
self._context.add_message(
{
"role": "tool",
"name": frame.function_name,
"content": json.dumps(frame.result),
"tool_call_id": frame.tool_call_id,
}
)
In line 575 of openai.py file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant