ToolCall should be serializable

[fightingmonk]

You can use Pydantic's BaseModel.model_dump_json() to serialize a Message for persistence. However, if the message has a ToolCall then model_dump_json throws an exception because ToolCall.InvocableTool is a Callable that Pydantic can't serialize.

Conversation history serialization is important for resilient workflows and agents. I want to snapshot the current message history and requested tool calls before I start invoking the tools, so I can resume the process if something goes wring while a tool is running. I'm starting to think this is a blocker to building out a resilient agent with ell. Also a huge help during development so you can make code changes and resume the workflow or agent from any state snapshot.

Do you have ideas on how you'd approach solving this? I'm happy to put together a PR.

Here's a reproduction of the issue:

import ell

@ell.complex(model="gpt-4o")
def test_no_tool():
    return "say hello"

message = test_no_tool()
print("Normal message serializes")
print(message.model_dump_json())

@ell.tool()
def be_a_tool():
    """This is the be_a_tool function"""
    return "hi"

@ell.complex(model="gpt-4o", tools=[be_a_tool])
def test_tool():
    return "call the be_a_tool tool"

message = test_tool()
if message.tool_calls:
    print("Message with a ToolCall throws an exception...")
    print(message.model_dump_json())
else:
    print("Test setup failed, LLM didn't request a tool call")

[alex-dixon]

Thanks for the issue!

Are you only looking to record the fact the model chose to invoke a tool? Or do you also need to call the tool later after serializing the messages?

If storing and invoking the tool later, there’s the general issue of whether we invoke the latest version of the tool function, or the exact version the model chose at that point in the conversation history.

If we want to ensure we always call the exact version, we need to serialize an invocable form of that exact code. This could be inline in the message history but that will probably bloat it.
ell’s lexical closure may already produce a code string that is “eval”-able, so long as the calling process can satisfy its imports. We already save tool functions at each revision to the database, so a reference to the tool function’s lmp id and version can pull out the exact code.

For Python, Pickle may provide a better serialization/deserialization format for functions. We could store the pickle representation in the db as a separate column so it would be available via the same lookup as the lexical closure. I’m not up on how Pickle actually accomplishes so there could be some gotchas with this approach I’m not mentioning here.

If we don’t need to guarantee the exact version of the tool is available, we could implement this with a registry of functions and a look up by name. ToolCall could be serialized to JSON as data (lmp-id, version, fqn) and later looked up in a registry of tool functions to be invoked if needed. Ell might be able to maintain this registry as a side effect of tool call function declaration. Ell could still check the calling process has the exact tool call function version and warn/error if not based on configuration. Overall this might be simpler. If we’re interested in supporting Typescript or other languages, this is more language agnostic than Pickle and would more directly support message serialization and tool calls across languages.

[fightingmonk]

Thanks for the ideas. I did some more thinking and maybe my proposed solution isn't quite right...

My workflow has some tools that solicit user input and/or trigger a complex external process, and then wait for a result to arrive in a message queue. My main process can restart or move to another node while waiting for that result - hence the need to serialize.

In pseudocode, the basic flow is:

@tool()
def tool_that_waits_for_queue(input) -> str:
    return start_process(input, queue_id=tool_call_id) # returns a queue_id we use to read a result from queue

@ell.complex(tools=[tool_that_waits_for_queue])
def prompt_that_uses_tool(messages: List[Message]):
    return messages # invoke LLM with message list and available tools

# start with a list of messages that instruct the LLM to invoke 1 or more tools
messages = [ell.system("please call the tool for me", ell.user("I need you to research this..."), ]

# invoke LLM with available tools, we get a response w/ ToolCall(s)
result = prompt_that_uses_tool(messages)

# when we call the tools, we get back a queue_id in the ToolResult's result
tool_queue_id_message = result.call_tools_and_collect_as_message()
queue_ids = [r.result[0].text for r in  tool_queue_id_message.tool_results)

# save message history, tool call response message, and queue_ids so we can continue this conversation once results are in
save(messages, result, queue_ids)

# sometime later, in another process or another node
research = get_results_for_ids(queue_ids)
# tool_call_id(s) was in the previous `result` Message, need to reconstruct that state so ell can match results up correctly
messages = messages + [ToolResult(result=tr, tool_call_id=???) for tr in research]
run_next_step(messages)

So maybe I don't need to serialize the ToolCall itself - I need to be able to re-construct the input Message list to the LLM and the fact that the LLM responded with a tool call request, and then once results are available in the queue I need to create ToolResult(s) from them, add those to the Message list, and call the LLM again. And I think I have to preserve the tool_call_ids to make sure everything gets stitched together correctly.

Since I'm invoking the tool as soon as it's requested by the LLM and then assuming the ToolResult will contain the value that shows up in the queue later, I don't need to preserve the actual callable tool. I just have to make the Message history look like it has valid ToolCalls.

The draft PR I opened is a simple serializer/validator patch on ToolCall that uses the tool's FQN to find the current tool implementation. It works for 0-param tools but is pretty fragile and has a lot of edge cases so I don't know that it's the right direction for what I need to accomplish.