Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Computer tool can not be used with Web search call. #254

Open
BowenBryanWang opened this issue Mar 20, 2025 · 1 comment
Open

Computer tool can not be used with Web search call. #254

BowenBryanWang opened this issue Mar 20, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@BowenBryanWang
Copy link

Please read this first

  • Have you read the docs?Agents SDK docs - yes
  • Have you searched for related issues? Others may have faced similar issues.

Describe the bug

I tried a customized example with a triage agent managing two agents, which are ComputerAgent with Computer tool and SearchAgent with WebSearchTool, I'm trying to reproduce the examples shown in the demo video.

However, when make call independently to the Triage Agent, they both works well.

I've tried instructions below:

  1. Help me search for the NBA finals results in the last 5 years.
  2. Help me create a new markdown file and save it as "NBA.md".

They both worked and the Triage Agent correctly handled the request to one of the agents.

However when I tried "can you help me search for the NBA finals result of last 5 years and then write the results in a new markdown file using MarkText, save it as 'NBA.md'" to see if they can integrate together, I came across an error.

Please help me with that,

Debug information

  • Agents SDK version: (e.g. v0.0.4)
  • Python version (e.g. Python 3.9)

Repro steps

from computers import DockerComputer
from agents import (
    Agent,
    ComputerTool,
    ModelSettings,
    Runner,
    WebSearchTool,
)
with DockerComputer() as computer:
    computer_agent = Agent(
        name="Computer Agent",
        instructions="You are a real user computer agent, which means that you are connected to a real user's computer and granted full access to it. Your task is to help transfer the user's instructions to the computer and do the actions on the computer iteratively to finish the task. Also, you can handoff the task to the Search Agent as needed if you need to do online information retrieval.",
        tools=[ComputerTool(computer)],
        model="computer-use-preview",
        model_settings=ModelSettings(truncation="auto"),
        handoff_description="A real user computer environment to do GUI actions.",
    )
    search_agent = Agent(
        name="Search Agent",
        instructions="You are a searching agent connected to the Internet, you can help do information retrieval to gather useful information for the user's instruction. However, you cannot do any GUI actions on the computer unless you handoff the task to the Computer Agent.",
        tools=[WebSearchTool(user_location={"type": "approximate", "city": "New York"})],
        handoff_description="A search engine to do retrival actions.",
    )
    computer_agent.handoffs.append(search_agent)
    search_agent.handoffs.append(computer_agent)
    triage_agent = Agent(
        name="Triage Agent",
        instructions="You are a general digital agent. Your task is to help understand the user's instructions and help execute the task in a real computer environment, which is controlled by the Computer Agent. You can break down the task into smaller steps and delegate the steps to the corresponding agents. Remember always ground the task into the real computer environment by assigning the Computer Agent to do the actions. You can handoff the task to the Search Agent as needed if you need to do online information retrieval. But ALWAYS remember: you can only handoff the sub-task to one Agent at a time, which means you cannot handoff the task to both Computer Agent and Search Agent at the same time.",
        handoffs=[computer_agent, search_agent],
    )

async def main():
    result = await Runner.run(
        triage_agent,
        input="help me search for the NBA finals result of last 5 years and then write the results in a new markdown file using MarkText, save it as 'NBA.md'",
    )
    print(result)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())

Expected behavior

Expectedly the SearchAgent's context can be correctly transferred to the ComputerAgent.

@BowenBryanWang BowenBryanWang added the bug Something isn't working label Mar 20, 2025
@bajpainaman
Copy link

Hey @BowenBryanWang,

This occurs because the Agents SDK doesn't automatically pass context from one agent/tool call to another. To fix this, explicitly orchestrate the context flow:

Recommended Workaround:

  1. Run the Search Agent first to gather results explicitly in Markdown format.
  2. Then pass that Markdown content explicitly to the Computer Agent.

Example workaround in code:

search_results = await Runner.run(
    search_agent,
    input="Search NBA finals results for the last 5 years in Markdown."
)

await Runner.run(
    computer_agent,
    input=f"Use MarkText to create 'NBA.md' with this content:\n\n{search_results.output}"
)

This explicit two-step orchestration reliably solves your issue.

Hope this helps!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants