[Bug]: Timeout on analysis of long and complex Spark applications

[jankogasic]

What is your environment, configuration, and command?

MacOS. Running MCP server with task

servers:
  local:
    default: true 
    url: "http://my.spark.history.server.com:18080/"

mcp:
  transports:
    - streamable-http
  port: "18888"
  debug: true
  address: localhost

What did you do and What did you see instead?

Tool call over MCP Inspector fails after ~10 seconds with "MCP error -32001: Request timed out".

Other (more simple) tools are working for this app. After searching repository, I didn't find any setting related to changing timeout.

Additional Information. Logs.

MCP server logs

INFO:     ::1:63245 - "POST /mcp HTTP/1.1" 307 Temporary Redirect
INFO:     ::1:63246 - "POST /mcp/ HTTP/1.1" 200 OK
2025-08-04 13:58:28,605 - INFO - Processing request of type CallToolRequest
INFO:     ::1:63250 - "POST /mcp HTTP/1.1" 307 Temporary Redirect
INFO:     ::1:63270 - "POST /mcp/ HTTP/1.1" 202 Accepted

[nabuskey]

Can you give us rough size of your even logs? Delays could come from the Spark History Server itself or the MCP server. I think the 10 seconds timeout is coming from Inspector. I don't think our timeout values are set to 10.

[jankogasic]

You’re right, the timeout issue was indeed tied to MCP Inspector.

I see that you pushed changes yesterday and I’ve now set the TIMEOUT environment variable to 10 minutes on MCP side. After that, I tried avoiding MCP Inspector and switching to MCPO, which uses httpx.ReadTimeout set to 5 minutes by default. Unfortunately, it doesn’t seem easy to override that value directly.

Given that the Thrift log I’m working with is 35GB, and can go up to 250GB 😅, I’m starting to wonder whether it’s worth tweaking timeouts on the MCPO side at all. Would it be impractical for MCP to extract data that large?

Instead, would it make more sense to pinpoint a specific job or stage, and ask the LLM to analyze it for summary, bottlenecks, and recommendations?

[nabuskey]

Which tool are you calling? I did notice there are issues with certain tools where returned context is too large.

There are two places that timeout could occur.

1. From SHS especially when the event log size is large. One of my jobs that run for ~12 hours, SHS took about ~10 minutes to process with 4 cores and ~16 GB memory. Once it's processed and if you have hybrid store enabled, it's much much faster.
2. From our MCP side where python processing takes too long due to the amount of data returned. For this, we could try and improve our logic to process them faster.

Instead, would it make more sense to pinpoint a specific job or stage, and ask the LLM to analyze it for summary, bottlenecks, and recommendations?

You could do that too. If you are hitting an issue with point 1 about though, you will still see timeouts.