How to handle ServerDisconnectedError gracefully ?

[erikseulean]

Code:

async def entity_finder(
      entity_name: str, type, session: aiohttp.ClientSession
) -> Tuple[str, Optional[str]]]:
    query = urllib.parse.urlencode({"query": entity_name, "type": type})
    async with session.get(f"{SEARCH_URL}?{query}", ssl=False) as response:
        if response.status != 200:
            print(f"Could not search for {entity_name} type={type}")
       ... (handles the response)


timeout = aiohttp.ClientTimeout(total=3600)
async with aiohttp.ClientSession(trust_env=True, timeout=timeout) as session:
    results = await asyncio.gather(*[entity_finder(entity_name, session) for entity_name in entities])

Now the entity_finder function can raise ServerDisconnectedError for example if there are way too many open connections or on a time based. keep_alive I don't think it would work in this case.
One solution is create a session for each entity and in case the exception happens, wait and retry, however this would create a lot of ClientSessions which is not ideal.

Is there another way to reconnect ? The API doesn't seem to provide anything as such.

[Dreamsorcerer]

The exception should relate to only 1 call. "reconnect" just means retry the session.get() call, you can do that with a retry loop or similar.

If you just want to ensure you get the results of the other tasks which didn't error, then use return_exceptions=True:
https://docs.python.org/3/library/asyncio-task.html#asyncio.gather

[Dreamsorcerer]

As for having too many connections, you can always throttle the number of requests to limit the load on the server. See my example here:
#7075 (reply in thread)

Or it might work by just passing a BaseConnector with the limit parameter changed: https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.BaseConnector

[erikseulean]

The exception should relate to only 1 call. "reconnect" just means retry the session.get() call, you can do that with a retry loop or similar.

Would this work ? If you get ServerDisconnectedError the connection is set to closed and calling another get wouldn't work:

aiohttp/aiohttp/client.py

Line 362 in 7d1ada8

raise RuntimeError("Session is closed")

. The second part is a different behavior and a workaround rather than handling the error in a way that you can continue reusing the same ClientSession.

[erikseulean]

I actually think you're right on this and I was wrong. I think the connection is reopened somewhere as just retrying the call after some time, works. Thanks for this.

[Dreamsorcerer]

Would this work ? If you get ServerDisconnectedError the connection is set to closed and calling another get wouldn't work:

That connection is closed, not the session. There is a connection pool and new connections will replace closed connections.