No retries on ConnectTimeoutError

[arogozhnikov]

I'm observing ConnectTimeoutError sometimes, which is likely caused by the network and searching for a right way to handle thimeouts.

I see s3fs has retries with exponential backoff, but it ignores

How this can be tested/reproduced:

sudo apt install iproute2    # likely you already have it
sudo tc qdisc add dev eth0 root netem delay 3500ms    # set delay on network
# sudo tc qdisc del dev eth0 root     # use to remove delay when done with testing.

now in python:

import s3fs
s3fs.S3FileSystem().exists('s3://example-bucker/example-folder')

This crashes with

ConnectTimeoutError: Connect timeout on endpoint URL: "https://example-bucker.s3.us-west-2.amazonaws.com/example-folder"

This failure is expected, BUT when I insert logging in s3fs, it does not retry this exception (though default retries=5). Should this exception be retried?

[martindurant]

This failure is expected, BUT when I insert logging in s3fs, it does not retry this exception (though default retries=5). Should this exception be retried?

Although networks can be unreliable, I would think that ConnectionTimeout should not in general be retriable at the s3fs level. It is rather long and I believe the network stack itself does retries at a lower level. If we were to have 5 retries, it might take a very long time for a genuine error to propagate to the user.

[arogozhnikov]

thanks, makes sense.

I believe the network stack itself does retries at a lower level

That's correct, aiohttp does retry internally - I just did not find a way to control number of retries/timeouts in a call to s3fs. Would be great to have this knob, as I need to deal with shaky network.

[martindurant]

Sorry, I don't know how to pipe connection-level arguments down the stack in aiobotocore. Perhaps ask on their tracker?