Expose setting CURLOPT_LOW_SPEED_TIME

[cailinw]

Hello, I noticed that CURLOPT_LOW_SPEED_TIME gets set to 20 seconds, so if there is no response from the server in 20 seconds, the connection will time out. However, I have a use case where I am requesting content from an AWS API Gateway, and I would like to allow at least 30 seconds. Is it possible to set this value to 30 seconds? Thank you.

I am currently running into the following error:

Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds

[cailinw]

Found a way to do this, by passing in a custom Downloader! Thank you

downloader = Downloads.downloader
downloader.easy_hook = (easy, info) ->
       Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_LOW_SPEED_TIME, 30)
request(..., downloader=downloader)

[StefanKarpinski]

We could also bump it up to 30 seconds. Other people have hit this recently and maybe 20s is too low.

[einarpersson]

@cailinw thank you for posting your workaround. Your code is not runnable as-is so I'll just add this to help any others get this working:

downloader = Downloads.Downloader()
downloader.easy_hook = (easy, info) -> Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_LOW_SPEED_TIME, 60)

Downloads.download(url, "foo.json"; downloader=downloader)

@StefanKarpinski
As a newcomer it's easy to think that this would be solved by increasing the timeout parameter, but this does not seem to affect this particular problem. I don't know what is the right way of amending this.

1. Add a note in the documentation in context of timeout?
2. Make it possible to pass in this option when using the download or somehow connect it to the timeout parameter if that doesn't cause other problems..

Bump the default value sounds like a good idea. Note however that 30s did not make a difference in my case, I increased it to 60s.

Consider reopening the issue?

[cailinw]

Thanks!

The only reason 30s worked in my case was because the API I was working always took less than (but close to) 30s to respond.

I agree that the timeout parameter can be confusing, especially since the CURLOPT_LOW_SPEED_TIME is set to 20s, the timeout parameter is only useful if you want to timeout in less than 20s, which isn't helpful for slow requests/endpoints.

[StefanKarpinski]

The timeout parameter is a time limit on the entire operation: if the download hasn't completed entirely by the time the timeout has elapsed, the download is canceled. This can be useful if you know a request shouldn't take very long, e.g. it's a small file, but it's not a good idea to set generally since a download can take an arbitrarily long time for good reason if the file is large. The low speed time controls how long the download will wait if the server doesn't send any data at all. No matter how large a download is, there shouldn't be too much time where the server isn't sending any data at all. We don't currently expose an option for this because it seems unusual for a server to send no data for as long as 20 seconds, but apparently it happens sometimes. Regarding:

I agree that the timeout parameter can be confusing, especially since the CURLOPT_LOW_SPEED_TIME is set to 20s, the timeout parameter is only useful if you want to timeout in less than 20s, which isn't helpful for slow requests/endpoints.

How so? You could set a timeout of 5 minutes to guarantee that a download finishes within five minutes.

I don't see a great way to connect the low speed timeout to the overall timeout—they're just different things. We could do something like if the timeout is set, then don't set a low speed time limit since the download can't take arbitrarily long waiting for a non-responsive server because of the timeout. But it's a little weird that the way to increase how long a download will allow a server to send no data for is to set a timeout on the download, which is otherwise unbounded. It's probably ok to make the low speed time higher, something like 60 seconds.

[dmoliveira]

@StefanKarpinski It has been observed that in certain cases, some applications may encounter slow response times when accessing private or government websites, possibly due to inadequate hardware and bandwidth on the server side. This is especially relevant in countries with lower levels of infrastructure development. In one instance, it was noted that a server intentionally responded slowly when a request for public data was made. To address this issue, it would be beneficial to consider adding an option to allow for the continuation of a connection even if the response is slow. This would help to ensure that users in these types of situations are able to fully utilize the application.

[LilithHafner]

At the least, this should be better documented.

[gitboy16]

Hi, is there an plan to fix that problem which has been outstanding for more than 3 years please? Being able to set CURLOPT_LOW_SPEED_TIME before julia starts would be great. Thank you.

[gitboy16]

Just adding a bit of context on where I am having this issue. I have this issue on a fresh julia installation using juliaup, when I enter the package manager for the first time and type "st" I get the following error:
Exception RequestError: HTTP/1.0 200 OK (Operation too slow. Less than 1 bytes/sec transferred the last 20 seconds) ....

[LilithHafner]

@gitboy16, that's very interesting. When I turn off internet and run ]st I get no error. It seems that Pkg.status tries and fails to provide results with or without internet. If you can get a reproducer, please report that issue to Pkg.