When Following Redirects on_headers Only Emits On First Response

[Kyrluckechuck]

Hello!

I don't know if we'll see this issue fixed, but after a ton of debugging and headaches, I've finally tracked down what I think is the cause of all of my redirect issues in the typhoeus release 1.4.0 (from version 1.3.1). That also forced us to move from ethon version 0.12.0 to 0.16.0.

After looking at the ethon code diff, I had come to the same conclusion that other users in the typhoeus had, that the changes here had some unintended side effects. Please see typhoeus/typhoeus#705 for more details.

Filing this issue here since this appears to be the package in question that is having undesired behaviour, which was introduced with #221.

Apologies if this is just being mis-used in typhoeus and this package is not at fault, but that repo has not seen any maintenance in a while (and those tickets haven't drawn any attention) so wanted to file this here in the hopes we may either get a proper workaround (aside from just ignoring the on_headers call since now it's not entirely useful), or even better, an update to fix that.

The current workaround I'm using is not ideal because it leads to the download flow being executed just to do basic content-length header checks, but it's manageable. This is my current on_headers method in order to work around this bug and still do the content-length check for most scenarios (when not involving a redirect):

req.on_headers do |response|
  if (response.code != 301 && response.code != 302)
      HttpHelper.assert_status_code!(response.code)
      raise SomePackage::DownloadFailure, 'Invalid Download File Type' if response.headers['Content-Length'].to_i.zero?
  end
end

For extra context though, this is not the case for on_complete, which will only emit after the final destination or max redirects are reached, which is the behaviour we'd expect.