Long-lived connection breaks on `requests.exceptions.ChunkedEncodingError`

[sachalevy]

Running a long-lived connection with hulse host breaks after a while due to a requests.exceptions.ChunkedEncodingError. Need to debug on which side does the error come from, and potentially restart the connection automatically on failure.

Trace:

Traceback (most recent call last):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 700, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 441, in _error_catcher
    yield
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 767, in read_chunked
    self._update_chunk_length()
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 704, in _update_chunk_length
    raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/requests/models.py", line 760, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 575, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 796, in read_chunked
    self._original_response.close()
  File "/Users/sachalevy/miniconda3/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/urllib3/response.py", line 458, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sachalevy/IMPLEMENT/hulse-py/hulse/utils.py", line 237, in run_host
    handle_producer_stream(r, api_key)
  File "/Users/sachalevy/IMPLEMENT/hulse-py/hulse/utils.py", line 62, in handle_producer_stream
    for line in response.iter_lines():
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/requests/models.py", line 804, in iter_lines
    for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/requests/models.py", line 763, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sachalevy/IMPLEMENT/releasenv/bin/hulse", line 11, in <module>
    load_entry_point('hulse', 'console_scripts', 'hulse')()
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/sachalevy/IMPLEMENT/releasenv/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/sachalevy/IMPLEMENT/hulse-py/hulse/cli.py", line 24, in host
    utils.run_host(api_key=settings.CONFIG.get("api_key"))
  File "/Users/sachalevy/IMPLEMENT/hulse-py/hulse/utils.py", line 239, in run_host
    raise errors.HulseError(expression=e)
hulse.errors.HulseError