[BUG] - Strange behaviour on long running program (1769-L18ER-BB1B)

[arkAD93]

Pre-checks

• running latest version
• enabled logging
• consulted the docs

Description
We have a long lived application which periodically writes and reads tags, and sometimes communicates on "demand" (when changing params on GUI app). Everything works without issues, but after some time (40mins - couple of days) we suddenly see a lot of errors and "on demand" communication no longer works.

On "demand" which was mentioned, works such that we connect to PLC, write tags, and close connection (all done using context manager). Periodic writes and reads are done in separate threads, having their own connections which are constantly utilized - no issues there.

When we encounter errors, restarting application solves problem, everything works for some time and then eventually issues come back.

Target PLC
Model: 1769-L18ER-BB1B
Firmware Revision: 32.015

Logs

  # first example of error
[ ControllerProcess-Thread-2 (_syste] 09-30 08:41:51 ERROR: plc.py::from_plc()::212: Failed to get PLC info
Traceback (most recent call last):
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 328, in get_plc_info
    raise ResponseError(f"get_plc_info did not return valid data - {response.error}")
pycomm3.exceptions.ResponseError: get_plc_info did not return valid data - Failed to parse reply - Error packing -50 as USINT

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\******\backend\src\comm\plc.py", line 205, in from_plc
    with self._plc as comm:
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
    self.open()
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
    self._initialize_driver(**self._init_args)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 174, in _initialize_driver
    self._info = self.get_plc_info()
                 ^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 336, in get_plc_info
    raise ResponseError("Failed to get PLC info") from err
pycomm3.exceptions.ResponseError: Failed to get PLC info
[ ControllerProcess-MainThread      ] 09-30 08:41:51 ERROR: plc.py::to_plc()::176: failed to get attribute list
Traceback (most recent call last):
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 499, in _get_instance_attribute_list_service
    raise ResponseError(
pycomm3.exceptions.ResponseError: send_unit_data returned not valid data - Failed to parse reply - Error packing -116 as USINT

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\******\backend\src\comm\plc.py", line 168, in to_plc
    with self._plc as comm:
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
    self.open()
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
    self._initialize_driver(**self._init_args)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 192, in _initialize_driver
    self.get_tag_list(program="*" if init_program_tags else None)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 81, in wrapped
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 422, in get_tag_list
    tags = self._get_tag_list()
           ^^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 438, in _get_tag_list
    all_tags = self._get_instance_attribute_list_service(program)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 511, in _get_instance_attribute_list_service
    raise ResponseError("failed to get attribute list") from err
pycomm3.exceptions.ResponseError: failed to get attribute list
[ ControllerProcess-MainThread      ] 09-30 08:41:51 ERROR: plc.py::to_plc()::176: Failed to get PLC info
Traceback (most recent call last):
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 328, in get_plc_info
    raise ResponseError(f"get_plc_info did not return valid data - {response.error}")
pycomm3.exceptions.ResponseError: get_plc_info did not return valid data - Failed to parse reply - Error packing -128 as USINT

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\******\backend\src\comm\plc.py", line 168, in to_plc
    with self._plc as comm:
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
    self.open()
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
    self._initialize_driver(**self._init_args)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 174, in _initialize_driver
    self._info = self.get_plc_info()
                 ^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 336, in get_plc_info
    raise ResponseError("Failed to get PLC info") from err
pycomm3.exceptions.ResponseError: Failed to get PLC info

Those errors keeps going forever, when finally we receive another type of error

  # second example of error
[ ControllerProcess-Thread-2 (_syste] 09-30 09:00:00 ERROR: plc.py::from_plc()::212: failed to send message
Traceback (most recent call last):
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\socket_.py", line 53, in send
    sent = self.sock.send(msg[total_sent:])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 587, in _send
    self._sock.send(message)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\socket_.py", line 58, in send
    raise CommError("socket connection broken.") from err
pycomm3.exceptions.CommError: socket connection broken.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\******\backend\src\comm\plc.py", line 205, in from_plc
    with self._plc as comm:
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
    self.open()
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
    self._initialize_driver(**self._init_args)
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 171, in _initialize_driver
    target_identity = self._list_identity()
                      ^^^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 264, in _list_identity
    response = self.send(request)
               ^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 1387, in send
    return super().send(request)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 574, in send
    self._send(request.build_request(**request_kwargs))
  File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 589, in _send
    raise CommError("failed to send message") from err
pycomm3.exceptions.CommError: failed to send message

Unfortunately we tried different techniques (ex reseeding message) but eventually "on demand" communication is blocked entirely and only restart helps