Intermittent ConnectionResetError in script runner script that runs in the background

[0lionelzhang0]

We have a script that runs in a while loop that sends 10 commands spaced 0.2 seconds apart then sleeps for 20 seconds (this particular script is using time.sleep() but we're aware we can use wait()). Every once in awhile (sometimes 10-20 mins), we get the error below (command params omitted). For now, we've added try/except to avoid having to manually retry. But it would be nice to know what might cause this.

The openc3-cosmos-cmd-tlm-api container has been up for the past 4 days. This particular log also does not seem to be present in the logs of openc3-cosmos-cmd-tlm-api container. In the couple times this occurred, it seemed to trip on the first command after having slept for the 20 seconds.

openc3.io.json_drb_object.JsonDRbError: No response from server: Request: {'url': 'http://openc3-cosmos-cmd-tlm-api:2901/openc3-api/api', 'data': '{"jsonrpc": "2.0", "method": "cmd", "params": ["<OUR_COMMAND_PARAMS>"], "keyword_params": {"timeout": null, "log_message": null, "validate": true, "scope": "DEFAULT"}, "id": 0}', 'headers': {'User-Agent': 'OpenC3 / v5 (ruby/openc3/lib/io/json_drb_object)', 'Content-Type': 'application/json-rpc', 'Authorization': 'openc3service'}} ::: None ::: Exception: ChunkedEncodingError(ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')))

[ryanmelt]

The python implementation uses Session which uses a long lived TCP connection. The error you are getting could happen if the Puma server decided to respawn a worker process for some reason. I think we should add a single automatic retry if this occurs.