lots of gRPC error output in diag logs

[shawaj]

diag logs are getting spammed with a lot of this type of stuff on startup:

INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
INFO:root:Running periodic hardware diagnostics
2023-04-14 03:35:56,160 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(42) - rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
ERROR:hm_pyhelper.miner_param:rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
2023-04-14 03:35:56,161 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(43) - <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 87, in get_pubkey
    encoded_key = self.stub.pubkey(local_pb2.pubkey_req()).address
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
ERROR:hm_pyhelper.miner_param:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 87, in get_pubkey
    encoded_key = self.stub.pubkey(local_pb2.pubkey_req()).address
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
2023-04-14 03:35:56,178 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(42) - rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
ERROR:hm_pyhelper.miner_param:rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
2023-04-14 03:35:56,180 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(43) - <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 68, in get_region
    region_id = self.get_region_enum()
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 61, in get_region_enum
    return self.stub.region(local_pb2.region_req()).region
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
ERROR:hm_pyhelper.miner_param:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 68, in get_region
    region_id = self.get_region_enum()
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 61, in get_region_enum
    return self.stub.region(local_pb2.region_req()).region
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
2023-04-14 03:35:56,312 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(38) - gateway_mfr response stdout: b'{\n  "key": "112dvU8U5q7acovjtvnMBynBT6rQZ5RUqiy8BoK95yPuXUhT3tbn",\n  "name": "wobbly-pickle-cheetah"\n}\n'

@MuratUrsavas can we catch these and show a better message rather than spamming the logs?

[shawaj]

also seen this one:

2023-04-14 03:48:02,468 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(103) - Found the following BLE Devices: [{'Address': 'DC:A6:32:6A:36:C0', 'Name': 'nebra-6A36BE.local', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
INFO:hw_diag.utilities.hardware:Found the following BLE Devices: [{'Address': 'DC:A6:32:6A:36:C0', 'Name': 'nebra-6A36BE.local', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
2023-04-14 03:48:02,469 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(149) - Retrieving list of LTE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of LTE device(s)
2023-04-14 03:48:02,475 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(176) - Found the following LTE Devices: []
INFO:hw_diag.utilities.hardware:Found the following LTE Devices: []
INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
INFO:root:Running periodic hardware diagnostics
Exception in thread Thread-164 (_run):
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "src/python/grpcio/grpc/_cython/_cygrpc/thread.pyx.pxi", line 53, in grpc._cython.cygrpc._run_with_context._run
  File "/opt/python-dependencies/grpc/_channel.py", line 1547, in _poll_connectivity
    event = channel.watch_connectivity_state(connectivity,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 510, in grpc._cython.cygrpc.Channel.watch_connectivity_state
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 377, in grpc._cython.cygrpc._watch_connectivity_state
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 385, in grpc._cython.cygrpc._watch_connectivity_state
ValueError: Cannot monitor channel state: Channel closed!
2023-04-14 03:49:02,881 - [INFO] - hm_pyhelper.miner_param 

[shawaj]

https://nebraltd.atlassian.net/browse/NEBD-110