python: IQM consumes 100% CPU when telemetry server is unreachable

[ymorin-orange]

What:

When the telemetry server is unreachable (e.g. is down, or has network issues), the IQM spins forever in the telemetry client, using 100% CPU.

When the telemetry server becomes reachable again (e.g. got restarted), the IQM eventually settles down.

[ymorin-orange]

@mathieu1fb FYI

[ymorin-orange]

Initial analysis:

• the issue happens in the IoT3 SDK, in the OTLP client
• when an MQTT message is received or emitted by the MQTT client in the SDK, an OTLP trace span is queued in a Python queue.SimpleQueue()
• the queue.SimpleQueue() is unbounded, so it can queue an arbitrarily large number of messages
• when the IQM receives more messages than it can send OTLP traces for, that queue grows
• if the OTLP collector is unreachable, the backlog of traces to send is always full, so the OTLP client attempts to send the backlog; as it fails, it keeps the pending traces
• when the stream of incoming messages eventualy subsides, the queue is slowly emptied one message at a time, and for each traces attempts to send it to the OTLP collector
• when the queue is eventually empty, the OTLP client periodically attempts to send the backlog of traces