[exporter/otlp] Report runtime status

[mwear]

Description

This PR adds runtime status reporting for the otlp and otlphttp exporters. It's an updated version of #8788 which was one of several approaches experimented with to add this functionality. This work was paused to allow the consumererror work to evolve, as it appeared as though there might be a way to uniformly apply the same logic to all exporters (via the exporterhelper), but that work has taken a different direction, and it looks like a uniform approach will not be possible.

This PR implements runtime status reporting as discussed in #9957. The choices for which statuses represent permanent errors are up for debate, but the key point is that a permanent error is an error discovered at runtime that will require user intervention to fix.

This implementation makes use of the finite state machine that underlies the status reporting system. The finite state machine will ensure that:

• Only changes in status are reported. Repeat reports of the same status will no-op.
• If a component transitions into a PermanentError all further status reports will no-op.

This means the exporter does not have to reason about current or previous statuses. It can report status based on its current view of the world and the status reporting system will handle the rest. Flapping between recoverable and ok is meant to be handled by watchers consuming status events. The healthcheckv2extension handles this by using a time based approach (e.g. recovery interval). Other watchers may choose to handle this situation differently.

For more information on status reporting, the state machine, etc see: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md

While all components report status during start and shutdown (via automation), this is the first component to report runtime status. This will allow the healthcheckv2extension to replace the currently non-functioning check_collector_pipeline capability of the original healthcheckextension and should serve as an example for other components that wish to report runtime status moving forward.

Link to tracking issue

Implements #9957 for the otlp and otlphttp exporters

Testing

units/manual

Documentation

code comments

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.50%. Comparing base (866c7fc) to head (1a6e939).
Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #11366      +/-   ##
==========================================
+ Coverage   91.47%   91.50%   +0.03%     
==========================================
  Files         435      435              
  Lines       23748    23815      +67     
==========================================
+ Hits        21723    21793      +70     
+ Misses       1648     1646       -2     
+ Partials      377      376       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bogdandrutu]

Before implementation, can we document somewhere (ideal in RFC) or in https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md when an exporter is consider permanent error? I feel like we have different understandings of this and I don't want you to make lots of code changes until we agree on the basic definition.

[mwear]

Before implementation, can we document somewhere (ideal in RFC) or in https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-status.md when an exporter is consider permanent error?

I have these documented here: #9957. Is it ok to discuss on that issue, or would you prefer it to be an RFC?

[mwear]

If you're suggesting that the errors identified as Permanent in this PR (and #9957) are in many cases recoverable, we can identify all errors as recoverable and skip permanent for now.

[bogdandrutu]

I am not the biggest expert, but I feel at least some are recoverable, and some are contextual (example auth when passing auth tokens from the client).

If you're suggesting that the errors identified as Permanent in this PR (and #9957) are in many cases recoverable, we can identify all errors as recoverable and skip permanent for now.

I think would be safer correct?

[mwear]

I think going with recoverable for all errors, at least for now, makes sense. We can discuss what errors should be permanent on #9957, but I suspect we'll be able to come up with counter arguments for most of the cases.

[mwear]

I updated this PR to handle all errors as recoverable. If we are happy with this approach, we could consider making this an option for all exporters via exporter helper similar to: #8684 likely as a followup.

[bogdandrutu]

Are we not concerned for every request to report the status? Should we do it periodically, or only when changes?

[mwear]

The finite state machine handles this for us. It will report status only if there has been a change, otherwise it will no-op.