Contributing otel-collector as an Exporter to OpenTelemetry Collector Contrib

[haihh05]

I hope you're doing well. I'm curious why it hasn't been contributed as an exporter to the OpenTelemetry Collector Contrib project (https://github.com/open-telemetry/opentelemetry-collector-contrib).

Contributing your otel-collector to OpenTelemetry Collector Contrib could offer several benefits, including increased visibility, community-driven enhancements, and wider adoption. I would appreciate any insights you can provide on the decision to maintain the otel-collector separately and whether there are specific considerations involved.

Thank you for your time and any information you can share on this matter.

[ocraviotto]

I'd like to echo this request, as having the exporter being available as part of the OpenTelemetry Collector would make users' lives easier when already using the collector.

[lmangani]

Thanks for bumping this! We're planning around this problem, but first we need to explore any opportunity to extend the existing clickhouse exporter to maximize the potential also in terms of future maintenance. If that's not possible, we'll most definitely go ahead and join the contrib circle with our exporter.

Note: With the loki receiver working, the contrib represents a full replacement for the qryn-js ingestion pipeline.

[lauren-hinge]

Any updates here? Given that it doesn't seem like you can configure clickhouseexporter to work with qryn's schema, or qryn to work with clickhouseexporter's schema, the fact that this exporter is on a separate collector feels like a pretty big gap in qryn's featureset.

I'm already using collectors with other exporters, so I think I'd need to forward from there to an intermediary layer of additional qryn collectors, just to write data the way qryn expects.

---

Actually, turns out you can just send your qryn deployment raw OTLP (with otlphttp), bypassing the clickhouseexporter entirely!

Most of the docs pushed towards this collector repo; I didn't realize this was an option! So, leaving this here in case anyone else goes down the same path.

---

Edit 2: I'm not sure this collector is using async_insert when writing to Clickhouse, which means it really can't work at large scale. Qryn-js as ingest works ok, but it really seems like there's a lot of overhead at the JS layer, and writing http batches rather than TCP. Response times writing to qryn are kinda tough. Looking forward to testing out gigapipe!