Pushing traces to grafana cloud using the OTLP endpoint.

[Odinaka-git]

Hi, I'm currently trying to build an application that will push traces, metrics and logs directly to the grafana cloud. I've built my application but I'm having some issues. Please if you could look through my code and tell me what might be the issue that would be nice.

OTEL_EXPORTER_OTLP_ENDPOINT=OTLP Endpoint URL
OTEL_EXPORTER_OTLP_HEADERS=Basic base64 instanceID:token

code:

#pragma warning(disable : 4996)
#include "otel.h"
#include "network.h"

namespace trace = opentelemetry::trace;
namespace trace_sdk = opentelemetry::sdk::trace;
namespace otlp = opentelemetry::exporter::otlp;

int main(int argc, char* argv[])
{
	std::string mac;

	if (GetMacAddress(mac) < 0) return -1;

	std::string ip;

	if (GetIpAddress(ip) < 0) return -1;

	//Create instance to store options to initialize custome trace with.
	opentelemetry::exporter::otlp::OtlpHttpExporterOptions opts;

	//provide the "OTEL_EXPORTER_OTLP_ENDPOINT" here.
	opts.url = std::string(std::getenv("OTEL_EXPORTER_OTLP_ENDPOINT"));

	//provide the "OTEL_EXPORTER_OTLP_HEADERS" here.
	opts.http_headers.insert(std::make_pair<const std::string, std::string>("Authorization=",std::getenv("OTEL_EXPORTER_OTLP_HEADERS")));

	// Create OTLP exporter instance
	auto exporter = otlp::OtlpHttpExporterFactory::Create(opts);
	auto processor = trace_sdk::SimpleSpanProcessorFactory::Create(std::move(exporter));
	std::shared_ptr<opentelemetry::trace::TracerProvider> provider =
		trace_sdk::TracerProviderFactory::Create(std::move(processor));

	// Set the global trace provider
	trace::Provider::SetTracerProvider(provider);

	auto tracer = provider->GetTracer("Test-app", "v1.0");

	opentelemetry::trace::StartSpanOptions options;

	options.kind = opentelemetry::trace::SpanKind::kServer;

	auto span = tracer->StartSpan("Call to /myendpoint", {
	{ "http.method", "GET" },
	{ "net.protocol.version", "1.1" },
	{ "net.peer.mac", mac },
	{ "net.peer.ip", ip}
		}, options);

	auto scope = tracer->WithActiveSpan(span);

	span->End();

	// We call ForceFlush to prevent to cancel running exportings, It's optional.
	if (provider)
	{
		static_cast<trace_sdk::TracerProvider*>(provider.get())->ForceFlush();
	}

	std::shared_ptr<opentelemetry::trace::TracerProvider> none;
	trace::Provider::SetTracerProvider(none);

	return 0;

}

logs:

,Body:400 Bad Request text/plain; charset=utf-8otlp\src\otlp_http_client.cc:116 [OTLP HTTP Client] Export failed, Status:400, Header:   Connection: close
[Error] File: C:\.conan\14d192\1\src\exporters\otlp\src\otlp_http_exporter.cc:123 [OTLP HTTP Client] ERROR: Export 1 trace span(s) error: 1

[marcalff]

To add a key/value pair in the http headers:

  options.http_headers.insert(
      std::make_pair<const std::string, std::string>("Custom-Header-Key", "Custom-Header-Value"));

Now, how many key/values, which key names, and which values are expected by the endpoint depends on the vendor for this endpoint.

Now sure if this is the proper doc for graphana, but found this:

https://grafana.com/docs/grafana/latest/administration/provisioning/#custom-http-headers-for-data-sources

https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/

Do not add the "=" sign in the key name.

Pass a valid "Bearer XXXXXXXXX" (whatever that is) as value.

At the end of the day, if the endpoint refuses the connection, there is not much the opentelemetry-cpp client code can do about it, and there is not much error message that can be displayed beside connection failed.

This part is vendor dependent, check the vendor documentation.

[Odinaka-git]

Can you explain what you mean by a valid "Bearer XXXXXXXXX"

[Odinaka-git]

sorry, I just realized you said you don't know either, I'll just check the docks

[marcalff]

See the graphana oltp doc, it is a base64 encoding of id+token.

$(echo -n $GRAFANA_INSTANCE_ID:$TOKEN | base64 -w 0)

[marcalff]

Also, the following code:

	opts.http_headers.insert(std::make_pair<const std::string, std::string>("Authorization=",std::getenv("OTEL_EXPORTER_OTLP_HEADERS")));

is incorrect.

Either individual key and values are available, in which case opts.http_headers.insert() should be used, or the whole properly formatted ${OTEL_EXPORTER_OTLP_HEADERS} string is available in the env variable, in which case there is nothing to do: opentelemetry-cpp will read the OTEL_EXPORTER_OTLP_HEADERS variable if present, and populate opts.http_headers.

Likewise, this is not needed:

	opts.url = std::string(std::getenv("OTEL_EXPORTER_OTLP_ENDPOINT"));