bias documentation to preferring `ApiKey <API_KEY>` for auth, rather than `Bearer <SECRET_TOKEN>`

[trentm]

(Per @theletterf's request to open an issue to track this.)

There was some internal discussion recently that highlighted that it can be confusing to users what auth mechanism is appropriate to use for the Authorization header (e.g. when configuring the OTEL_EXPORTER_OTLP_HEADERS envvar for opentelemetry SDKs). There are two mechanisms:

1. Bearer token auth, using a "secret token". This requires ingest API requests (whether that is the APM server intake APIs or the various OTLP endpoints) to include the Authorization: Bearer $SECRET_TOKEN header.
2. API key auth, for which the Authorization: ApiKey $API_KEY header is needed.

Current support in various products/services:

• APM server's intake v2 API supports both mechanisms -- which means Elastic Cloud Hosted (ECH) and self-hosted support both mechanisms.
• APM server's OTLP endpoint supports both mechanisms. (I'm pretty sure of this, but not 100% confident.)
• EDOT Collector (handled by the https://github.com/elastic/opentelemetry-collector-components/tree/main/extension/apikeyauthextension/ collector extension) supports API key auth. (AFAIK, "Bearer" secret token auth is not supported.)
• Managed OTLP (mOTLP) supports only API key auth. (IIUC, mOTLP covers both the OTLP ingest endpoint for a serverless project and the OTLP endpoint for ECH. Again, I am not 100% confident on this part.)

Historically, Bearer token auth (using a secret token) was the initial auth mechanism. API key auth support came later and a clear migration was never heavily pushed or forced on customers.

Pros/Cons:

• API key auth is supported in all products/services.
• A "secret token" is created automatically when creating an ECH or self-hosted deployment, so there is one less step in a "getting started" guide when using Bearer token auth. A user must create an API key after the deployment is created for API key auth.
• API keys are revocable, support finer-grained permissions; you can have multiple API keys.

Note that this isn't just about docs. There are built-in getting-started flows in Kibana that default to one auth mechanisms or the other. So if docs are documenting what Kibana does, then a change must come in the Kibana flow first.

[theletterf]

@gbamparop Should we create an issue in the Kibana repo, too?

[gbamparop]

@gbamparop Should we create an issue in the Kibana repo, too?

What's the recommendation here? Once we have a change request we can create another issue in Kibana

[theletterf]

@gbamparop If I understood @trentm correctly, it would mean replacing bearer token auth examples in UI and docs with ApiKey examples. AFAIK we're that already in the OpenTelemetry Add Data screen, when you create an API key.

Do you have Kibana examples where bearer token is used, @trentm ?

[trentm]

Do you have Kibana examples where bearer token is used?

@theletterf If I walk through the add-data onboarding flow for Observability ($kibanaUrl/app/observabilityOnboarding) in a Kibana 9.1 deployment in ECH, select Application for "What do you want to monitor?", select OpenTelemetry for "Monitor your Application using", then the boilerplate configurations offered for each technology (Node.js, Django, ..., OpenTelemetry) on the next page (.../app/home#/tutorial/apm) all use secretToken.

[theletterf]

@trentm Thank you. Could you create that change request in the Kibana repo with a concrete recommendation? We'll be happy to follow it for the docs, too.

[mykolaharmash]

@theletterf If I walk through the add-data onboarding flow for Observability ($kibanaUrl/app/observabilityOnboarding) in a Kibana 9.1 deployment in ECH, select Application for "What do you want to monitor?", select OpenTelemetry for "Monitor your Application using", then the boilerplate configurations offered for each technology (Node.js, Django, ..., OpenTelemetry) on the next page (.../app/home#/tutorial/apm) all use secretToken.

One weird thing, the snippets for different technologies indeed show secret_token, but when copied using the copy button the snippet then uses api_key everywhere instead:) Not sure how that happened, but it definitely should show the same content as we put into user's clipboard and seems like it should be API key everywhere.

[trentm]

elastic/kibana#233040 to hopefully change the APM onboarding flow.

[mlunadia]

@vigneshshanmugam @axw what is your recommendation on this one?

[axw]

I would recommend we prefer API Key auth, since we do not plan to support "secret token" auth in MOTel as it exists in APM Server. We might later introduce a different kind of Bearer token auth, but I think we can revisit it then.