X-Grafana-* HTTP headers are not sent with requests

[micoq]

What happened: The X-Grafana-* and X-Forwarded-For HTTP headers are not sent with requests when we use the datasource normally in explore mode or in a panel. So we are unable to use a proxy-based authentication on OpenSearch by passing the username in X-Grafana-User header from Grafana to OpenSearch.
However, the headers are sent when we create/modify the datasource.

No options works (including Forward OAuth Identity or With Credential)

What you expected to happen: HTTP headers are sent with any request made to OpenSearch.

How to reproduce it (as minimally and precisely as possible): Just set up a standard proxy-based authentication on an OpenSearch node (https://docs.opensearch.org/docs/2.19/security/authentication-backends/proxy/)

config:
  dynamic:
    http:
      anonymous_auth_enabled: false
      xff:
        enabled: true
        internalProxies: '127\.0\.0\.1|0:0:0:0:0:0:0:1'
    authc:
      proxy_auth_domain:
        order: 0
        http_enabled: true
        transport_enabled: true
        http_authenticator:
          type: proxy
          challenge: false
          config:
            user_header: "X-Grafana-User"
        authentication_backend:
          type: noop

Request dumps:

A request made in explorer mode:

POST /_msearch?max_concurrent_shard_requests=5 HTTP/1.1
Host: my-host:9200
User-Agent: Grafana
Content-Length: 1966
Content-Type: application/x-ndjson
Accept-Encoding: gzip

{"ignore_unavailable":true,"index":"logs-aaaaa-*","search_type":"query_then_fetch"}
{"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"5m","min_doc_count":0,"extended_bounds":{"min":1745276995820,"max":1745320195820},"format":"epoch_millis"}}},"query":{"bool":{"filter":[{"range":{"@timestamp":{"format":"epoch_millis","gte":1745276995820,"lte":1745320195820}}},{"query_string":{"analyze_wildcard":true,"query":"test"}}]}},"size":0}
[...]

The request made by saving the datasource:

GET /logs-aaaaa-%2A/_mapping HTTP/1.1
Host: mhost:9200
User-Agent: Go-http-client/1.1
Accept: application/json, text/plain, */*
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: fr-FR,fr;q=0.8,en-US;q=0.5,en;q=0.3
Dnt: 1
Priority: u=4
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
X-Datasource-Uid: fe9b739d-9446-406b-9f09-2cf2a9e30934
X-Forwarded-For: 10.0.0.1
X-Grafana-Id: [...]
X-Grafana-Nocache: true
X-Grafana-Org-Id: 8
X-Grafana-Referer: https://myhost/grafana/connections/datasources/edit/fe9b739d-9446-406b-9f09-2cf2a9e30934
X-Grafana-User: myuser
X-Plugin-Id: grafana-opensearch-datasource

Environment:

• Grafana version: OSS v11.6.0
• OpenSearch version: 2.19.0
• Plugin version: 2.24.0

[iwysiu]

Hi @micoq! We'll take a look at this. Do you know if this used to work, and if so which version?

[micoq]

Hi @micoq! We'll take a look at this. Do you know if this used to work, and if so which version?

Hi! Yes, before that, we were using version 2.11.0 of the datasource without any problems (on Grafana 11.5.x and below). However, we haven't tried the intermediate versions.