How to minimize the configuration of API Key permissions when using Elasticsearch's API Key authentication

[MicroOps-cn]

When configuring Jaeger to use Elasticsearch's APIKey authentication, if I set the permission of the APIKey to only allow read and write permissions for the specified index, it will cause Jaeger to fail to start. Will Jaeger call other ES interfaces besides reading and writing ES?

permissions:

{
  "write-only-role": {
    "cluster": [],
    "indices": [
      {
        "names": ["jaeger-primary-*"],
        "privileges": ["read","write"]
      }
    ]
  }
}

log:

2025-08-26T09:44:11.141Z        info    [email protected]/service.go:276 Starting jaeger...      {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "Version": "v2.9.0", "NumCPU": 4}
2025-08-26T09:44:11.141Z        info    extensions/extensions.go:41     Starting extensions...  {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}}
2025-08-26T09:44:11.141Z        info    extensions/extensions.go:45     Extension is starting...        {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "healthcheckv2", "otelcol.component.kind": "extension"}
2025-08-26T09:44:11.141Z        debug   [email protected]/extension.go:98 Starting health check extension V2      {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "healthcheckv2", "otelcol.component.kind": "extension", "config": {"Endpoint":"localhost:13133","TLS":{},"CORS":{},"Auth":{},"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":null,"UseV2":true,"GRPCConfig":null,"HTTPConfig":{"Endpoint":"0.0.0.0:13133","TLS":{},"CORS":{},"Auth":{},"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"CompressionAlgorithms":null,"ReadTimeout":0,"ReadHeaderTimeout":0,"WriteTimeout":0,"IdleTimeout":0,"Middlewares":null,"Config":{"Enabled":false,"Path":"/config"},"Status":{"Enabled":true,"Path":"/status"}},"ComponentHealthConfig":null}}
2025-08-26T09:44:11.141Z        info    extensions/extensions.go:62     Extension started.      {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "healthcheckv2", "otelcol.component.kind": "extension"}
2025-08-26T09:44:11.141Z        info    extensions/extensions.go:45     Extension is starting...        {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "jaeger_storage", "otelcol.component.kind": "extension"}
2025-08-26T09:44:11.141Z        info    jaegerstorage/extension.go:159  Initializing storage 'primary_store'    {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "jaeger_storage", "otelcol.component.kind": "extension"}
2025-08-26T09:44:16.173Z        error   extensions/extensions.go:58     Failed to start extension       {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "jaeger_storage", "otelcol.component.kind": "extension", "error": "failed to initialize storage 'primary_store': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available"}
2025-08-26T09:44:16.173Z        info    [email protected]/service.go:341 Starting shutdown...    {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}}
2025-08-26T09:44:16.173Z        info    [email protected]/connector.go:227  Shutting down spanmetrics connector     {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "spanmetrics", "otelcol.component.kind": "connector", "otelcol.signal": "traces", "otelcol.signal.output": "metrics"}
2025-08-26T09:44:16.173Z        info    extensions/extensions.go:69     Stopping extensions...  {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}}
2025-08-26T09:44:16.173Z        info    [email protected]/extension.go:138        discarding event received after shutdown        {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}, "otelcol.component.id": "healthcheckv2", "otelcol.component.kind": "extension", "source": {}, "event": {}}
2025-08-26T09:44:16.173Z        info    [email protected]/service.go:355 Shutdown complete.      {"resource": {"service.instance.id": "cd27a5b2-e605-4f0d-9985-23cb11574882", "service.name": "jaeger", "service.version": "v2.9.0"}}
Error: failed to start extensions: failed to initialize storage 'primary_store': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available
2025/08/26 09:44:16 failed to start extensions: failed to initialize storage 'primary_store': failed to create Elasticsearch client: health check timeout: no Elasticsearch node available

[dosubot[bot]]

Hi @MicroOps-cn! I'm Dosu and I’m helping the jaeger team.

Jaeger requires more than just index-level read/write permissions when connecting to Elasticsearch. During startup, it performs cluster-level operations like health checks and version detection, which require permissions such as cluster:monitor/health and cluster:monitor/state in addition to your index privileges. Without these, you'll see errors like the health check timeout you encountered.

To minimize permissions, you can disable health checks and sniffing in Jaeger's Elasticsearch storage config (disable_health_check: true and sniffing.enabled: false), but some cluster-level calls (like version detection via ping) are still required and cannot be fully disabled. So, your API Key role should look something like:

{
  "minimal-jaeger-role": {
    "cluster": ["monitor", "monitor/health", "monitor/state"],
    "indices": [
      {
        "names": ["jaeger-primary-*"],
        "privileges": ["read", "write"]
      }
    ]
  }
}

This setup grants only the necessary cluster privileges for Jaeger to start and operate reliably. For more details, see the Jaeger Elasticsearch storage config and authentication code here.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[MicroOps-cn]

Thank you very much. According to your prompt, it has been successful.
The key configurations are as follows:

elasticsearch:
  indices:
    index_prefix: jaeger-primary
  server_urls: ["https://*****************.elastic-cloud.com:443"]
  auth:
    api_key: 
      file_path: /etc/elasticsearch.apikey
  disable_health_check: false
  sniffing:
    enabled: false
  create_mappings: false 

The permission configuration is as follows:

{
  "jaeger-role": {
    "cluster": [
      "monitor",
      "monitor_stats"
    ],
    "indices": [
      {
        "names": [
          "jaeger-primary-*"
        ],
        "privileges": [
          "read",
          "write",
          "create_index"
        ],
        "allow_restricted_indices": false
      }
    ],
    "applications": [],
    "run_as": [],
    "metadata": {},
    "transient_metadata": {
      "enabled": true
    }
  }
}