Skip to content

feat: add tag-based filtering API endpoints #7505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

feat: add tag-based filtering API endpoints #7505

wants to merge 8 commits into from

Conversation

wololowarrior
Copy link
Contributor

@wololowarrior wololowarrior commented Sep 7, 2025 •
edited
Loading

Which problem is this PR solving?

Description of the changes

  • This PR returns the list of attribute values for an attribute and service.
  • A new api was created /api/metrics/attributes. This accepts a few query parameters
    • [Mandatory] key or the tag for which the values are fetched for
    • [Optional] service, service name to filter down more
  • In elasticsearch we look for 3 nested documents defined in NestedTagFieldList.
  • We look back for past 1 hour to fetch for potential attribute values.

For example this api request can be used to fetch all the values for tag http.method and service driver.

[API]
curl -s "http://localhost:16686/api/metrics/labels?key=environment&service=driver"
{
  "data": [
    "staging"
  ],
  "total": 1,
  "limit": 0,
  "offset": 0,
  "errors": null
}

TBD

How was this change tested?

  • Localsetup

Prometheus

config-spm.yaml

You can update the docker compose file `jaeger/docker-compose/monitor/docker-compose.yml` to simulate two env: production & staging 

services:
  jaeger:
    networks:
      backend:
        # This is the host name used in Prometheus scrape configuration.
        aliases: [ spm_metrics_source ]
    image: jaegertracing/jaeger:${JAEGER_VERSION:-latest}
    volumes:
      - "./jaeger-ui.json:/etc/jaeger/jaeger-ui.json" # Do we need this for v2 ? Seems to be running without this.
      - "../../cmd/jaeger/config-spm.yaml:/etc/jaeger/config.yml"
    command: ["--config", "/etc/jaeger/config.yml"]
    ports:
      - "16686:16686"
      - "8888:8888"
      - "8889:8889"
      - "4317:4317"
      - "4318:4318"

  microsim:
    networks:
      - backend
    image: yurishkuro/microsim:v0.5.0@sha256:b7ee2dee51d2c9fd94de08a80278cfbf5a144ad0f22efce50f3d3be15cbfa2c7
    command: "-d 24h -s 500ms"
    environment:
      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4318/v1/traces
      - OTEL_RESOURCE_ATTRIBUTES=environment=production,region=us-east-1
    depends_on:
      - jaeger

  microsim-2:
    networks:
      - backend
    image: yurishkuro/microsim:v0.5.0@sha256:b7ee2dee51d2c9fd94de08a80278cfbf5a144ad0f22efce50f3d3be15cbfa2c7
    command: "-d 24h -s 5000ms"
    environment:
      - OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://jaeger:4318/v1/traces
      - OTEL_RESOURCE_ATTRIBUTES=environment=staging,region=us-east-2
    depends_on:
      - jaeger

  prometheus:
    networks:
      - backend
    image: prom/prometheus:v3.5.0@sha256:63805ebb8d2b3920190daf1cb14a60871b16fd38bed42b857a3182bc621f4996
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
    ports:
      - "9090:9090"

networks:
  backend:

You can update the config-spm.yaml. To push environment and region to the metrics in prometheus

service:
  extensions: [jaeger_storage, jaeger_query]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [attributes, batch]
      exporters: [jaeger_storage_exporter, spanmetrics]
    metrics/spanmetrics:
      receivers: [spanmetrics]
      exporters: [prometheus]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: detailed
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888
    logs:
      level: INFO

extensions:
  jaeger_query:
    storage:
      traces: some_storage
      metrics: some_metrics_storage
    ui:
      config_file: /etc/jaeger/jaeger-ui.json
  jaeger_storage:
    backends:
      some_storage:
        memory:
          max_traces: 100000
    metric_backends:
      some_metrics_storage:
        prometheus:
          endpoint: http://prometheus:9090
          normalize_calls: true
          normalize_duration: true

# The spanmetrics connector generates metrics from spans
connectors:
  spanmetrics:
    dimensions:
      - name: operation
        default: unknown-operation
      # Add tag dimensions for filtering
      - name: environment
        default: ""
      # Include original HTTP status code
      - name: http.status_code
      - name: region
        default: ""
    dimensions_cache_size: 1000
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
    metrics_flush_interval: 15s

receivers:
  otlp:
    protocols:
      grpc:
      http:
        endpoint: "0.0.0.0:4318"

processors:
  # The attributes processor copies resource attributes to span attributes
  # This is required for tag filtering in metrics
  attributes:
    actions:
      # Copy resource attributes to span attributes for metrics filtering
      - key: environment
        action: insert
        from_attribute: environment
      - key: region
        action: insert
        from_attribute: region
      - key: http.status_code
        action: insert
        from_attribute: http.status_code
  batch:

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage
  prometheus:
    endpoint: "0.0.0.0:8889"

and jaeger-ui.json can be updated to

{
  "monitor": {
    "menuEnabled": true,
    "tagFilters": [
      {
        "name": "environment",
        "label": "Environment"
      },
      {
        "name": "region",
        "label": "Region"
      }
    ]
  },
  "dependencies": {
    "menuEnabled": true
  }
}

which will configure the ui to show two filters Environment & Region

ElasticSearch

Checklist

@codecov Codecov
Copy link

codecov bot commented Sep 7, 2025 •
edited
Loading

Codecov Report

❌ Patch coverage is 96.02649% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.61%. Comparing base (b966cc5) to head (37144b2).

Files with missing lines Patch % Lines
...ternal/storage/metricstore/elasticsearch/reader.go 90.16% 4 Missing and 2 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #7505      +/-   ##
==========================================
+ Coverage   96.59%   96.61%   +0.01%     
==========================================
  Files         383      383              
  Lines       23238    23386     +148     
==========================================
+ Hits        22447    22594     +147     
  Misses        602      602              
- Partials      189      190       +1
Flag Coverage Δ
badger_v1 9.02% <0.00%> (ø)
badger_v2 1.68% <0.00%> (-0.02%) ⬇️
cassandra-4.x-v1-manual 11.67% <0.00%> (ø)
cassandra-4.x-v2-auto 1.67% <0.00%> (-0.02%) ⬇️
cassandra-4.x-v2-manual 1.67% <0.00%> (-0.02%) ⬇️
cassandra-5.x-v1-manual 11.67% <0.00%> (ø)
cassandra-5.x-v2-auto 1.67% <0.00%> (-0.02%) ⬇️
cassandra-5.x-v2-manual 1.67% <0.00%> (-0.02%) ⬇️
elasticsearch-6.x-v1 16.57% <100.00%> (ø)
elasticsearch-7.x-v1 16.61% <100.00%> (ø)
elasticsearch-8.x-v1 16.75% <100.00%> (ø)
elasticsearch-8.x-v2 1.68% <0.00%> (-0.02%) ⬇️
elasticsearch-9.x-v2 1.68% <0.00%> (-0.02%) ⬇️
grpc_v1 10.22% <0.00%> (ø)
grpc_v2 1.68% <0.00%> (-0.02%) ⬇️
kafka-3.x-v1 9.66% <0.00%> (ø)
kafka-3.x-v2 1.68% <0.00%> (-0.02%) ⬇️
memory_v2 1.68% <0.00%> (-0.02%) ⬇️
opensearch-1.x-v1 16.66% <100.00%> (ø)
opensearch-2.x-v1 16.66% <100.00%> (ø)
opensearch-2.x-v2 1.68% <0.00%> (-0.11%) ⬇️
opensearch-3.x-v2 1.68% <0.00%> (-0.11%) ⬇️
query 1.68% <0.00%> (-0.02%) ⬇️
tailsampling-processor 0.46% <0.00%> (-0.01%) ⬇️
unittests 95.61% <96.02%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

yurishkuro
yurishkuro reviewed Sep 7, 2025
View reviewed changes
cmd/query/app/http_handler.go Outdated
aH.handleFunc(router, aH.calls, "/metrics/calls").Methods(http.MethodGet)
aH.handleFunc(router, aH.errors, "/metrics/errors").Methods(http.MethodGet)
aH.handleFunc(router, aH.minStep, "/metrics/minstep").Methods(http.MethodGet)
aH.handleFunc(router, aH.attributeValues, "/metrics/labels").Methods(http.MethodGet)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
aH.handleFunc(router, aH.attributeValues, "/metrics/labels").Methods(http.MethodGet)
aH.handleFunc(router, aH.attributeValues, "/metrics/attributes").Methods(http.MethodGet)

yurishkuro
yurishkuro reviewed Sep 7, 2025
View reviewed changes
cmd/query/app/http_handler.go Outdated
func (aH *APIHandler) attributeValues(w http.ResponseWriter, r *http.Request) {
key := r.URL.Query().Get("key")
if key == "" {
aH.handleError(w, errors.New("key is required"), http.StatusBadRequest)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
aH.handleError(w, errors.New("key is required"), http.StatusBadRequest)
aH.handleError(w, errors.New("attribute key is required"), http.StatusBadRequest)

yurishkuro
yurishkuro reviewed Sep 7, 2025
View reviewed changes
internal/storage/v1/api/metricstore/interface.go Outdated
Comment on lines 81 to 82
AttributeTargetTag AttributeTarget = "tag"
AttributeTargetDefault AttributeTarget = "tags"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are these for?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, these are not needed now.

yurishkuro
yurishkuro reviewed Sep 7, 2025
View reviewed changes
internal/storage/v1/api/metricstore/interface.go Outdated
Comment on lines 79 to 80
AttributeTargetTags AttributeTarget = "tags"
AttributeTargetProcessTags AttributeTarget = "process.tags"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
AttributeTargetTags AttributeTarget = "tags"
AttributeTargetProcessTags AttributeTarget = "process.tags"
AttributeTargetSpan AttributeTarget = "span"
AttributeTargetProcess AttributeTarget = "scope"
AttributeTargetScope AttributeTarget = "process"

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 8, 2025 •
edited
Loading

Metrics Comparison Summary

Total changes across all snapshots: 252

Detailed changes per snapshot

summary_metrics_snapshot_opensearch

📊 Metrics Diff Summary

Total Changes: 73

  • 🆕 Added: 73 metrics
  • ❌ Removed: 0 metrics
  • 🔄 Modified: 0 metrics

🆕 Added Metrics

  • jaeger_storage_latency_seconds (18 variants)
View diff sample 
+jaeger_storage_latency_seconds{kind="opensearch",le="+Inf",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="0",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="10",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="100",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="1000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="10000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="25",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
...
- `jaeger_storage_requests` (1 variants)
View diff sample 
+jaeger_storage_requests{kind="opensearch",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
- `rpc_server_duration_milliseconds` (18 variants)
View diff sample 
+rpc_server_duration_milliseconds{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
- `rpc_server_requests_per_rpc` (18 variants)
View diff sample 
+rpc_server_requests_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
- `rpc_server_responses_per_rpc` (18 variants)
View diff sample 
+rpc_server_responses_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
### summary_metrics_snapshot_opensearch ## 📊 Metrics Diff Summary

Total Changes: 73

  • 🆕 Added: 73 metrics
  • ❌ Removed: 0 metrics
  • 🔄 Modified: 0 metrics

🆕 Added Metrics

  • jaeger_storage_latency_seconds (18 variants)
View diff sample 
+jaeger_storage_latency_seconds{kind="opensearch",le="+Inf",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="0",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="10",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="100",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="1000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="10000",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
+jaeger_storage_latency_seconds{kind="opensearch",le="25",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
...
- `jaeger_storage_requests` (1 variants)
View diff sample 
+jaeger_storage_requests{kind="opensearch",name="some_storage",operation="find_traces",otel_scope_name="jaeger-v2",otel_scope_schema_url="",otel_scope_version="",result="err",role="tracestore"}
- `rpc_server_duration_milliseconds` (18 variants)
View diff sample 
+rpc_server_duration_milliseconds{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_duration_milliseconds{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
- `rpc_server_requests_per_rpc` (18 variants)
View diff sample 
+rpc_server_requests_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_requests_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
- `rpc_server_responses_per_rpc` (18 variants)
View diff sample 
+rpc_server_responses_per_rpc{le="+Inf",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="0",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="100",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="1000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="10000",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
+rpc_server_responses_per_rpc{le="25",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc",otel_scope_schema_url="https://opentelemetry.io/schemas/1.34.0",otel_scope_version="0.62.0",rpc_grpc_status_code="2",rpc_method="FindTraces",rpc_service="jaeger.api_v3.QueryService",rpc_system="grpc"}
...
### summary_metrics_snapshot_cassandra ## 📊 Metrics Diff Summary

Total Changes: 53

  • 🆕 Added: 0 metrics
  • ❌ Removed: 53 metrics
  • 🔄 Modified: 0 metrics

❌ Removed Metrics

  • http_server_request_body_size_bytes (18 variants)
View diff sample 
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...
- `http_server_request_duration_seconds` (17 variants)
View diff sample 
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.005",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.01",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.025",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.05",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.075",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.1",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...
- `http_server_response_body_size_bytes` (18 variants)
View diff sample 
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...
### summary_metrics_snapshot_cassandra ## 📊 Metrics Diff Summary

Total Changes: 53

  • 🆕 Added: 0 metrics
  • ❌ Removed: 53 metrics
  • 🔄 Modified: 0 metrics

❌ Removed Metrics

  • http_server_request_body_size_bytes (18 variants)
View diff sample 
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...
- `http_server_request_duration_seconds` (17 variants)
View diff sample 
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.005",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.01",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.025",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.05",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.075",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_request_duration_seconds{http_request_method="GET",http_response_status_code="503",le="0.1",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...
- `http_server_response_body_size_bytes` (18 variants)
View diff sample 
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="+Inf",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="0",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="100",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="1000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="10000",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
-http_server_response_body_size_bytes{http_request_method="GET",http_response_status_code="503",le="25",network_protocol_name="http",network_protocol_version="1.1",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_schema_url="",otel_scope_version="0.62.0",server_address="localhost",server_port="13133",url_scheme="http"}
...

➡️ View full metrics file

yurishkuro
yurishkuro reviewed Sep 9, 2025
View reviewed changes
internal/storage/metricstore/elasticsearch/query_builder.go
// Create a global aggregation that will contain all nested aggregations
globalAgg := elastic.NewGlobalAggregation()

for i, path := range spanstore.NestedTagFieldList {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you loop through all these?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I ditched the ResourceTarget query parameter in favour of searching 3-4 places where a tag could be present in ES.

I took inspiration from the already working code in the search page where we filter by tag

jaeger/internal/storage/v1/elasticsearch/spanstore/reader.go

Lines 656 to 668 in fa9a5e7

func (s *SpanReader) buildTagQuery(k string, v string) elastic.Query {
objectTagListLen := len(objectTagFieldList)
queries := make([]elastic.Query, len(nestedTagFieldList)+objectTagListLen)
kd := s.dotReplacer.ReplaceDot(k)
for i := range objectTagFieldList {
queries[i] = s.buildObjectQuery(objectTagFieldList[i], kd, v)
}
for i := range nestedTagFieldList {
queries[i+objectTagListLen] = s.buildNestedQuery(nestedTagFieldList[i], k, v)
}
// but configuration can change over time
return elastic.NewBoolQuery().Should(queries...)

In span fitlter by tag, We only seem to search in 3 places, and its been working fine I guess?

jaeger/internal/storage/v1/elasticsearch/spanstore/reader.go

Line 78 in fa9a5e7

nestedTagFieldList = []string{nestedTagsField, nestedProcessTagsField, nestedLogFieldsField}

We are doing something similar in the new api to get tag values. So instead of providing the exact path to search at, which you know requires the user to dig into ES and figure out the nested path, we always search in some predefined places.

yurishkuro
yurishkuro reviewed Sep 9, 2025
View reviewed changes
internal/storage/metricstore/elasticsearch/reader_test.go Outdated
}
}

func TestGetLabelValues(t *testing.T) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add test that verifies which query is being built?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

yurishkuro
yurishkuro reviewed Sep 9, 2025
View reviewed changes
internal/storage/metricstore/elasticsearch/reader.go
return nil, fmt.Errorf("failed to execute attribute values query: %w", err)
}

// Collect values from all paths (path_0, path_1, path_2, etc.)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what do you mean by path_0 ... ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is simply the alias/unique identifier for each nested aggregation. The path_, idx is simply the index of path in nestedTagFieldList .

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its defined here

globalAgg.SubAggregation(fmt.Sprintf("path_%d", i), nestedAgg)

yurishkuro
yurishkuro reviewed Sep 9, 2025
View reviewed changes
internal/storage/metricstore/elasticsearch/query_builder.go
}

// Create a global aggregation that will contain all nested aggregations
globalAgg := elastic.NewGlobalAggregation()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this an aggregation query? Seems like you're doing something like GROUP BY, but isn't there a simpler SELECT DISTINCT flavor?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we search in multiple nested documents (three currently), defined here

jaeger/internal/storage/v1/elasticsearch/spanstore/reader.go

Line 78 in fa9a5e7

nestedTagFieldList = []string{nestedTagsField, nestedProcessTagsField, nestedLogFieldsField}

a simpler appraoch would work in case we just search in a given location. Like process.tags, but I feel predefining the location of the attributes is not needed.

This was referenced Sep 9, 2025
Open
Open
@wololowarrior wololowarrior marked this pull request as ready for review September 11, 2025 09:25
@wololowarrior wololowarrior requested a review from a team as a code owner September 11, 2025 09:25
@wololowarrior
feat: add tag-based filtering API endpoints
df6b491 
This commit introduces comprehensive tag-based filtering capabilities:

- Add GetAttributeKeyValues and GetAttributeValues to the metricstore interface
- Implement tag filtering for both Elasticsearch and Prometheus backends
- Add HTTP endpoints /api/v1/attribute-keys and /api/v1/attribute-values
- Include comprehensive test coverage with mock data and unit tests
- Update disabled reader with proper NotImplemented responses
- Add metrics decorator support for new API methods

The implementation allows users to filter metrics by specific tag keys and values,
improving the observability and querying capabilities of Jaeger metrics.

Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
Address cicd comments
9407158 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
Increasing test cover
164fa70 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
update api endpoint
0a59c2e 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
Fix test for more coverage
0c01cff 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
Fix service format
5245e5e 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
add query in test
5a7781f 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior
Replace '.' with '_' in prometheus
37144b2 
Signed-off-by: Harshil Gupta <[email protected]>
@wololowarrior wololowarrior force-pushed the api-to-get-tag-values-from-backend-squashed branch from e8550bb to 37144b2 Compare September 30, 2025 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yurishkuro yurishkuro yurishkuro left review comments

@albertteoh albertteoh Awaiting requested review from albertteoh albertteoh is a code owner automatically assigned from jaegertracing/jaeger-maintainers

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area/storage storage/elasticsearch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[SPM] Filter metrics on page by tag

2 participants

@wololowarrior @yurishkuro