jaeger issue with tracing

[subhodeeproy766]

we have deployed istio ambient mode 1-24-1 via gateway api, everything seems to be work apart from tracing, we have a simple webapp, an orchestrator and 2 backend ms (one dotnet another node).
we are using kiali: 2.0.0, jaeger ui 1.37.0, jaeger v1.53.0, we are also doing open telemetry instrumentation via otel and configuring it in sourcecode (dockerfile) itself .

Below is the error when we stream jaeger -query logs:

{"level":"error","ts":1742586647.7421339,"caller":"app/grpc_handler.go:157","msg":"failed when searching for traces","error":"service Name must be set","stacktrace":"github.com/jaegertracing/jaeger/cmd/query/app.(*GRPCHandler).FindTraces\n\tgithub.com/jaegertracing/jaeger/cmd/query/app/grpc_handler.go:157\ngithub.com/jaegertracing/jaeger/proto-gen/api_v2._QueryService_FindTraces_Handler\n\tgithub.com/jaegertracing/jaeger/proto-gen/api_v2/query.pb.go:1048\ngoogle.golang.org/grpc.(*Server).processStreamingRPC\n\tgoogle.golang.org/[email protected]/server.go:1666\ngoogle.golang.org/grpc.(*Server).handleStream\n\tgoogle.golang.org/[email protected]/server.go:1787\ngoogle.golang.org/grpc.(*Server).serveStreams.func2.1\n\tgoogle.golang.org/[email protected]/server.go:1016"}
{"level":"info","ts":1742800752.062128,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""","system":"grpc","grpc_log":true}
{"level":"info","ts":1742800753.1036847,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""","system":"grpc","grpc_log":true}
{"level":"info","ts":1742800755.3721342,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""","system":"grpc","grpc_log":true}
{"level":"info","ts":1742800756.4174945,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET / HTTP/1.1\\r\\nHost: lo\""","system":"grpc","grpc_log":true}
{"level":"info","ts":1742800758.6240022,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET /jaeger HTTP/1.1\\r\\nHo\""","system":"grpc","grpc_log":true}
{"level":"info","ts":1742800759.6764448,"caller":"[email protected]/server.go:983","msg":"[core][Server #1] grpc: Server.Serve failed to create ServerTransport: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"GET /jaeger HTTP/1.1\\r\\nHo\""","system":"grpc","grpc_log":true}

##########################

Below is our control plane manifest

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
name: istio-control-plane
spec:
profile: ambient
components:
pilot:
k8s:
# nodeSelector:
# <##############>
priorityClassName: system-cluster-critical
tolerations:
- effect: NoSchedule
key: sku
operator: Equal
value: istio
ingressGateways:
- enabled: false
name: istio-ingressgateway
cni:
enabled: true
# namespace: kube-system
ztunnel:
enabled: true
# namespace: kube-system
meshConfig:
enablePrometheusMerge: true
defaultConfig:
holdApplicationUntilProxyStarts: true
discoverySelectors:
- matchLabels:
istio-discovery: enabled
outboundTrafficPolicy:
mode: ALLOW_ANY
enableTracing: true
tracing:
sampling: 100.0
extensionProviders:
- name: otel-tracing
opentelemetry:
port: 4317
service: istio-collector..svc.cluster.local
# http:
# path: "/v1/traces"
# timeout: 10s
resource_detectors:
environment: {}
# tracing 3.1 END
# defaultProviders:
# metrics:
# - prometheus
values:
global:
proxy:
autoInject: disabled
istio_cni:
enabled: true

######################

This is the manifest for the open telemetry collector object,

apiVersion: v1
items:

• apiVersion: opentelemetry.io/v1beta1
  kind: OpenTelemetryCollector
  metadata:
  annotations:
  kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"opentelemetry.io/v1beta1","kind":"OpenTelemetryCollector","metadata":{"annotations":{},"}}
  creationTimestamp: "2025-03-12T13:50:11Z"
  finalizers:

  • opentelemetrycollector.opentelemetry.io/finalizer
    generation: 1
    labels:
    app.kubernetes.io/managed-by: opentelemetry-operator
    name: istio
    namespace: <>
    spec:
    config:
    exporters:
    debug:
    verbosity: detailed
    otlp/jaeger:
    endpoint: http://jaeger-collector..svc.cluster.local:4317
    tls:
    insecure: true
    extensions:
    health_check: {}
    pprof: {}
    zpages:
    endpoint: 0.0.0.0:55679
    processors:
    batch: {}
    filter/ottl:
    error_mode: ignore
    traces:
    span:
    - IsMatch(attributes["user_agent.original"], ".kube-probe.")
    - IsMatch(attributes["http.route"], ".healthz.")
    - IsMatch(attributes["http.route"], ".readyz.")
    receivers:
    otlp:
    protocols:
    grpc:
    endpoint: 0.0.0.0:4317
    http:
    endpoint: 0.0.0.0:4318
    service:
    extensions:
    • health_check
    • pprof
    • zpages
      pipelines:
      traces:
      exporters:
      • debug
      • otlp/jaeger
        processors:
      • batch
      • filter/ottl
        receivers:
      • otlp
        telemetry:
        logs:
        level: debug
        configVersions: 3
        daemonSetUpdateStrategy: {}
        deploymentUpdateStrategy: {}
        ingress:
        route: {}
        managementState: managed
        mode: deployment
        observability:
        metrics: {}
        podDisruptionBudget:
        maxUnavailable: 1
        replicas: 1
        resources: {}
        targetAllocator:
        allocationStrategy: consistent-hashing
        filterStrategy: relabel-config
        observability:
        metrics: {}
        prometheusCR:
        scrapeInterval: 30s
        resources: {}
        upgradeStrategy: automatic

  ###################

kiali configuration is:

apiVersion: v1
items:

• apiVersion: kiali.io/v1alpha1
  kind: Kiali
  metadata:
  annotations:
  kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"kiali.io/v1alpha1","kind":"Kiali","metadata":{"annotations":{},"cre****************"}}
  finalizers:
  • kiali.io/finalizer
    generation: 53
    name: kiali
    namespace:
    resourceVersion: "1330345407"
    uid: d15172bb-beed-4e77-ba3c-581b483bccd4
    spec:
    auth:
    strategy: token
    deployment:
    accessible_namespaces:
    • '**'
      logger:
      log_level: trace
      external_services:
      istio:
      config_map_name: istio-1-24-2
      istio_sidecar_injector_config_map_name: istio-sidecar-injector-1-24-2
      istiod_deployment_name: istiod-1-24-2
      prometheus:
      cache_duration: 10
      cache_enabled: true
      cache_expiration: 300
      url: http://kube-prometheus-stack-prometheus..svc.cluster.local:9090
      tracing:
      enabled: true
      internal_url: http://jaeger-query..svc.cluster.local:16685/jaeger
      namespace_selector: false
      provider: jaeger
      use_grpc: true
      whitelist_istio_system:
  istio_namespace: istio-system

[subhodeeproy766]

[dosubot[bot]]

Hey @subhodeeproy766! 👋 I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's tackle this together!

I found a similar discussion titled "jaeger Query UI doesn't show any Service" which is still open and unsolved ^[1]. This might be relevant to your issue with the "service Name must be set" error.

To resolve the "service Name must be set" error in Jaeger, ensure that the service.name attribute is correctly set in your OpenTelemetry instrumentation. This can be done by configuring the service name in your OpenTelemetry setup. Here's an example in Go:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/semconv/v1.4.0"
)

func setupTracer() {
    res, err := resource.New(
        context.Background(),
        resource.WithAttributes(
            semconv.ServiceNameKey.String("your-service-name"),
        ),
    )
    if err != nil {
        log.Fatalf("failed to create resource: %v", err)
    }

    // Set up your tracer provider with the resource
    tp := trace.NewTracerProvider(
        trace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
}

Replace "your-service-name" with the actual name of your service to ensure the service name is set correctly ^[2].

Regarding the 'grpc: Server.Serve failed to create ServerTransport' error, common causes include invalid port numbers or closed listeners. Ensure that your gRPC server is configured with a valid port and that the listener is open before starting the server ^[3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[yurishkuro]

The service-name related error is from the query path. Jaeger UI doesn't make queries like that, so you might be getting this from Kiali. The other errors look like you're sending HTTP payloads to GRPC port.

[subhodeeproy766]

how can i check that? can you help me with that,