Issue with High Trace Volume in Jaeger v2.2.0 Setup

[edo2509]

Hi team,

I’m encountering an issue with Jaeger after upgrading to version 2 (currently using v2.2.0). My telemetry setup includes multiple Podman containers on a RHEL 9.5 server (16GB RAM). While everything works as expected with a small trace volume, high trace traffic (20k+ traces) leads to errors in both Jaeger and the OpenTelemetry Collector.

Setup Details:

• Running Containers (in order of data flow):

1. Traefik: Acts as a reverse proxy
2. OpenTelemetry Collector (v0.112.0): Receives telemetry data (OTLP format), processes it, and exports traces to Jaeger.
3. Jaeger (v2.2.0): Ingests the traces from the OpenTelemetry Collector and stores them in Cassandra.
4. Cassandra (v4.1.3): Serves as the backend database for storing trace data.

System Configuration:

• Host: Single machine, same network for all containers
• Cassandra DB initialized with plugin/storage/cassandra/schema/v004.cql.tmpl
• Adjusted Cassandra settings:
  • read_request_timeout: 9000ms
  • write_request_timeout: 4000ms

Observed Behavior:

When the trace volume increases to 20k+ traces, I encounter the following issues:

Jaeger Logs:
traces export: context deadline exceeded: rpc error: code = DeadlineExceeded desc = context deadline exceeded

OpenTelemetry Collector Logs:

2025-01-13T08:58:17.364Z        info    internal/retry_sender.go:126    Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "traces", "name": "otlp/jaeger", "error": "rpc error: code = DeadlineExceeded desc = context deadline exceeded", "interval": "11.797811565s"}
2025-01-13T08:58:24.980Z        error   internal/queue_sender.go:92     Exporting failed. Dropping data.        {"kind": "exporter", "data_type": "traces", "name": "otlp/jaeger", "error": "no more retries left: rpc error: code = DeadlineExceeded desc = context deadline exceeded", "dropped_items": 5}

The logs show that retries are attempted but eventually fail, and data gets dropped.

Configuration Details:

Jaeger config.yaml:

service:
  extensions: [jaeger_storage, jaeger_query, healthcheckv2]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger_storage_exporter]
  telemetry:
    resource:
      service.name: jaeger
    metrics:
      level: basic
      address: 0.0.0.0:8888
    logs:
      level: WARN

extensions:
  healthcheckv2:
    use_v2: true
    http:

  jaeger_query:
    storage:
      traces: some_storage
    grpc:
      endpoint: 0.0.0.0:16685
    http:
      endpoint: 0.0.0.0:16686

  jaeger_storage:
    backends:
      some_storage:
        cassandra:
          schema:
            keyspace: "jaeger_db"
          query:
            timeout: 20s
          connection:
            servers:
              - "10.10.10.252" # address of Cassandra container
            port: 9042
            timeout: 20s
            tls:
              insecure: true

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  jaeger:
    protocols:
      grpc:
      thrift_binary:
      thrift_compact:
      thrift_http:

processors:
  batch:
    timeout: 10s
    send_batch_size: 500
    send_batch_max_size: 2000

exporters:
  jaeger_storage_exporter:
    trace_storage: some_storage
    queue:
      num_consumers: 10
      queue_size: 100

OpenTelemetry Collector config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  attributes:
    actions:
      - action: insert
        key: severityText
        from_attribute: SeveretyText 
  batch:
    send_batch_size: 8000
    timeout: 20s
    send_batch_max_size: 30000

exporters:
  otlphttp/jaeger:
    endpoint: http://jaeger:4318
    timeout: 30s
    tls:
      insecure: true


service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/jaeger]

What I’ve Tried:

• Adjusting settings in both Jaeger and OpenTelemetry Collector (e.g., timeout, send_batch_size, send_batch_max_size).
• Increasing Cassandra query timeouts (query.timeout and connection.timeout).
• Verifying network communication between containers (no issues found).

Despite these attempts, the problem persists. Cassandra logs show no warnings or errors during this time.
Could you please help identify the root cause and suggest potential solutions? Specifically:
Is this related to a misconfiguration in Jaeger, OpenTelemetry Collector, or Cassandra?
Are there known limitations or optimizations for handling high trace volumes in Jaeger v2?
Any recommendations for tuning the batch processors or Cassandra settings?

I hope I’m asking for help in the right place.
Thank you in advance for your assistance!

[dosubot[bot]]

Hello, @edo2509!

I'm here to help you with any bugs, questions, or contributions you have. Let's work together to resolve your issue.

To address the issues you're experiencing with high trace volumes in Jaeger v2.2.0, here are some recommendations and optimizations you can consider:

1. Batch Processor Configuration:

   • In Jaeger, increase the collector.queue-size to a higher value to handle sudden spikes in traffic. The default is 2000, but you might need to increase it based on your memory capacity.
   • Adjust collector.num-workers to ensure the queue remains close to zero. A general rule is one worker per 50 items in the queue. For a collector.queue-size of 2000, about 40 workers might be needed for fast storage ^[1].

2. Cassandra Optimizations:

   • Increase the number of connections per host to allow more concurrent operations.
   • Set the maximum number of retry attempts for reading from Cassandra to handle transient failures.
   • Adjust the query and connection timeouts to ensure they do not hang indefinitely.
   • Choose an appropriate consistency level based on your performance and data consistency requirements ^[2].

3. OpenTelemetry Collector Configuration:

   • Ensure that the send_batch_size and send_batch_max_size are set appropriately. You might need to experiment with these values to find the optimal configuration for your setup.
   • Consider increasing the timeout settings in both the batch processor and the exporter to accommodate longer processing times under high load.

4. General Recommendations:

   • Use Apache Kafka as an intermediate buffer by setting SPAN_STORAGE_TYPE to kafka in the Jaeger collector. This can help manage frequent traffic spikes by buffering traces before they are written to storage ^[1].

These adjustments should help improve the handling of high trace volumes in your setup. If the problem persists, you may need to further investigate the network latency or resource constraints on your server.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[yurishkuro]

I would say Cassandra is your bottleneck, especially if you're running everything on a single host. I would look for write latency metrics (Jaeger should have them) to see if they are increasing with the increase of the load, to the point where they are starting to exceed the timeout settings.

Increasing the timeout is not going to help, it would just lead to more overload in the system and other problems.