Skip to content

Java agent for Spark Streaming Kafka consumer #14174

Open
@patpatpat123

Description

@patpatpat123

Is your feature request related to a problem? Please describe.

Hello team,

First of all, as this is my first post here, I would like to say thanks for the amazing work on the Java agent.
If not anything else, just thank you for this awesome project.

With that said, I would like to reach out with an enhancement request.

As of today, with Spark 4 and Java agent2, it seems it is not possible to have traces for Spark structured streaming Kafka consumer tracing.

By that I mean: suppose a Kafka consumer, written in Go, Python, Java, or else, which puts in the Kafka header a valid W3C trace ID.

Using Spark and the option to include headers, the Spark streaming service is able to consume the message and even see the valid trace.

However, even with the latest Java agent version 2, there is no way to have the traces propagated.

Describe the solution you'd like

As of today, the only solution possible is for the user to perform manual work on the Spark side to continue and propagate the trace.

Instead of having to ask each and every Spark structured streaming Kafka consumer job in the world to perform its own implementation, it would be great if the Java agent could help with the tracing.

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestneeds triageNew issue that requires triage

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions