METRICS-4792 : OpenTelemetry Traces Protobuf InputFormat #129

kamal-narayan · 2023-02-13T10:38:20Z

Description

Add an OpenTelemetry Traces InputFormat so that Druid can ingest records that are in the form of an OpenTelemetry Traces Protobuf.
Note: This does not cover some fields in Otel Trace proto such as Events and Links. The extension as a whole is meant to be contributed upstream for further improvements and maintenance.

Key changed/added classes in this PR

Refactor common functionality between Metrics and Traces into abstract classes OpenTelemetryInputFormat / OpenTelemetryProtobufReader
OpenTelemetryTracesProtobufReader
OpenTelemetryTracesProtobufInputFormat

Testing

Unit tests
Add Benchmark test for traces

Benchmark                                  (instrumentationScopeCount)  (resourceSpanCount)  (spansCount)   Mode  Cnt       Score       Error  Units
OpenTelemetryTracesBenchmark.measureSerde                            1                    1             1  thrpt    5  290712.694 ± 16258.114  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    1             2  thrpt    5  179720.468 ±  3255.420  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    1             4  thrpt    5   97546.502 ±  2833.230  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    1             8  thrpt    5   54750.250 ±  1943.527  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    2             1  thrpt    5  145419.049 ±  3366.026  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    2             2  thrpt    5   90352.972 ±  3579.583  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    2             4  thrpt    5   48941.613 ±  5208.820  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    2             8  thrpt    5   27515.141 ±  3776.942  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    4             1  thrpt    5   72962.145 ±  2408.336  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    4             2  thrpt    5   45559.232 ±  1580.323  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    4             4  thrpt    5   26834.922 ±  1049.008  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    4             8  thrpt    5   13932.725 ±   739.088  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    8             1  thrpt    5   38448.021 ±  1102.249  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    8             2  thrpt    5   23787.015 ±   775.349  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    8             4  thrpt    5   13311.163 ±   299.108  ops/s
OpenTelemetryTracesBenchmark.measureSerde                            1                    8             8  thrpt    5    7073.504 ±   336.945  ops/s

Local testing
1. Deployed druid locally.
2. Ingested Kafka stream with Otel trace data.
3. Verified the data in the data source.

This PR has:

been self-reviewed.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

Config unit tests Fix dimension test Introduce config class Fix style Add serde test Prior to unit testing Fix style errors Add some dimensions

Benchmark + Test coverage Add Benchmark Add Benchmark / Fix tests Improved test coverage Fix line breaks Fix config builder clean up test cases fix distribution file

CLAassistant · 2023-02-13T10:38:27Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

distribution/pom.xml

xvrl · 2023-02-13T18:02:36Z

extensions-contrib/opentelemetry-extensions/pom.xml

+        <dependency>
+            <groupId>nl.jqno.equalsverifier</groupId>
+            <artifactId>equalsverifier</artifactId>
+            <scope>test</scope>
+        </dependency>


adding a whole new dependency for a single test seems a little heavyweight? Can we accomplish that without this additional dependency?

The Equals tests are required to meet the code coverage threshold. (More details in the comment below)

Given there are 10 fields to check, I think it would be better to use this library as it covers all the negative branch cases too.

ok, I wonder why we don't need it for metrics input format. It looks like we already use this dependency elsewhere in druid so I'm ok using that here.

...e/druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufInputFormat.java

...uid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufInputFormatTest.java

...druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufConfiguration.java

...he/druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufReaderTest.java

...c/main/java/org/apache/druid/data/input/opentelemetry/protobuf/OpenTelemetryInputFormat.java

...druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufConfiguration.java

...ain/java/org/apache/druid/data/input/opentelemetry/protobuf/OpenTelemetryProtobufReader.java

fixed config tests fix default test Fix access

...ns/src/main/java/org/apache/druid/data/input/opentelemetry/protobuf/OpenXProtobufReader.java

...e/druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufInputFormat.java

kamal-narayan · 2023-03-21T16:16:14Z

pom.xml

@@ -212,9 +212,9 @@
        <module>extensions-contrib/aliyun-oss-extensions</module>
        <module>extensions-contrib/prometheus-emitter</module>
        <module>extensions-contrib/opentelemetry-emitter</module>
+        <module>extensions-contrib/opentelemetry-extensions</module>


Reason for this move up:

Semaphore build jobs were failing with the following error:

[ERROR] Failed to execute goal de.thetaphi:forbiddenapis:3.2:check (default-cli) on project druid-opencensus-extensions: Check for forbidden API calls failed while scanning class 'org.apache.druid.data.input.opencensus.protobuf.OpenCensusProtobufInputRowParser' (OpenCensusProtobufInputRowParser.java): java.lang.ClassNotFoundException: org.apache.druid.data.input.opentelemetry.protobuf.AbstractProtobufReader (while looking up details about referenced class 'org.apache.druid.data.input.opencensus.protobuf.OpenCensusProtobufReader') -> [Help 1]

It looked like the forbiddenapis check was failing because of some classpath issue, even though the compilation step was passing. I wasn't able to reproduce this failure consistently on my laptop (but failed every time in Semaphore)

This issue looks to be closely related.

I initially tried using the @SuppressForbidden annotation for the OpenCensusProtobufInputRowParser class and that led to the same error for the OpenCensusProtobufReader class.
Ultimately Semaphore jobs passed with this change. I'm not sure how this change in module ordering fixes up the forbiddenapis check.

Empty-Commit

kamal-narayan added 12 commits January 25, 2023 21:40

Upgrade proto version

c53d7c6

Fix names and tests - Upgrade version

9352441

Fix open census tests

f574fa4

fix test name

dff447f

Abstract class

874b2bc

tests pass with abstract class

ce812c9

Added trace converter skeleton

c5ba6e9

Added files

b3c2c85

Merge branch '24.0.2-confluent' into kamal/add_otel_trace_converter

3e401df

All tests + Squash

128c704

Config unit tests Fix dimension test Introduce config class Fix style Add serde test Prior to unit testing Fix style errors Add some dimensions

Merge branch '24.0.2-confluent' into kamal/add_otel_trace_converter

94faf8a

Squash of Benchmark, test cleanup and distribution

d0d5cc7

Benchmark + Test coverage Add Benchmark Add Benchmark / Fix tests Improved test coverage Fix line breaks Fix config builder clean up test cases fix distribution file

kamal-narayan requested review from a team as code owners February 13, 2023 10:38

xvrl requested changes Feb 13, 2023

View reviewed changes

kamal-narayan added 6 commits February 14, 2023 19:59

Convert to Utils

dc6e3dc

Utils test cleanup

c8e3a77

Delete configuration class

04a5434

Introduce OpenX class

fdfe198

fixed config tests fix default test Fix access

Fix transitive dependecy checks

6dffb45

remove stirng utils

63cd879

kamal-narayan requested a review from xvrl February 17, 2023 15:39

Merge branch '24.0.2-confluent' into kamal/add_otel_trace_converter

e896053

xvrl reviewed Mar 17, 2023

View reviewed changes

...ns/src/main/java/org/apache/druid/data/input/opentelemetry/protobuf/OpenXProtobufReader.java Outdated Show resolved Hide resolved

xvrl reviewed Mar 17, 2023

View reviewed changes

...e/druid/data/input/opentelemetry/protobuf/traces/OpenTelemetryTracesProtobufInputFormat.java Outdated Show resolved Hide resolved

Address comments

348c0be

kamal-narayan force-pushed the kamal/add_otel_trace_converter branch from 3682487 to 348c0be Compare March 21, 2023 00:21

Fix up defaults

c05a185

kamal-narayan force-pushed the kamal/add_otel_trace_converter branch from b74a499 to c05a185 Compare March 21, 2023 16:11

kamal-narayan requested a review from xvrl March 21, 2023 16:14

kamal-narayan commented Mar 21, 2023

View reviewed changes

Empty-Commit

7f5e5ba

Empty-Commit

kamal-narayan force-pushed the kamal/add_otel_trace_converter branch from f03e847 to 7f5e5ba Compare March 21, 2023 20:51

kamal-narayan closed this Jul 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

METRICS-4792 : OpenTelemetry Traces Protobuf InputFormat #129

METRICS-4792 : OpenTelemetry Traces Protobuf InputFormat #129

kamal-narayan commented Feb 13, 2023

CLAassistant commented Feb 13, 2023 •

edited by cla-assistant bot

Loading

xvrl Feb 13, 2023

kamal-narayan Feb 14, 2023

xvrl Mar 23, 2023

kamal-narayan Mar 21, 2023 •

edited

Loading

METRICS-4792 : OpenTelemetry Traces Protobuf InputFormat #129

METRICS-4792 : OpenTelemetry Traces Protobuf InputFormat #129

Conversation

kamal-narayan commented Feb 13, 2023

Description

Key changed/added classes in this PR

Testing

CLAassistant commented Feb 13, 2023 • edited by cla-assistant bot Loading

xvrl Feb 13, 2023

Choose a reason for hiding this comment

kamal-narayan Feb 14, 2023

Choose a reason for hiding this comment

xvrl Mar 23, 2023

Choose a reason for hiding this comment

kamal-narayan Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

CLAassistant commented Feb 13, 2023 •

edited by cla-assistant bot

Loading

kamal-narayan Mar 21, 2023 •

edited

Loading