Upgrage storage integration test to storage v2 API #6388

ekefan · 2024-12-19T13:41:35Z

Which problem is this PR solving?

Resolves Upgrade storage integration tests to Storage v2 API #6366

Description of the changes

Incrementally swaps the fields of StorageIntegration to align with v2 storage api while supporting v1 api
Updates test functions accordingly to work with the updated fields

How was this change tested?

make test

Checklist

I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
I have signed all commits
I have added unit tests for the new functionality
I have run lint and test steps successfully
- for jaeger: make lint test
- for jaeger-ui: npm run lint and npm run test

ekefan · 2024-12-19T14:49:20Z

Hello @yurishkuro,

please review this implementation.

yurishkuro · 2024-12-19T15:40:29Z

I need to go through it more carefully but on a brief scan looks very much in the right direction.

plugin/storage/integration/integration.go

yurishkuro · 2024-12-21T23:36:49Z

storage_v2/v1adapter/ptrace_model.go

+			spans := modelSpansFromOtelTrace(otelTrace)
+			for _, span := range spans {
+				traceId := span.TraceID.ToOTELTraceID()
+				if _, exists := tracesByID[traceId]; !exists {


Now you went into opposite direction - not trusting the contact of the iterator. The contract says that if a trace is split into more than one chunk then those chunks will be consecutive. That means you can re-assemble the model traces without maintaining a map, simply by keeping track of the last seen trace ID.

jaeger/storage_v2/tracestore/reader.go

Lines 22 to 24 in 42a20f0

// Chunking requirements:

// - A single ptrace.Traces chunk MUST NOT contain spans from multiple traces.

// - Large traces MAY be split across multiple, *consecutive* ptrace.Traces chunks.

// Chunking requirements: // - A single ptrace.Traces chunk MUST NOT contain spans from multiple traces. // - Large traces MAY be split across multiple, *consecutive* ptrace.Traces chunks.

You can use AggregateTraces function from internal/jptrace/aggregator.go together with iter.CollectWithErrors:

traces := iter.iter.CollectWithErrors(jptrace.AggregateTraces(seqTrace))

The traces is now a plain slice where each element is a complete trace that can be passed to modelSpansFromOtelTrace

yurishkuro · 2024-12-21T23:38:20Z

storage_v2/v1adapter/ptrace_model.go

+// PTracesSeq2ToModel consumes an otel trace iterator and returns a jaeger model trace.
+//
+// When necessary, it groups chunks of traces into one single trace
+func PTracesSeq2ToModel(seqTrace iter.Seq2[[]ptrace.Traces, error]) ([]*model.Trace, error) {


please add a test

yurishkuro · 2024-12-21T23:39:44Z

plugin/storage/integration/integration.go

@@ -213,11 +217,14 @@ func (s *StorageIntegration) testGetLargeSpan(t *testing.T) {

 	expected := s.writeLargeTraceWithDuplicateSpanIds(t)
 	expectedTraceID := expected.Spans[0].TraceID
+	expectedOtelTraceID := expectedTraceID.ToOTELTraceID()

 	var actual *model.Trace
 	found := s.waitForCondition(t, func(_ *testing.T) bool {
 		var err error


Suggested change

var err error

yurishkuro · 2024-12-21T23:40:22Z

plugin/storage/integration/integration.go

@@ -286,14 +293,17 @@ func (s *StorageIntegration) testGetTrace(t *testing.T) {

 	expected := s.loadParseAndWriteExampleTrace(t)
 	expectedTraceID := expected.Spans[0].TraceID
+	expectedOtelTraceID := expectedTraceID.ToOTELTraceID()

 	var actual *model.Trace
 	found := s.waitForCondition(t, func(t *testing.T) bool {
 		var err error


Suggested change

var err error

`StorageIntegrationV2` supports v2 storage api interfaces as StorageIntegration still supports v1 api Provide a function for extracting flat array of spans from otel traces Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Accept combination of change in the file Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

…ckend Signed-off-by: Ebenezer <[email protected]> Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

…2ToModel` Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

yurishkuro · 2024-12-23T19:02:04Z

sorry, I don't follow what recent changes are you making, the code looks the same as before.

ekefan · 2024-12-24T00:12:26Z

sorry, I don't follow what recent changes are you making, the code looks the same as before.

Apologies, I haven't been able to make changes yet. I made a mistake trying to update this branch with upstream/main - I added commits, I didn't author. The recent push helped me fix that mistake so this branch can sync properly.

yurishkuro · 2024-12-25T14:54:18Z

make sure to hit Update Branch button before continuing.

mahadzaryab1 · 2024-12-25T15:19:34Z

storage_v2/v1adapter/ptrace_model.go

+// PTracesSeq2ToModel consumes an otel trace iterator and returns a jaeger model trace.
+//
+// When necessary, it groups chunks of traces into one single trace
+func PTracesSeq2ToModel(seqTrace iter.Seq2[[]ptrace.Traces, error]) ([]*model.Trace, error) {


@yurishkuro can this file go in package jptrace (potentially in translator.go)? I could see this being reused in the future.

It feels to me that it makes more sense in v1adapter than in jptrace - the latter is only meant to deal with ptrace domain objects.

ekefan requested a review from a team as a code owner December 19, 2024 13:41

ekefan requested a review from albertteoh December 19, 2024 13:41

dosubot bot added area/storage v2 labels Dec 19, 2024

ekefan force-pushed the upgrage-storage-integration branch 3 times, most recently from 7b5ae83 to b3a2eb1 Compare December 19, 2024 14:36

yurishkuro added the changelog:ci Change related to continuous integration / testing label Dec 19, 2024

yurishkuro reviewed Dec 19, 2024

View reviewed changes

plugin/storage/integration/integration.go Outdated Show resolved Hide resolved

ekefan force-pushed the upgrage-storage-integration branch from b3a2eb1 to 3908c5a Compare December 20, 2024 10:19

ekefan requested a review from yurishkuro December 21, 2024 22:09

yurishkuro reviewed Dec 21, 2024

View reviewed changes

ekefan force-pushed the upgrage-storage-integration branch 2 times, most recently from 6d7366b to 7ccf5f7 Compare December 23, 2024 17:02

ekefan added 5 commits December 23, 2024 18:05

Add implementation for v2 storage api

1908c95

`StorageIntegrationV2` supports v2 storage api interfaces as StorageIntegration still supports v1 api Provide a function for extracting flat array of spans from otel traces Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Resolve merge conflict

d065f71

Accept combination of change in the file Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Initialize StorageIntegration with v2 TraceReader for each storage ba…

672a6ba

…ckend Signed-off-by: Ebenezer <[email protected]> Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Consume otel trace sequence to produce model traces

eb9b4db

Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

Refactor storage integrationtest methods use v1adapters `PTracesSeq…

1760562

…2ToModel` Signed-off-by: Emmanuel Emonueje Ebenezer <[email protected]>

ekefan force-pushed the upgrage-storage-integration branch from 7ccf5f7 to 1760562 Compare December 23, 2024 17:06

mahadzaryab1 reviewed Dec 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrage storage integration test to storage v2 API #6388

Upgrage storage integration test to storage v2 API #6388

ekefan commented Dec 19, 2024 •

edited

Loading

ekefan commented Dec 19, 2024

yurishkuro commented Dec 19, 2024

yurishkuro Dec 21, 2024 •

edited

Loading

yurishkuro Dec 25, 2024

yurishkuro Dec 21, 2024

yurishkuro Dec 21, 2024

yurishkuro Dec 21, 2024

yurishkuro commented Dec 23, 2024

ekefan commented Dec 24, 2024 •

edited

Loading

yurishkuro commented Dec 25, 2024

mahadzaryab1 Dec 25, 2024

yurishkuro Dec 25, 2024

	// Chunking requirements:
	// - A single ptrace.Traces chunk MUST NOT contain spans from multiple traces.
	// - Large traces MAY be split across multiple, consecutive ptrace.Traces chunks.

Upgrage storage integration test to storage v2 API #6388

Are you sure you want to change the base?

Upgrage storage integration test to storage v2 API #6388

Conversation

ekefan commented Dec 19, 2024 • edited Loading

Which problem is this PR solving?

Description of the changes

How was this change tested?

Checklist

ekefan commented Dec 19, 2024

yurishkuro commented Dec 19, 2024

yurishkuro Dec 21, 2024 • edited Loading

Choose a reason for hiding this comment

yurishkuro Dec 25, 2024

Choose a reason for hiding this comment

yurishkuro Dec 21, 2024

Choose a reason for hiding this comment

yurishkuro Dec 21, 2024

Choose a reason for hiding this comment

yurishkuro Dec 21, 2024

Choose a reason for hiding this comment

yurishkuro commented Dec 23, 2024

ekefan commented Dec 24, 2024 • edited Loading

yurishkuro commented Dec 25, 2024

mahadzaryab1 Dec 25, 2024

Choose a reason for hiding this comment

yurishkuro Dec 25, 2024

Choose a reason for hiding this comment

ekefan commented Dec 19, 2024 •

edited

Loading

yurishkuro Dec 21, 2024 •

edited

Loading

ekefan commented Dec 24, 2024 •

edited

Loading