[feat][storage] Add SpanKind support for badger

[Manik2708]

Which problem is this PR solving?

• Fixes: Badger storage plugin: query service to support spanKind when retrieve operations for a given service. #1922

Description of the changes

• Queries with span kind will now be supported for Badger

How was this change tested?

• Writing unit tests

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: npm run lint and npm run test

[Manik2708]

I have changed the structure of cache which is leading to these concerns:

1. Will a 3D map be a viable option for production?
2. Cache will never be able to retrieve operations of old data! When kind is not sent by the user, all operations related to new data will be sent. I have a probable solution for this! We might have to introduce boolean which when true will load the cache from old data (old index key) and mark all the span of kind UNSPECIFIED
3. To maintain consistency, we must take the service name from the newly created index, but extracting service name from serviceName+operationName+kind is the challenge! The solution which I have thought is reserving the last 7 places for len(serviceName)+len(operationName)+kind in the new index. This has an issue that we have to limit the length of serviceName and operationName to 999. This way we can get rid of the c.services map also. Removing this map is optional and a matter of discussion because for this we have to decide between storage and iteration, removing this map will lead to extra iterations in GetServices, I also thought of a solution for this:

data = map[string]struct
// Here this struct can be defined as
type struct {
expiryTime uint64
operations map[trace.SpanKind]map[string]uint64
}

Once the correct approach is discussed I will handle some more edge cases and make the e2e tests pass (making GetOperationsMissingSpanKind: false!

[codecov]

Codecov Report

❌ Patch coverage is 95.85492% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.02%. Comparing base (06cc410) to head (e6cadda).
⚠️ Report is 444 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/storage/v1/badger/spanstore/reader.go	93.47%	4 Missing and 2 partials ⚠️
internal/storage/v1/badger/spanstore/kind.go	89.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6376      +/-   ##
==========================================
- Coverage   96.03%   96.02%   -0.01%     
==========================================
  Files         364      365       +1     
  Lines       20690    20823     +133     
==========================================
+ Hits        19870    19996     +126     
- Misses        626      631       +5     
- Partials      194      196       +2     

Flag	Coverage Δ
badger_v1	10.19% <51.81%> (+0.37%)	⬆️
badger_v2	1.86% <0.00%> (-0.03%)	⬇️
cassandra-4.x-v1-manual	14.66% <0.00%> (-0.20%)	⬇️
cassandra-4.x-v2-auto	1.85% <0.00%> (-0.03%)	⬇️
cassandra-4.x-v2-manual	1.85% <0.00%> (-0.03%)	⬇️
cassandra-5.x-v1-manual	14.66% <0.00%> (-0.20%)	⬇️
cassandra-5.x-v2-auto	1.85% <0.00%> (-0.03%)	⬇️
cassandra-5.x-v2-manual	1.85% <0.00%> (-0.03%)	⬇️
elasticsearch-6.x-v1	18.94% <0.00%> (-0.26%)	⬇️
elasticsearch-7.x-v1	19.02% <0.00%> (-0.26%)	⬇️
elasticsearch-8.x-v1	19.19% <0.00%> (-0.26%)	⬇️
elasticsearch-8.x-v2	1.86% <0.00%> (-0.03%)	⬇️
grpc_v1	10.72% <0.00%> (-0.15%)	⬇️
grpc_v2	7.76% <0.00%> (-0.11%)	⬇️
kafka-3.x-v1	9.98% <0.00%> (-0.14%)	⬇️
kafka-3.x-v2	1.86% <0.00%> (-0.03%)	⬇️
memory_v2	1.86% <0.00%> (-0.03%)	⬇️
opensearch-1.x-v1	19.07% <0.00%> (-0.26%)	⬇️
opensearch-2.x-v1	19.07% <0.00%> (-0.26%)	⬇️
opensearch-2.x-v2	1.86% <0.00%> (-0.03%)	⬇️
tailsampling-processor	0.47% <0.00%> (-0.01%)	⬇️
unittests	94.92% <95.85%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Manik2708]

I have changed the structure of cache which is leading to these concerns:

1. Will a 3D map be a viable option for production?
2. Cache will never be able to retrieve operations of old data! When kind is not sent by the user, all operations related to new data will be sent. I have a probable solution for this! We might have to introduce boolean which when true will load the cache from old data (old index key) and mark all the span of kind UNSPECIFIED
3. To maintain consistency, we must take the service name from the newly created index, but extracting service name from serviceName+operationName+kind is the challenge! The solution which I have thought is reserving the last 7 places for len(serviceName)+len(operationName)+kind in the new index. This has an issue that we have to limit the length of serviceName and operationName to 999. This way we can get rid of the c.services map also. Removing this map is optional and a matter of discussion because for this we have to decide between storage and iteration, removing this map will lead to extra iterations in GetServices, I also thought of a solution for this:

data = map[string]struct
// Here this struct can be defined as
type struct {
expiryTime uint64
operations map[trace.SpanKind]map[string]uint64
}

Once the correct approach is discussed I will handle some more edge cases and make the e2e tests pass (making GetOperationsMissingSpanKind: false!

@yurishkuro Please review the approach and problems!

[Manik2708]

@yurishkuro I have added more changes which reduces the iterations in prefill to 1 but it limits the serviceName to length of 999. Please review!

[Manik2708]

I have an idea for old data without using the migration script! We can store the old data in two other data structures in cache (without kind). But then the only question which rises then: What to return when no span kind is given by user? Operations of new data of all kind or operations of old data (kind marked as unspecified) or an addition of both?

[yurishkuro]

What to return when no span kind is given by user?

then we should return all operations regardless of the span kind

[Manik2708]

What to return when no span kind is given by user?

then we should return all operations regardless of the span kind

That means including all spans of old data also (Whose kind is not there in cache)?

[Manik2708]

My current approach is leading to errors in unit test of factory_test.go. The badger is throwing this error infinetly times:

runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1700, retrying
badger 2024/12/23 01:12:11 ERROR: error flushing memtable to disk: error while creating table err: while creating table: /tmp/badger116881967/000002.sst error: open /tmp/badger116881967/000002.sst: no such file or directory
unable to open: /tmp/badger116881967/000002.sst
github.com/dgraph-io/ristretto/v2/z.OpenMmapFile

This is probably because f.Close is closed before the completion of prefill. That implies creation of new index for old data is slow. Hence I think we have only one way, if we want to skip even auto migration and that is using this function:

func getSpanKind(txn *badger.Txn, service string, timestampAndTraceId string) model.SpanKind {
	for i := 0; i < 6; i++ {
		value := service + model.SpanKindKey + model.SpanKind(i).String()
		valueBytes := []byte(value)
		operationKey := make([]byte, 1+len(valueBytes)+8+sizeOfTraceID)
		operationKey[0] = tagIndexKey
		copy(operationKey[1:], valueBytes)
		copy(operationKey[1+len(valueBytes):], timestampAndTraceId)
		_, err := txn.Get(operationKey)
		if err == nil {
			return model.SpanKind(i)
		}
	}
	return model.SpanKindUnspecified
}

The only problem is that, during prefilling 6*NumberOfOperations Get Queries will be called. Please review this approach @yurishkuro and I think we need to discuss about autoCreation of new index or should we skip the creation of any new index and use the function given above?

[Manik2708]

@yurishkuro I finally got rid of migration and now I think its ready for review! Please ignore my previous comments. The current commit has no linkage them!

[Manik2708]

make test is passing locally, should we rerun in CI?

[Manik2708]

@yurishkuro A humble reminder to review this PR!

[Manik2708]

Not sure about this approach. But my aim was to lower the diff!

[Manik2708]

Can't find any use of store in Cache when the responsibility to fill cache is given to reader

[Manik2708]

I am confused about the two options:

1. Versions used
2. Whether it should be the issue which should be linked or pull request, as issue is not talking about this change directly!

[Manik2708]

@yurishkuro Sorry for disturbance, but can you please review this PR and resolve the doubts?

[yurishkuro]

/gemini review

[gemini-code-assist]

Code Review

The pull request introduces SpanKind support for Badger, which is a valuable enhancement. The changes seem well-structured and include a backward compatibility test. However, there are a few areas that could benefit from further review and refinement.

Summary of Findings

Merge Readiness

The pull request appears to be in good shape overall, but I recommend addressing the comments provided below before merging. I am unable to directly approve this pull request, so please ensure that other reviewers also examine the changes and provide their approval before proceeding with the merge.

[Copilot]

PR Overview

This PR adds support for SpanKind in Badger storage by updating index keys, reader/writer logic, the cache structure, and integration tests.

• Introduces new type mappings and conversion functions for SpanKind.
• Updates factory, reader, writer, and cache layers to create and query indexes that incorporate span kind.
• Adds and adjusts unit and integration tests to verify backward compatibility and new functionality.

Reviewed Changes

File	Description
internal/storage/v1/badger/spanstore/backward_compatibility_test.go	Adds tests to ensure backward compatibility after the index changes.
internal/storage/v1/badger/spanstore/kind.go	Introduces new type and mapping functions for span kind conversion.
internal/storage/v1/badger/factory.go	Registers a new feature gate and updates cache initialization and reader construction.
internal/storage/v1/badger/spanstore/reader.go	Updates TraceReader to accept a dual lookup flag and prefill operations by span kind.
internal/storage/v1/badger/spanstore/rw_internal_test.go	Adjusts tests to use the new TraceReader and CacheStore signatures.
internal/storage/v1/badger/spanstore/writer.go	Modifies index key creation to incorporate SpanKind for operations.
internal/storage/v1/badger/spanstore/cache.go	Updates cache structure for operations to be keyed by span kind.
internal/storage/integration/badgerstore_test.go, cmd/jaeger/internal/integration/badger_test.go	Remove legacy flags and update integration tests to reflect SpanKind support.

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

internal/storage/v1/badger/spanstore/reader.go:60

• [nitpick] Consider renaming 'dualLookUp' to 'dualLookup' to follow common naming conventions.

dualLookUp bool

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. You may re-open it if you need more time.

[graphite-app]

There's a typo in the feature gate stage description: enabed should be enabled.

Suggested change

	featuregate.StageBeta, // enabed by default
	featuregate.StageBeta, // enabled by default

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. You may re-open it if you need more time.