[exporter][batching] Serialized bytes based batching for logs

[sfc-gh-sili]

Description

This PR implements serialized bytes based batching. Right now, the feature is exposed to test only via an additional field in MaxSizeConfig, but we plan to evolve the config API soon (see discussion in #12154)

type MaxSizeConfig struct {
	MaxSizeItems int `mapstructure:"max_size_items"`
	MaxSizeBytes int `mapstructure:"max_size_bytes"`
}

Link to tracking issue

#3262

Testing

BenchmarkSplittingBasedOnItemCountManySmallLogs-10                    	     409	   2822478 ns/op	 3148718 B/op	   71082 allocs/op
BenchmarkSplittingBasedOnByteSizeManySmallLogs-10                     	     412	   2890594 ns/op	 3156103 B/op	   71242 allocs/op
BenchmarkSplittingBasedOnItemCountManyLogsSlightlyAboveLimit-10       	      38	  27579702 ns/op	43531582 B/op	  700471 allocs/op
BenchmarkSplittingBasedOnByteSizeManyLogsSlightlyAboveLimit-10        	      24	  43067219 ns/op	43639889 B/op	  702927 allocs/op
BenchmarkSplittingBasedOnItemCountHugeLogs-10                         	      40	  29722710 ns/op	41922925 B/op	  690300 allocs/op
BenchmarkSplittingBasedOnByteSizeHugeLogs-10                          	      16	  64321198 ns/op	42107144 B/op	  694144 allocs/op

Caveat

We book keep both item count and byte size regardless of batching option, which adds an overhead in both cases.

[codecov]

Codecov Report

Attention: Patch coverage is 91.95402% with 7 lines in your changes missing coverage. Please review.

Project coverage is 92.17%. Comparing base (f024b0d) to head (60f71c9).

Files with missing lines	Patch %	Lines
pdata/plog/pb.go	0.00%	6 Missing ⚠️
exporter/exporterhelper/logs_batch.go	98.03%	0 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (91.95%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12299      +/-   ##
==========================================
- Coverage   92.19%   92.17%   -0.03%     
==========================================
  Files         466      467       +1     
  Lines       25325    25371      +46     
==========================================
+ Hits        23348    23385      +37     
- Misses       1577     1585       +8     
- Partials      400      401       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ms-jcorley]

It still makes me slightly uncomfortable that split will accept a cfg with both MaxSizeItems and MaxSizeBytes set and simply ignore the second. I'd rather this function enforce the expectations that only one is set. But given the context, the change overall seems fine to me.

[dmitryax]

Suggested change

	// DeltaCapacity() returns the delta size of a proto slice when a new item is added.
	// DeltaSize() returns the delta size of a proto slice when a new item is added.

[dmitryax]

Can you please add a link to the Size() function here

[jmacd]

Yes, and you might add that the constant 1 below is specialized for OTLP; it relies on protobuf tags < 16 (which are expressed w/ a single byte of overhead).

[jmacd]

Suggested change

	// currSize := (prevSize + deltaCapacity(proto2.Size()))
	// currSize := (prevSize + sizer.DeltaSize(proto2.Size()))

[jmacd]

	// DeltaSize() returns the delta size when a ResourceLog, ScopeLog or LogRecord is added.

[jmacd]

Yes, and you might add that the constant 1 below is specialized for OTLP; it relies on protobuf tags < 16 (which are expressed w/ a single byte of overhead).