Measure average batch byte size and event count#18000

Merged

andsel merged 36 commits intoelastic:mainfrom

andsel:feature/average_metric_byte_size_event_count

Sep 23, 2025

Commits on Sep 19, 2025

Instantiated metric 'pipelines.<pipeline id>.batch.count' to count number of matches to compure the average events and byte per batch
andsel
committed
Exposed just the lifetime value for a flow metric computed using total in events divided by number of batches
andsel
committed
Instantiated metric 'pipelines.<pipeline id>.batch.total_bytes' to sumup all the batch event's byte estimation. Exposed metric 'pipelines.<pipeline id>.batch.byte_size.average.lifetime' containing …
andsel
committed
[Code clean] Removed unused code
andsel
committed
Added RubyNil in the list of class types that estimateMemory can work with
andsel
committed
Refact: used the BYTES constant for each Java scalar type
andsel
committed
Added RubySymbol and RubyFloat to the list of class types that estimateMemory can work with
andsel
committed
[Test] added integration test to verify monitoring API return expected batch subtree metrics
andsel
committed
Introduced 'pipeline.batch.metrics' to manage the way queue reader client can collect batch metrics related to byte size and event count, this commit spread the setting and parameter around doesn't…
andsel
committed
Implemented logic to use in batch metrics creation and exposure
andsel
committed
[Test] Extracted common part of test code in test initializer method
andsel
committed
[Log] added the batch metric sampling value on pipeline start
andsel
committed
Changed event_count metric to use a specific event counter of batch filtered events and not the existing 'events.in'
andsel
committed
[Refact] Extracted batch metric initialisation in separate method
andsel
committed
[Minor] Fixed import and removed dev comments
andsel
committed
[Fix] Used Byte.BYTES constant for Bytes size in bytes
andsel
committed
[Refactoring] Separated batch's metrics code in its own class
andsel
committed
Renamed setting to manage the collection type from pipeline.batch.metrics to pipeline.batch.metrics.sampling_type
andsel
committed
Rename setting to pipeline.batch.metrics.sampling_mode and the value none is renamed as disabled
andsel
committed
Updated estimate memory of a batch to log an error if one of any event fields is of unrecognized type
andsel
committed
Fixes after rebase to main
andsel
committed
[Fix] Avoid to increment batch counts for empty batches
andsel
committed
[Test] Covered QueueReadClientBatchMetrics with tests
andsel
committed
[Cleanup] Inverted condition to improve readability
andsel
committed
[Fix] Avoid to update batch and total event counter if we encouter an error in batch estimation
andsel
committed
Round bytes_size and event_count per batch to integers because float is not meaningfull in such case
andsel
committed
Fixes after reabse to main to pull Settings builder
andsel
committed
Fixed failing API tests after switching to settings builder
andsel
committed
[Refactoring] extracted common code in method to decode string setting into batch metric mode enum
andsel
committed
Fixed constructor of JRubyWrappedAckedQueueExt to retrieve batch metric mode and valorize it, not only in the initialize method
andsel
committed

Commits on Sep 23, 2025