EPS degrade between 8.14 and 8.15 on memory queue with 16 workers

[kaisecheng]

The ingestion pipeline: 4 filebeats -> Logstash -> ES cloud
filebeat and Logstash run on docker. Logstash has 4 CPU and 4G ram

buildkite benchmark task starts doing daily benchmark since June 22th 2024 at version 8.15.0-SNAPSHOT
Benchmark on ECK is done based on 8.13, where 16 workers has better eps than 8 workers. However, the daily benchmark show worse eps on 8.15 with 16 workers.

The overall EPS has also dropped.

I have done benchmark on memory queue from 8.7.0 to 8.15.0

[kaisecheng]

The benchmark cluster. 8.6.0 to 8.16.0-SNAPSHOT

Another ES server. 8.13.0 to 8.16.0-SNAPSHOT

8.16.0-SNAPSHOT has gained back some EPS

[kaisecheng]

handle non-unicode payload caused EPS degradation in 8.14.0 and Json normalization performance brought EPS back to the baseline