ZStd JNI vs Aircompressor pure java performance question

[nknize]

OpenSearch is evaluating the pure java implementation of Zstd using AirCompressor. I stumbled on a thoughtful comment here which certainly aligns to the reasons we avoid jni code as "top level" modules or plugins on OpenSearch core (we have similar reasons on Lucene). We realize that comment is now five years old, and so @reta ran benchmarks that seem to show significant performance differences between the pure java implementation (hotspot has gotten better of course) and jni. Do these number look valid to folks on this project? What are the potential pitfalls to running those benchmarks that we need to be aware of? Are there certain config conditions that should be followed to squeeze better performance?

Thanks in advance for any assistance that can be provided.

[reta]

For context, I run the io.airlift.compress.benchmark.CompressionBenchmark (i7-10750H × 12, 64Gb, Linux/Ubuntu, JDK 17.0.8), no modifications, selectively picked the airlift_zstd / zstd_jni since this is what we've been looking for, thank you.

[nknize]

Quick ping check in on this... any maintainers here able to give some guidance?

[tianjiqx]

Perhaps I have the same test results and found that zstd_jni actually performs better.
The same implementation of Lz4 also results in better performance for JNI.

Jdk: 17
Os: ubuntu22.04
CPU: AMD ® Ryzen 7 7840hs

[raunaqmorarka]

airliftcompressor now uses native zstd through the Foreign Function Interface recently introduced by new Java versions.
This issue is obsolete now.
FWIW I also observed faster performance with JNI over the previous pure Java implementation a couple of years ago.
cc: @wendigo @dain