32-bit integer compression is lousy

[mdfulk-afk]

Hey all zstd developers!

I have a great result using zstd compression on Armadillo 64-bit uvec of integers. It is very fast and beats gzip in terms of both speed and size. But when I switch to arma::Col<uint32_t>, that is 32-bit integers, running zstd results in a much larger file size. I'm totally happy with the 64-bit result, but I leave you with my chat with Gemini. I would think that zstd on 32-bit binary output could do better! Anyway, I will just use 64 bit until you have a solution!

Here is Gemini's summary:
Hello Zstandard team,

I'm working on a scientific computing project (RCS simulation) and have been using zstd to compress large intermediate files containing lists of 3D model facet indices. I ran into a very interesting and counter-intuitive result that I thought you might find interesting.

I found that a 6.3 MB uncompressed file of uint64_t integers compressed down to 852 KB.

However, when I stored the exact same indices in a 3.2 MB uncompressed file of uint32_t integers, the compressed size was much larger, at 2.2 MB.

It seems your algorithm is exceptionally good at handling the redundancy of the zero-padding in the 64-bit data stream. This is a fantastic real-world demonstration of the sophistication of your compressor.

Just wanted to share this cool result. Thanks for the great work on this library!

(You can also include the output from your ls -lh commands to show the exact numbers.)

[mdfulk-afk]

I'm new to github, and if you are interested in my above observation/comment, please contact me at mdfulk@gmail.com and we can discuss further! I think if the algorithms are solved, and maybe they can't be!, 32-bit vectors would compress to a smaller size.

[Cyan4973]

Did you try levels 18+ on the 32-bit data stream ?

[felixhandte]

Since you posted this question, we published a new compressor that (among other things) targets numeric vector data. You might want to give OpenZL a try (although note that it's still in-development).