Zstdmt count wrong number of cores

[sergey-dryabzhinsky]

On my i7-6850K zstdmt claims there is only 6 cores available. But cpuinfo shows 12 cores, and siblings, command grep -c processor /proc/cpuinfo also shows 12.

To Reproduce
Steps to reproduce the behavior:

1. Install zstd v1.5.7; `apt install zstd
2. Run `zsrdmt with flags -14 -v
3. Scroll up on the log to 'Note: 6 physical core(s) detected'
4. See error

Expected behavior
Number must be equal to number of siblings or logical coree.

Screenshots and charts
If applicable, add screenshots and charts to help explain your problem.

Desktop (please complete the following information):

• OS: Debian 7+
• Version:1.5.7
• Compiler: gcc 4.7.2
• Flags: -O2
• Other relevant hardware specs: 6-core
• Build system: Debian default

[sergey-dryabzhinsky]

I think function in util.c should count "processor" lines in /proc/cpuinfo like grep.

[felixhandte]

Zstd counts hardware cores, not hardware threads. That choice is intentional.

I didn't make that decision, but my understanding of the reasoning is: Zstd is a very memory intensive application, especially at higher compression levels. As you horizontally scale the work by adding more threads, you run into memory bandwidth constraints before you fill all the hardware threads. So there isn't much point in adding more threads and it might actually be worse.

You can always give it a try: try the same workload with -T6 vs -T12 and see what their performance is like.

[sergey-dryabzhinsky]

@felixhandte thanks for info, I'll give it a try

[sergey-dryabzhinsky]

12 threads was faster:

6 threads:

time zstdmt -14 -T6 vzdump-openvz-20001-2025_10_09-15_57_53.tar -vv
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Compressing with 6 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=6 --content-size
zstd: vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst already exists; overwrite (y/n) ? y
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04%   (1803284480 B => 397499354 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 24.11 sec  (cpu load : 537%)
real    0m32.402s
user    2m8.582s
sys     0m1.023s

12 threads:

time zstdmt -14 -T12 vzdump-openvz-20001-2025_10_09-15_57_53.tar -vv
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Compressing with 12 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=12 --content-size
zstd: vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst already exists; overwrite (y/n) ? y
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04%   (1803284480 B => 397499354 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 18.23 sec  (cpu load : 907%)
real    0m22.702s
user    2m44.259s
sys     0m1.257s

I still think there is error in logic in code from util.c - there is resulting siblings count divided.

[sergey-dryabzhinsky]

Need more debug ouput of what values it really found.

[Cyan4973]

As mentioned by @felixhandte , it scales with the number of physical cores, thus disregarding hyper-threading.
This is not an error, it's intentional.

Now we can certainly discuss if this really is a good idea.
The policy was decided about a decade ago, and things can change.

The initial reasoning was as follows: zstd tends to have a very high IPC, something in the vicinity of ~3 instructions per clock. Consequently, since the best cpu of that time features a max theoretical IPC of 4 (which can never be reached), there is no breathing room left for another thread to be active at the same time on the same core.

But there are ways to make that conclusion wrong. To begin with, cpus are becoming wider. I think modern cpus are more in the ~6 IPC range nowadays, perhaps more for Apple. More importantly, the high IPC of zstd is actually only true for decompression and fast compression modes (-1). High compression modes are different: they use more resources, have larger window sizes, so the probability of cache-misses is much higher. Whenever the cpu needs to fetch data from main memory, that's a > 100 cycles penalty, during which the execution units are pretty much idle, and could benefit from multi-threading.

And I guess that's what you are observing at higher compression levels.

If you were to test at level 1, I presume the additional threads from hyperthreading wouldn't provide you much gains.

Anyway, we could revisit what -T0 means. I don't think it's a good idea to paralyze the local system by taking over all the hardware threads available, but we could certainly load them more to extract more performance. Maybe something closer to what lz4 is doing these days.

[sergey-dryabzhinsky]

@Cyan4973 then outputed info about cores leaded to misunderstanding.
And for level 1 there is almost no difference:

6 threads:

time zstdmt -1 -T6 vzdump-openvz-20001-2025_10_09-15_57_53.tar -vv -f
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Compressing with 6 worker threads
--zstd=wlog=19,clog=13,hlog=14,slog=1,mml=7,tlen=0,strat=1
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=6 --content-size
Decompression will require 524288 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 27.14%   (1803284480 B => 489479404 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 1.32 sec  (cpu load : 478%)
real    0m1.422s
user    0m5.440s
sys     0m0.970s

12 threads:

time zstdmt -1 -T12 vzdump-openvz-20001-2025_10_09-15_57_53.tar -vv -f
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Compressing with 12 worker threads
--zstd=wlog=19,clog=13,hlog=14,slog=1,mml=7,tlen=0,strat=1
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=12 --content-size
Decompression will require 524288 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 27.14%   (1803284480 B => 489479404 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 1.32 sec  (cpu load : 503%)
real    0m1.448s
user    0m5.840s
sys     0m0.937s

[sergey-dryabzhinsky]

Then it comes to lxc things going worse: container may got only 4 cores, but cpuinfo still shows 12 siblings / 6 cores and only 4 "online"