[question] Performance of parallel_download

[acmeijer]

Hi!

I thought I'd share some performance numbers from testing out the new parallel_download feature released in 1.23.0. The below numbers were obtained on a VM running Ubuntu 18.04 with 4 virtual cores and 8GB of RAM.

The test set-up was running conan create on a conanfile.py which ends up installing 120 packages each time. The build step only takes 10-20s, so the time taken is dominated by installing packages. I cleared the local cache between each run.

parallel_download = OFF (commented out):
real	4m31.090s
user	0m51.465s
sys	0m10.899s

parallel_download = 2:
real	3m16.367s
user	0m54.972s
sys	0m14.303s

parallel_download = 4:
real	3m19.584s
user	0m56.525s
sys	0m19.725s

parallel_download = 8:
real	4m24.657s
user	1m5.907s
sys	0m35.089s

parallel_download = 16:
real	4m14.652s
user	1m10.295s
sys	0m40.923s

Some points to note:

• going higher on the parallel_download number seems to at some point make the process slower
• CPU load increases when increasing the parallel_download number but does not necessarily speed up whole process
• It looks like decompression speed decreases a reasonable amount per indivual package when using the parallel_download feature. I know this is quite unscientific but I don't know a good way to measure this. Happy to take suggestions to get useful numbers.

The performance increase is great but I was expecting a bit more, so I was wondering if there might be some easy gain to be had - my question is whether there be something slowing down the decompression when using the parallel_download feature and if so if there is a known method of improving this?

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @acmeijer

Thanks very much for your detailed investigation and report. This is really useful.

It looks like decompression speed decreases a reasonable amount per indivual package when using the parallel_download feature. I know this is quite unscientific but I don't know a good way to measure this. Happy to take suggestions to get useful numbers.

yes, for some reason, it seems that the decompression is the bottleneck in this process, we need to investigate this and think of alternatives.

This is also aligned with what we are seeing with the new download_cache feature, which might also help speeding up this process, but still the file decompression dominates too. Did you manage to try it too? Maybe together with the parallel download or separately.

[acmeijer]

Thanks for getting back to me so quickly - your work is super appreciated from my side and I think you and the team do a fantastic job!

yes, for some reason, it seems that the decompression is the bottleneck in this process, we need to investigate this and think of alternatives.

Good to hear that you are having a similar experience to me.

This is also aligned with what we are seeing with the new download_cache feature, which might also help speeding up this process, but still the file decompression dominates too. Did you manage to try it too? Maybe together with the parallel download or separately.

No I have not tried this but I could see this being pretty useful in CI so we will definitely try it out. However on developers machines am I right in thinking this would not make a difference since there is (typically) only one user home? In the above example, some of the lower-level packages are used by many of the other packages, so a change in one of them can lead to a large number of packages needing to be re-downloaded due to a change in package_id.

[memsharded]

However on developers machines am I right in thinking this would not make a difference since there is (typically) only one user home? In the above example, some of the lower-level packages are used by many of the other packages, so a change in one of them can lead to a large number of packages needing to be re-downloaded due to a change in package_id.

Yes, the download_cache is especially targeted to CI. In developers machine, even if they only typically have 1 cache in their user home they might benefit too if they do clean the cache often while working.

[jarro2783]

I was looking at this on my company's build servers because we have several GB to download and conan install was taking up to 15 minutes. When I looked at the CPU usage I noticed that there was rarely more than about 1 CPU worth of usage even though this was a 32 core machine and parallel_download was 16, so I looked into the untar code and saw that it is all running inside python code. This means that even though there are multiple threads, only one at a time can run. So you actually just have 16 threads fighting for the global interpreter lock which is slower than just having 2.
I suspect that two is optimal because one can be downloading while the other is decompressing, but more just causes contention.

[memsharded]

Well the idea is that not all are untarring at the same time, but some might be downloading and other might be untarring. For the download part, as it will be more bounded by network than by python, it is possible that there could be larger gains with more than 2, depends on the download speed too I guess.

[jarro2783]

For us, and I suspect the op, it is very much bounded by cpu untarring. The files download faster than they decompress, more so when parallel is set too high. I think the whole thing would speed up by an order of magnitude if it could shell out to tar and do the unpacking properly in parallel.

[memsharded]

Hi all.

I am resuming work on this, now that Python 3.14 has built-in zstd compression.

The current PR, derived from this one is in: #19337

We are planning to try to include it in Conan 2.25, in January 2026. It would be very useful to have testing and feedback, so if you want to experiment running from source from my branch in #19337, that would be very useful. Many thanks!

This wouldn't be relying on plugins, but built-in.