Use LZMA compression for transmission of binary packages

[annulen]

Gzip is not really efficient for compressing large amounts of binary data. Also, LZMA has special filters improving compression ratio for executable code. You could use 7z or tar.xz for transmission of packages

[memsharded]

Thanks for telling. I think this might be a duplicate to: #52

We never worked on this feature because a preliminary attempt showed that this will be very complicated from the operations point of view: creating the conan application for PyPi, creating the installers that bundle it for different systems (Win installer, deb, archlinux, OSX homebrew...), as the mainstream python version is still Python2.7, that do not include such functionality.

However, this would be a very nice to have functionality, and very decoupled, not necessary to know about the very internals of conan, if someone knows how to do it and is able to do the work, it will be very welcome.

[annulen]

This is not a duplicate of #52 as that issue is talking about use of 7z in conanfile code, though it seems to be a necessary prerequisite.

[memsharded]

Yes, you are right. Basically the underlying problem is the same, once provided support for LZMA in conan, such issue #52 would be trivial.

[tru]

@memsharded would you be open to having some kind of opt-in to use lzma for packages instead of gzip? lzma makes a LOT of sense for the package storage and transfer, and in our case we can guarantee that all clients are using python3. Would a environment or conan.conf option for uploading .xz packages be something you would accept if we created such a patch?

[lasote]

The problem is maintaining the installers and pypi package, the distinction between py2 and py3 will be a pain. Also managing different compressed files on the server side could be very hard to control if the users start to mix them in the public repositories. It's possible but it looks like a nightmare for us.

[tru]

I see that - I am not really that worried about the public repos (selfish I know), but in our closed ecosystem this would still be a huge win for us.

[memsharded]

I wouldn't oppose to have a branch with it, see how it looks like, and think if there is some way to guarantee 100% that it won't be possible to use it in public repos.

[lasote]

I think we should probably wait for the definitive dropping of python 2 support. But maybe we should start implementing in the next Conan release a kindly error when a conan_package.7z is found to be ready to the future when (probably Conan 2.x) starts to upload 7z packages to public repositories. Then past versions from 1.6 could raise nicely saying: "The package xxx uses 7z compression and your conan version doesn't support it, please upgrade to the latest".
@memsharded @danimtb @markgalpin WDYT?