[feature] Support zstd compression for conan cache save

[mrjoel]

What is your suggestion?

I'd like to request that Zstandard (zstd) compression support be added for conan cache {save,restore}. For a random sample of saved packages, comparing the default gzip -9 to zstd -19 gives an approximately 50% space reduction. Presumably this would also provide a core.zstd:compresslevel config option. I'd even go so far as to say that the default for saving should use zstd, with a gzip option for fallback/compatibility when importing with older versions.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Thanks for the feedback @mrjoel

It seems this can be a low hanging fruit, so proposed a draft in #18276:

• reusing the same core.gzip:compresslevel to avoid to create yet another conf
• Opt-in based on name file extension .xz: conan cache save --file=mysave.xz (it can be breaking to change it as default)

If you want to test it from my source branch of the PR, that would be very good feedback.

[mrjoel]

I appreciate the update, however I'll note two things initially:

• that PR adds xz compression, not Zstandard/zstd
• Zstandard supports a different range of compression levels, so I'm not sure it's advisable to reuse the core.gzip:compresslevel config. It could easily be the case that a user sets compression level to 19 for zstd, then tries to generate a tgz and it will cause an error since gzip only supports values 1-9

Zstandard is not currently supported by the built-in tarfile module (python/cpython#81276), but could potentially use xtarfile or just generate a plain tar with tarfile and subsequently compress with pyzst.

PEP in progress, but not yet available - https://peps.python.org/pep-0784/

[memsharded]

Zstandard is not currently supported by the built-in tarfile module (python/cpython#81276), but could potentially use xtarfile or just generate a plain tar with tarfile and subsequently compress with pyzst.

Exactly, that is the issue. We are avoiding as much as possible to have to depend on external libraries, specially libraries that require native extensions, as they tend to be more problematic and less portable, as Conan is installed in a huge variety of platforms and systems. So we would not be planning to add an external dependency such as pyzst.
And even if the zstd gets into the official Python, Conan supports back to Python 3.6 (yes, even if deprecated and EOL, but there are users stuck in it, and we are trying to support them as much as possible), so it will be complicated to have mainstream usage, should be optional, etc.

I am willing to leave the xz compression for this feature, in theory it can have advantages, do you think it would be an improvement that you would use? Or in that case you would prefer to stay with gz compression?

Zstandard supports a different range of compression levels, so I'm not sure it's advisable to reuse the core.gzip:compresslevel

Yes, I knew about the different range, but as this conf was only needed for the conan cache save command, it shouldn't affect other compressions. But I agree it is possible that users might not realize this, put it in the profile, and then have problems, so it will be better to change it, in the case the .xz compression moves forward, thanks very much for pointing this issue!

[mrjoel]

do you think it would be an improvement that you would use? Or in that case you would prefer to stay with gz compression?

I'll test it with our archives and let you know, but past experience has shown that xz is very CPU intensive and slow at decompressing (not to mention compressing, that that impact is more isolated) so it may not be worth it to add at this phase.

If anything it's probably be preferred to just support writing an uncompressed tar based on detected extension, and then we could zstd it after the fact. That would have the huge disadvantage of not being directly restorable though.

[memsharded]

If anything it's probably be preferred to just support writing an uncompressed tar based on detected extension, and then we could zstd it after the fact. That would have the huge disadvantage of not being directly restorable though.

I guess that you can always tar.gz with compression level=0, then compress with zstd, uncompress on the other side, the restore, but yeah, it sounds a bit too much of a hassle.

[Artalus]

May I propose that Conan only creates a list of files to be archived, and then that list is fed into whatever program the user specifies — similar to how you'd use tar -I 'gzip -9' ... — defaulting to Python native gzip otherwise? Then people could use zstd -19, gz -1, parallel implementations like pzstd or pigz, plain tar, or whatever they prefer.
It would also solve #18185

[memsharded]

May I propose that Conan only creates a list of files to be archived, and then that list is fed into whatever program the user specifies — similar to how you'd use tar -I 'gzip -9' ... — defaulting to Python native gzip otherwise? Then people could use zstd -19, gz -1, parallel implementations like pzstd or pigz, plain tar, or whatever they prefer.
It would also solve #18185

I am afraid this would not be enough, because entries in the DB also need to be done. It is not just zipping and unzipping the files, but updates to the DB are also necessary.

[Artalus]

It is not just zipping and unzipping the files, but updates to the DB are also necessary.

Ah, so no plain list of files then. But still, I assume the TarFile stream could be fed into a subprocess, no? So instead of / in addition to the current implementation of

        with open(tgz_path, "wb") as tgz_handle:
            tgz = gzopen_without_timestamps(name, mode="w", fileobj=tgz_handle,

, there would be something like

        with open(tgz_path, "wb") as tgz_handle:
            with subprocess.Popen(compress_cmd, stdin=subprocess.PIPE, stdout=tgz_handle) as proc:
                with tarfile.open(fileobj=proc.stdin, mode='w|') as tgz:

[memsharded]

Yes, it might be possible to try to define some plugin that could inject any arbitrary compression and decompression routines, provided by users.

[mrjoel]

That could be possible, but as I mentioned in #18255 (comment), just the saving part is only half of the round-trip. If a generated archive is not directly restorable it greatly diminishes the value of alternate compression. The same custom decompression option on restore could be provided, but a user needing to know what it is definitely reduces ergonomics of the solution.