[FR] Implement building wheels from sdist/tarball artifacts

[webknjaz]

This is an important use case for building platform-specific matrixes of wheels. In this scenario, one would build an sdist in a separate job, save it as an artifact, and then a matrix of subsequent jobs would create wheels using the same artifact. As a bonus, other binary artifacts from external ecosystems, like RPM, normally also get built from sdist so such jobs would also be runnable in parallel.

I suggest supporting the following invocation:

$ python -m build --wheel dist/some-tarball.tar.gz

I think this UX is good. And since the CLI already accepts a directory-type source in that place, pointing at a file would be natural.

Refs:

• main: build binary distributions via sdists by default #304 (comment)
• Build wheel from sdist (optionally?) #257 (comment)

[henryiii]

What would happen if a user did this:

$ python -m build dist/some-tarball.tar.gz

(no --wheel flag) Would this only produce a wheel, since it's already an sdist, would it copy the sdist to the outdir, or would it make an sdist from the sdist? Or would it throw an error (seems a bit harsh for the default?). How about the --sdist flag?

Other than working out details for what happens, I think this sounds great, and I think it will be easy after #304. I'd also like support for this in cibuildwheel, Xref pypa/cibuildwheel#597

[webknjaz]

I think erroring out with "be verbose" is a good idea. Especially since I envision this to be primarily a CI use case where long verbose options are encouraged for maintainability.

[layday]

Reusing the --wheel flag will likely add to the confusion. #304 says that the --wheel flag signifies that the wheel is built from source instead of the source distribution.

[FFY00]

--wheel will build from the specified source, that could be srcdir, or as this issue proposes, that could be a tarball. But I believe in doing that, we would need to improve the help message more than what #304 does.

We have tried to keep the CLI simple, so I'd like to hear what downsides the following approach would have.

python -m tarfile -e $project-$version.tar.gz
python -m build -w $project-$version

So far, I have not been able to come up with anything that would hold any reasonable weight against that.

Also, maybe I am mistaken, but don't RPM build tools extract the source by default.

[webknjaz]

--wheel will build from the specified source, that could be srcdir, or as this issue proposes, that could be a tarball. But I believe in doing that, we would need to improve the help message more than what #304 does.

Yep, that's what I was thinking: both dir (whether a Git checkout or just a plain old folder) and a tarball are technically source. So it'd be natural to accept either as the same CLI arg.

We have tried to keep the CLI simple, so I'd like to hear what downsides the following approach would have.

python -m tarfile -e $project-$version.tar.gz
python -m build -w $project-$version

So far, I have not been able to come up with anything that would hold any reasonable weight against that.

I see where you're coming from and now I'm feeling a little bit conflicted about this. OTOH since our goal is to streamline this as the best practice, it may be best to keep the number of steps low meaning I'd prefer to only run one command (build). It seems like the overhead of having unpacking+copying within build could be minimal.

Also, maybe I am mistaken, but don't RPM build tools extract the source by default.

AFAIK it's not rpm-build but the macros packagers use in the spec files that do the extraction. The spec only lists the sources but how they are processed is not specified. Those source dists often are even downloaded and put in place by external tools. The build picks them up and may unpack or do something else. For example, in the recent spec I've got, there's one main source that gets unpacked but there are also sources (build deps) of projects from PyPI that aren't available in the RPM repos so I pip install those in temporary build location.

[di]

I'm in favor of this! My $0.02 based on trying to do this and realizing the feature didn't exist: I expected python -m build dist/some-tarball.tar.gz to just work.

This would mean that srcdir would need to become something like:

src - source directory or source distribution (defaults to current directory)

that it would probably need to support both .tar.gz and .zip files, and that the existing flags would continue to behave as-is.

[layday]

Should python -m build dist/some-tarball.tar.gz build both an sdist and a wheel or just the wheel? Should a .zip file be considered an sdist or just a zipped up source directory?

[di]

Should python -m build dist/some-tarball.tar.gz build both an sdist and a wheel or just the wheel?

I would expect it to work by unzipping/untarring the file into some temporary directory and then running as usual on that source directory. This would mean that python -m build dist/some-tarball.tar.gz would produce a new sdist (and a wheel).

This might have the effect of rebuilding the sdist (for example, upgrading it to a newer metadata version), and I think that behavior would probably be desirable.

Should a .zip file be considered an sdist or just a zipped up source directory?

Given my comment above, this probably wouldn't make a difference?

[layday]

Yeah, I think that'd be preferable too, that it should function the same way regardless of input. Does anybody wanna take this up?

[FFY00]

I am reluctant of accepting this feature, but if we do I think we should only support .tar.gz sdists as they are required by PEP 517, and are the de facto standard, as per https://packaging.python.org/specifications/source-distribution-format/#source-distribution-file-name.

[layday]

Would we perform any kind of validation then or do we assume that any gzipped tarball is an sdist?

[FFY00]

We would do some validation: check if the name matches the format, and check for PKG-INFO.