Add initial support for License-Expression (PEP 639)

[cdce8p]

Summary of changes

• Update validate_pyproject to 0.22
  https://github.com/abravalheri/validate-pyproject/blob/v0.22/CHANGELOG.rst
• Initial support for License-Expression core metadata

 [project]
 name = "spam"
 ...
-license = {text = "MIT"}
+license = "MIT"

Missing: The license expression isn't yet validated. The next version of validate_pyproject will likely add that once a new version of packaging has been released. From the PEP:

Build and publishing tools SHOULD check that the License-Expression field contains a valid SPDX expression, including the validity of the particular license identifiers (as defined above).

It should be fine to omit that part for the moment.

Refs: #4629

[abravalheri]

Hi @cdce8p thank you very much for having a look on this.

We probably have to organise so that PEP 639 lands after PEP 685 implementation (ideally we want a couple of days/weeks between the two so that users have time to report potential bugs).

[abravalheri]

Would it be possible to implement this feature without changing the _distutils folder?

In general, we don't modify the contents to that folder directly. Instead, the code is maintained in the pypa/distutils repository and merged back in setuptools periodically, (see also our development guidelines). So changes in the folder have to be submitted first in pypa/distutils and later merged here.

Additionally, there are some people that still use the deprecated SETUPTOOLS_USE_DISTUTILS=stdlib in Python3.9 ... 3.11, so we need to ensure it is also backwards compatible with the stdlib version.

[cdce8p]

Would it be possible to implement this feature without changing the _distutils folder?

Yes. Removed those changes as they weren't necessary to begin with. I added them initially but the better solution was to add license_expression to _DISTUTILS_UNSUPPORTED_METADATA in setuptools/dist.py. This is also fully compatible with SETUPTOOLS_USE_DISTUTILS=stdlib. 390b95f

[cdce8p]

We probably have to organise so that PEP 639 lands after PEP 685 implementation (ideally we want a couple of days/weeks between the two so that users have time to report potential bugs).

Maybe I'm missing something obvious but I'm not quite sure how PEP 685 is related. Would you mind explaining it a bit more?

The way I've designed the PR, it shouldn't depend on anything. It just allows users to specify license = MIT in pyproject.toml which will be used for the License-Expression field. Note, the metadata version is still 2.2 (not 2.4 which is the first to "require" support for PEP 639).

[cdce8p]

Opened #4734 with just the validate-pyproject update. This will make the changes here easier to review.

[abravalheri]

Maybe I'm missing something obvious but I'm not quite sure how PEP 685 is related. Would you mind explaining it a bit more?

The problem is that the metadata version of the core metadata is incremental:

• PEP 685 stablishes 2.2
• PEP 639 stablishes 2.3

So we cannot implement PEP 639 before 685, otherwise metadata validation may fail.

Even if the PR is independent we cannot merge it now because of the nature of the releases of setuptools (we cannot risk PEP 639 support being released before PEP 685 support).

[cdce8p]

So we cannot implement PEP 639 before 685, otherwise metadata validation may fail.

The validation will only happen once we update the metadata version.
https://github.com/pypa/packaging/blob/24.2/src/packaging/metadata.py#L826-L831

If not, a lot of wheel uploads would probably fail as we currently include License-File metadata that's not up to spec. #3596 (Fix: #4728).

What I'm proposing here is simply to add the basic support to setuptools. It won't get validated by PyPI until we're ready to move to 2.4 but then all projects using the "new" syntax for pyproject.license will already be ready to go.

[abravalheri]

For sure the existence of the License-File in the setuptools-produced metadata with version 2.1 is violating the specs. However, this is a well known violation that the Python ecosystem has grown acquainted to since the early prototype implementation of PEP 639, and for better or worse, learned how to deal with.

By accepting only some aspects of PEP 639, we add other kinds of spec violations that some tools may not be prepared to deal with.

So for the sake of reducing the unknown risks, I propose we pipeline the changes you kindly contributed after the implementation of 685. Unfortunately, it means that people cannot start modifying their pyproject.toml immediately, but it feels safer.

[abravalheri]

Sorry, I have been using the wrong PEP number, sorry for the confusion.

PEP 685 is important, but the really big one for us to implement before 639 is PEP 643 (the one with Dynamic).

[cdce8p]

So for the sake of reducing the unknown risks, I propose we pipeline the changes you kindly contributed after the implementation of 685. Unfortunately, it means that people cannot start modifying their pyproject.toml immediately, but it feels safer.

I understand your reasoning. One last suggestion. The last days I've been looking into how hatchling does it as I noticed it was one of the earliest adopters of License-Expression in the metadata. Turns out the packaging update caused upload failures pypa/hatch#1786. I can't reproduce it with PyPI at the moment. However, packaging==24.2 rejects any License-Expression and License-File keys with metadata <2.4. That would be relevant for us too. After the issue report, they removed the PEP 639 keys from the metadata and now include the project.license as part of the License key.

My suggestion

• Remove the License-File keys from the generate metadata as this might cause upload failures once the new packaging version is rolled out to PyPI.
• Keep the license files in the .dist-info folder but move them to the licenses subfolder in according to the spec. Essentially Store license-files in licenses subfolder #4728
• Add support for project.license by back-populating the License key, similar to Back-populate string license fields hatch#1805 Together with Update validate-pyproject to 0.23.0 #4734, this would allow users to use the new syntax while keeping the old metadata format.

What do you think?