Move metadata to pyproject.toml #9951

cdce8p · 2024-11-17T22:59:33Z

What do these changes do?

The metadata changes:

 ...
-Maintainer: aiohttp team <[email protected]>
-Maintainer-email: [email protected]
+Maintainer-email: aiohttp team <[email protected]>
 License: Apache-2.0 AND MIT
-Home-page: https://github.com/aio-libs/aiohttp
+Project-URL: Homepage, https://github.com/aio-libs/aiohttp
 ...

Open questions

Should this PR be backported? It's possible, might just require some manual work due to merge conflicts.

codecov · 2024-11-17T23:05:51Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.76%. Comparing base (8afdc4d) to head (a8a178b).

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #9951   +/-   ##
=======================================
  Coverage   98.76%   98.76%           
=======================================
  Files         129      129           
  Lines       43374    43374           
  Branches     2323     2323           
=======================================
  Hits        42837    42837           
  Misses        383      383           
  Partials      154      154

Flag	Coverage Δ
CI-GHA	`98.64% <ø> (ø)`
OS-Linux	`98.38% <ø> (ø)`
OS-Windows	`96.80% <ø> (ø)`
OS-macOS	`97.68% <ø> (+<0.01%)`	⬆️
Py-3.10.11	`97.33% <ø> (ø)`
Py-3.10.18	`97.72% <ø> (-0.01%)`	⬇️
Py-3.11.13	`97.92% <ø> (+<0.01%)`	⬆️
Py-3.11.9	`97.52% <ø> (ø)`
Py-3.12.10	`97.62% <ø> (-0.01%)`	⬇️
Py-3.12.11	`98.02% <ø> (ø)`
Py-3.13.3	`98.28% <ø> (ø)`
Py-3.9.13	`97.21% <ø> (ø)`
Py-3.9.23	`97.61% <ø> (ø)`
Py-pypy7.3.16	`89.72% <ø> (+2.99%)`	⬆️
VM-macos	`97.68% <ø> (+<0.01%)`	⬆️
VM-ubuntu	`98.38% <ø> (ø)`
VM-windows	`96.80% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

codspeed-hq · 2024-11-17T23:05:51Z

CodSpeed Performance Report

Merging #9951 will not alter performance

_{Comparing cdce8p:move-metadata (a8a178b) with master (8afdc4d)}

Summary

✅ 59 untouched benchmarks

webknjaz · 2024-11-17T23:20:59Z

Let's not rush this. Keeping a dedicated config is a conscious choice. Also, PRs must not reformat half a project along with changing the packaging configs.

FWIW I'd rather keep it as is for now. Especially, since I was going to sync the configs across the projects and this is going to cause a lot of headache since it conflicts with how things are now.

I'm 👎 on this migration right now. It'd be interesting to gather the current dependency limitations in the downstreams to make an informed choice. Additionally, the move would have to happen in the dependencies first.

cdce8p · 2024-11-17T23:30:42Z

Let's not rush this. Keeping a dedicated config is a conscious choice. Also, PRs must not reformat half a project along with changing the packaging configs.

That's actually a bug with the current black config. Since it can't detect the lowest supported version, it assumes py33 and doesn't use the correct style. In f718fa1 I've pinned the lowest version which doesn't cause any reformatting.

FWIW I'd rather keep it as is for now. Especially, since I was going to sync the configs across the projects and this is going to cause a lot of headache since it conflicts with how things are now.

I'm 👎 on this migration right now. It'd be interesting to gather the current dependency limitations in the downstreams to make an informed choice. Additionally, the move would have to happen in the dependencies first.

I'm probably missing something here. This PR doesn't affect any downstream projects. The project metadata change is minimal and spec compliant.

Tbh I just thought it might be a good time to move the static metadata after I learned that it can work fine with setup.py. It will probably still take a while (couple of years) but I expect setuptools to eventually deprecate the setup.cfg config.

There is no strict need to do that now, so if you're opposed feel free to close the PR.

asvetlov

I personally don't care where the project metadata sits until it works and I personally shouldn't spend additional time on the infrastructure maintenance.

asvetlov · 2024-11-18T21:13:24Z

That's actually a bug with the current black config.

@cdce8p could you please extract a fix for this particular problem into a separate PR? A small patch has a great chance to be reviewed and approved quickly.

asvetlov · 2024-11-18T21:14:48Z

Ooh, sorry. I've pushed the wrong button and closed the PR. Please accept my apologizes.

cdce8p · 2024-11-18T21:25:42Z

That's actually a bug with the current black config.

@cdce8p could you please extract a fix for this particular problem into a separate PR? A small patch has a great chance to be reviewed and approved quickly.

@asvetlov See #9962. It looks bigger than it actually is.

asvetlov · 2024-11-19T10:30:20Z

Thanks, @cdce8p
The PR is merged and back ported

cdce8p · 2025-04-11T01:23:39Z

Rebased the PR. Also moved the dependencies and optional-dependencies back to setup.cfg to purely focus on the static metadata. Please let me know if you're interested in moving this forward. If not or not at this time, feel free to close it.

Dreamsorcerer · 2025-04-17T20:17:49Z

setup.cfg

-# https://setuptools.readthedocs.io/en/latest/setuptools.html#setting-the-zip-safe-flag
-zip_safe = False
-include_package_data = True
-
 install_requires =


I would have thought the dependencies would go in the same file as the rest of the metadata..?

Yes they can be moved as well. Just the current tooling expects them to be in setup.cfg. Since this PR stalled for so long I figured I'd revert that part for now. See my last commit. ec90225

I'd be happy to re-include it here or do it in a followup if there is interest.

Right. I'll defer to @webknjaz here to decide how it should go.

It should move to pyproject.toml. The tooling should be updated.

asvetlov · 2025-04-29T11:43:14Z

pyproject.toml

 build-backend = "setuptools.build_meta"

+[project]


Does dependabot understand this format?
I know that dependabot for poetry still requires the old [tool.poetry] section instead of the recommended [project], I wonder if the setuptools-based check is different.

Does dependabot understand this format?

Yes, that's a poetry only bug in dependabot.

I don't think dependabot updates abstract deps anyway. Renovate probably would. Though, in general, this shouldn't be updated automatically. It's not a lock file and mustn't pin things or contain top boundaries in general. This is something that pip-tools constraints would solve, as they'd lock envs w/o the dangers of causing problems in dependency resolution for the end users.

As mentioned earlier, I reverted the part that moved the dependencies to only focus on the static metadata here.

Ah, I didn't notice that you marked those as dynamic.

But still, there's no need to separate those anyway. So let's get them back into this config.

webknjaz · 2025-04-29T12:11:42Z

Rebased the PR. Also moved the dependencies and optional-dependencies back to setup.cfg to purely focus on the static metadata. Please let me know if you're interested in moving this forward. If not or not at this time, feel free to close it.

Not sure if this would work well with setuptools. I think when it detects PEP 621 mode, it would only source the spec-compliant bits from pyproject.toml. There might be some backend configs that would be sourced from other places, but anything that's not listed as dynamic explicitly is likely to be disregarded or otherwise cause breakage. So with this move we'll need to migrate things atomically.

webknjaz · 2025-04-29T12:13:47Z

pyproject.toml

+version = {attr = "aiohttp.__version__"}
+
+[tool.setuptools.packages.find]
+include = ["aiohttp*"]


This will probably happen automatically.

webknjaz · 2025-04-29T12:15:57Z

setup.cfg

-  aiohttp
-  aiohttp._websocket
-# https://setuptools.readthedocs.io/en/latest/setuptools.html#setting-the-zip-safe-flag
-zip_safe = False


This is no longer necessary?

webknjaz · 2025-04-29T12:17:15Z

pyproject.toml

+[tool.setuptools.package-data]
+"*" = ["*.so"]
+
+[tool.setuptools.exclude-package-data]
+"*" = ["*.c", "*.h"]


I'm not entirely sure if these are still needed. Might need some manual checks.

webknjaz · 2025-04-29T12:23:13Z

CHANGES/9951.packaging.rst

@@ -0,0 +1 @@
+Moved static metadata from ``setup.cfg`` to ``pyproject.toml`` -- by :user:`cdce8p`.


Suggested change

Moved static metadata from ``setup.cfg`` to ``pyproject.toml`` -- by :user:`cdce8p`.

Moved core packaging metadata from :file:`setup.cfg` to :file:`pyproject.toml` per :pep:`621`

-- by :user:`cdce8p`.

(it's not really just static)

webknjaz · 2025-04-29T12:28:44Z

pyproject.toml

-requires = [
-    "setuptools >= 46.4.0",
-]
+requires = ["setuptools >= 70.0.0"]


What's the specific feature that requires at least v70?

webknjaz · 2025-04-29T12:30:23Z

I'm 👎 on this migration right now. It'd be interesting to gather the current dependency limitations in the downstreams to make an informed choice. Additionally, the move would have to happen in the dependencies first.

I'm probably missing something here. This PR doesn't affect any downstream projects. The project metadata change is minimal and spec compliant.

Tbh I just thought it might be a good time to move the static metadata after I learned that it can work fine with setup.py. It will probably still take a while (couple of years) but I expect setuptools to eventually deprecate the setup.cfg config.

What I meant back then was that I'd like the move to first happen in projects that are direct deps of aiohttp (like yarl). This is because I've implemented some advanced packaging techniques there (like an in-tree build backend). I'm going to eventually migrate aiohttp to use the same model. So packaging updates would flow from those places where improvements were first implemented.

Upon reflection, thought, I think that this PR doesn't block that so you're fine. Talking about the downstreams, they've been historically behind some upstream packaging standards, but AFAIK they've mostly implemented their own PEP 517 helpers by now. Still, there might be problems where the setuptools version they ship is older than our spec (it'd conflict, forcing them to patch it out on their side). Also, if they patch it out, there's a possibility that it's older than PEP 621 effectively not producing most of the metadata. So we should at least be careful about https://github.com/aio-libs/aiohttp/pull/9951/files#r2066217591.

cdce8p · 2025-04-30T10:58:54Z

That's for the review @webknjaz! I'll address all comments in one to keep it readable. Please let me know if you have any followup questions.

Move dependencies to pyproject.toml

Sounds good. I'd just recommend to do that in a followup PR. Otherwise the backports PRs will likely not apply cleanly. In the end it's just a revert of ec90225

With dependencies and optional-dependencies marked as dynamic, they are still read from setup.cfg as well. Checked the generated METADATA file again just to be sure.

[tool.setuptools.packages.find]
include = ["aiohttp*"]
This will probably happen automatically.

Unfortunately not. Setuptools will throw an error Multiple top-level packages discovered in a flat-layout: ['vendor', 'aiohttp', 'CHANGES']. We can get rid of exclude = ["examples"] though.

zip_safe = False
This is no longer necessary?

Yes. From the setuptools docs

Obsolete - only relevant for pkg_resources, easy_install and setup.py install in the context of eggs (deprecated).

https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html#setuptools-specific-configuration

package-data and exclude-package-data

I'm not entirely sure if these are still needed. Might need some manual checks.

package-data can indeed be removed. The default for pyproject.toml configs is include-package-data = true. We still need exclude-package-data though. Otherwise the wheels would include *.c and *.h files. I plan to do a followup to optimize the sdist / wheel sizes a bit by extending the list further. E.g. we still include the raw cython *.pyx files in the wheels which probably aren't necessary.

Edit: Pushed one more commit to do directly: f06239c. Mainly just to ignore the generated *.hash files. The global-include aiohttp *.pyi instruction was redundant as everything inside aiohttp and .pyi files are included by setuptools automatically.

What's the specific feature that requires at least v70?

Double checked that. It's actually >= 66.1 that's required, so I'd suggest >= 67.0. Previous versions relied on pkgutil.ImpImporter which isn't available on Python 3.12+.

https://setuptools.pypa.io/en/latest/history.html#v66-1-0

Still, there might be problems where the setuptools version they ship is older than our spec

Yes, I've seen a few reports in the light of the PEP 639 adoption. Support for it was only added in v77. I believe most (?) distributors still use an older version with no build isolation so that caused some confusion. That's one of the reasons I haven't updated the metadata here yet. v67.0 should be fine.

cdce8p · 2025-07-22T00:05:05Z

Pushed a new commit to incorporate the license metadata changes from #11226.

@webknjaz Would you mind taking a look at this again? I believe I've addressed all your comments in #9951 (comment). Please let me know if you have any more questions.

webknjaz · 2025-07-24T01:06:02Z

MANIFEST.in

I'd rather not make unrelated changes. The change note talks about moving the metadata definitions, but the patch has some random changes without any justification. Let's not do this.

Could you specify which parts you're referring to? AFICT all changes are related / a result of the metadata move.
E.g. tool.black.target-version can be removed now that project.requires-python is specified.

webknjaz · 2025-07-24T01:06:47Z

setup.cfg

-long_description_content_type = text/x-rst
-maintainer = aiohttp team <[email protected]>
-maintainer_email = [email protected]
-license = Apache-2.0 AND MIT


This SPDX identifier got lost somewhere.

It's included as part of project.license.text.

webknjaz · 2025-07-24T01:08:19Z

pyproject.toml

 ]
 build-backend = "setuptools.build_meta"

+[project]
+name        = "aiohttp"
+license     = {text = "Apache-2.0 AND MIT"}


This is incorrect. SPDX must not be added as license text: https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license

This is only supported in setuptools v77 which most (?) distributors don't support just yet. That's why I've used license.text which is map to the same metadata as the old license field.

See also my longer comment about it: #9951 (comment)

webknjaz · 2025-07-24T01:10:35Z

I believe I've addressed all your comments in #9951 (comment). Please let me know if you have any more questions.

All other threads need to be addressed as well. There's a number of those with zero response.

cdce8p · 2025-07-24T01:16:09Z

I believe I've addressed all your comments in #9951 (comment). Please let me know if you have any more questions.

All other threads need to be addressed as well. There's a number of those with zero response.

I've addressed them in #9951 (comment) as I personally find the Github code comments hard to follow when it's a discussion about one or more related topics.

cdce8p requested review from webknjaz and asvetlov as code owners November 17, 2024 22:59

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Nov 17, 2024

cdce8p requested a review from bdraco November 17, 2024 22:59

cdce8p force-pushed the move-metadata branch from bba81fd to f718fa1 Compare November 17, 2024 23:19

cdce8p force-pushed the move-metadata branch from 0d487cf to 92f533f Compare November 17, 2024 23:33

asvetlov reviewed Nov 18, 2024

View reviewed changes

asvetlov closed this Nov 18, 2024

asvetlov reopened this Nov 18, 2024

cdce8p force-pushed the move-metadata branch from 92f533f to 51bb1d2 Compare November 19, 2024 07:50

cdce8p added 3 commits April 11, 2025 03:05

Move metadata to pyproject.toml

1c4e954

Fix sync-direct-runtime-deps script

5d09ade

Move dependencies back to setup.cfg

ec90225

cdce8p force-pushed the move-metadata branch from 51bb1d2 to ec90225 Compare April 11, 2025 01:18

Dreamsorcerer reviewed Apr 17, 2025

View reviewed changes

Dreamsorcerer assigned webknjaz Apr 17, 2025

asvetlov reviewed Apr 29, 2025

View reviewed changes

webknjaz reviewed Apr 29, 2025

View reviewed changes

webknjaz added backport-3.11 backport-3.12 Trigger automatic backporting to the 3.12 release branch by Patchback robot labels Apr 29, 2025

webknjaz reviewed Apr 29, 2025

View reviewed changes

cdce8p added 3 commits April 30, 2025 11:58

Merge remote-tracking branch 'upstream/master' into move-metadata

ddc14a8

Update news entry

fe24651

Code review

927c3a1

cdce8p added 2 commits April 30, 2025 20:18

Adjust MANIFEST.in and exclude-package-data

f06239c

Merge branch 'master' into move-metadata

72b4d10

Dreamsorcerer removed the backport-3.11 label Jul 10, 2025

cdce8p added 2 commits July 15, 2025 12:38

Merge branch 'master' into move-metadata

a98d26b

Merge branch 'master' into move-metadata

d3153d0

cdce8p added the backport-3.13 Trigger automatic backporting to the 3.13 release branch by Patchback robot label Jul 21, 2025

Incorporate license and license files metadata changes

a8a178b

cdce8p force-pushed the move-metadata branch from 5364f14 to a8a178b Compare July 22, 2025 00:03

webknjaz reviewed Jul 24, 2025

View reviewed changes

		@@ -0,0 +1 @@
		Moved static metadata from ``setup.cfg`` to ``pyproject.toml`` -- by :user:`cdce8p`.

	Moved static metadata from ``setup.cfg`` to ``pyproject.toml`` -- by :user:`cdce8p`.
	Moved core packaging metadata from :file:`setup.cfg` to :file:`pyproject.toml` per :pep:`621`
	-- by :user:`cdce8p`.

Uh oh!

Move metadata to pyproject.toml #9951

Are you sure you want to change the base?

Move metadata to pyproject.toml #9951

Conversation

cdce8p commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Uh oh!

codecov bot commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codspeed-hq bot commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #9951 will not alter performance

Summary

Uh oh!

webknjaz commented Nov 17, 2024

Uh oh!

cdce8p commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asvetlov left a comment

Choose a reason for hiding this comment

Uh oh!

asvetlov commented Nov 18, 2024

Uh oh!

asvetlov commented Nov 18, 2024

Uh oh!

cdce8p commented Nov 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

asvetlov commented Nov 19, 2024

Uh oh!

cdce8p commented Apr 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz commented Apr 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

webknjaz commented Apr 29, 2025

Uh oh!

cdce8p commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cdce8p commented Jul 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cdce8p commented Nov 17, 2024 •

edited

Loading

codecov bot commented Nov 17, 2024 •

edited

Loading

codspeed-hq bot commented Nov 17, 2024 •

edited

Loading

cdce8p commented Nov 17, 2024 •

edited

Loading

cdce8p commented Nov 18, 2024 •

edited

Loading

webknjaz Apr 29, 2025 •

edited

Loading

cdce8p commented Apr 30, 2025 •

edited

Loading

cdce8p Jul 24, 2025 •

edited

Loading