Speed up update-python3-dependencies
#7234
Conversation
|
This is a rather conservative PR, there are other places we could be swapping in |
There was a problem hiding this comment.
This looks like a great improvement. Given the relatively privileged place pip-compile or uv occupies in our toolchain—as the producer of hashes for build-time verification—I would question the adoption of uv if it were closed-source. (Or at least I'd want us to do something like verify its output with pip-compile in CI.) But since it's (a) open-source, (b) format-compatible, and (c) as easily swapped out as you've swapped it in here, I don't much care how it's funded. :-)
I'll leave this open for other feedback for twenty-four hours. Barring objections, I'll approve and merge it tomorrow.
cfm
added
blocked
needs/discussion
cfm
removed
blocked
needs/discussion
|
Discussed further out of band: no objections. |
|
I'm gonna make one change, we don't actually need to run this in the container anymore, so it'll be even faster. |
The `make update-python3-dependencies` step is slow for two main reasons: container image layering and pip-compile. Switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the `--allow-unsafe` because uv is entirely independent of setuptools and pip-tools. Because uv is independent of the system Python (it's pure Rust), we can run it on the host and outside the container, so no image rebuilding happens during the updates, it'll just be needed once afterwards. uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed. Overall `make update-python3-dependencies` now takes seconds to run instead of minutes \o/ Refs <freedomofpress/securedrop-tooling#16>.
Status
Ready for review
Description of Changes
The
make update-python3-dependenciesstep is slow for two main reasons: container image layering and pip-compile.Switch to the new uv tool, which reimplements pip-compile in a much faster way. The output is basically the same, except the sorting is smarter (e.g. pytest comes before pytest-cov) and package names are properly normalized. We can also drop the
--allow-unsafebecause uv is entirely independent of setuptools and pip-tools.Because uv is independent of the system Python (it's pure Rust), we can run it on the host and outside the container, so no image rebuilding happens during the updates, it'll just be needed once afterwards.
uv is still quite new to the Python ecosystem, but this allows us to begin using it without any lock-in, it should be trivial to swap back to pip-tools if needed.
Overall
make update-python3-dependenciesnow takes seconds to run instead of minutes \o/Refs freedomofpress/securedrop-tooling#16.
Testing
How should the reviewer test this PR?
make update-python3-dependenciesyourself.Deployment
Any special considerations for deployment? n/a
Checklist
make lint) and tests (make test) pass in the development containermake -C admin test) pass in the admin development container