conda install wheels with repodata

[danyeaw]

Checklist

• I added a descriptive title
• I searched open issues and couldn't find a duplicate

What?

1. Add core wheel package support to conda-index
2. Add install hook to conda-pypi that can install from wheel repodata

This focuses on the essential functionality needed: auto-detection of wheel packages, metadata extraction, repodata generation, and compatibility with conda-pypi for basic install commands. Future work will include full lockfile support.

Why?

• Reduces package not found for users of conda through access to more packages through wheels as long as you have access to a server hosting wheels
• A potential future state could include Repodata for portions of PyPI
• Compared to the conda pypi install command, users don't need to know which packages are only on PyPI

User impact

Users should be able to run python -m conda_index <path> on directories containing wheel packages, have the tool automatically detect and index them, generate repodata.json with wheel packages, and then use conda install to install those wheels through conda-pypi. The system should handle basic wheel metadata extraction and conversion to conda-compatible format.

For most users, their interface will be conda install <package> and it will include wheels based on that channels priority with other channels.

Goals

• Users can index wheel packages with conda-index
• Generated repodata.json is compatible with conda-pypi
• Basic wheel installation works through conda install using an install hook
• Auto-detection correctly identifies wheel vs conda packages
• Existing conda package indexing continues to work
• Comprehensive test coverage for core functionality

Tasks

No response

This epic is blocked by:

This epic blocks:

[jaimergp]

I have a few questions:

• How does that the CLI look like for a normal day-to-day install operation? conda install -c my-pypi-channel python some-pypi-package?
• Are we going to use regular channel priority to choose where packages should come from?
• We are going to teach conda how to process .whl files so they are extracted and metadata-annotated in conda-meta/*.json. Isn't this equivalent to converting the .whl file to .conda first using whl2conda? The only difference is annotating the origin of that package as a .whl so lockfiles are accurate; then target machines would reprocess the .whl again to a .conda archive on their end.
• Assuming we do have a repodata.json with a packages.wheel section, how are we going to serve the wheels? Forwarding to PyPI? Copying them over to the server? Because if we are already serving wheels, why not store the autogenerated .conda and get rid of the conversion complexity?

Overall it feels that many of these goals are more or less satisfied by https://github.com/jaimergp/conda-pypi-channel, with the difference that we do need to set up an ephemeral channel in a separate process and that's a bit clunky.

That said, food for thought: I wonder if the extra complexity with contributions to so many moving parts (PRs in conda, conda-index, maybe conda-package-handling) as proposed in this ticket is worth it, and whether considering the "magic channel" approach might be easier.

I feel that if teach conda we teach conda how to convert wheels to .conda files on the fly, while keeping the original URL around (it's two lines of code with whl2conda), the server approach is not that clunky.

[jaimergp]

Note aside, regardless the method, we should also cover name mapping somewhere in the pipeline (probably in the conda-index part?).

[danyeaw]

Hi @jaimergp, thanks for the questions and feedback!

How does that the CLI look like for a normal day-to-day install operation? conda install -c my-pypi-channel python some-pypi-package?

Yes

Are we going to use regular channel priority to choose where packages should come from?

Yes

We are going to teach conda how to process .whl files so they are extracted and metadata-annotated in conda-meta/*.json. Isn't this equivalent to converting the .whl file to .conda first using whl2conda? The only difference is annotating the origin of that package as a .whl so lockfiles are accurate; then target machines would reprocess the .whl again to a .conda archive on their end.

Yes, I think we should do whl2conda, with some light processing / annotating to keep the origin for lockfiles.

Assuming we do have a repodata.json with a packages.wheel section, how are we going to serve the wheels? Forwarding to PyPI? Copying them over to the server?

Normally the repodata would go along with the channel where the wheels are, for example, Anaconda would build and host wheels on a wheel simple index. We have also discussed potentially serving portions of PyPI itself as repodata as well for the community.

Because if we are already serving wheels, why not store the autogenerated .conda and get rid of the conversion complexity?

Our vision should be that conda can natively install wheels with no conversion. Hosting .conda files doesn't get us closer to that. In that future state, the conda / conda-forge ecosystem could then stop building conda recipes for pure-python packages entirely and just use the wheels.

I feel that if teach conda we teach conda how to convert wheels to .conda files on the fly, while keeping the original URL around (it's two lines of code with whl2conda), the server approach is not that clunky.

I don't see any advantage to a local server approach if we have the capability to host the repodata on an actual server. Can you explain more why you think that is needed?

[jaimergp]

Yes, I think we should do whl2conda, with some light processing / annotating to keep the origin for lockfiles.

Good. I think we can subclass the main class it so we don't have to fork / vendor too much.

I don't see any advantage to a local server approach if we have the capability to host the repodata on an actual server.

Fair enough. My point is that, if we our solution is to rely on a server serving wheel-metadata-as-repodata and generating the repodata involves processing the wheel, at that point why bother delegating the format conversion to the client.

But yea, client side metadata obtention is a bit slow right now (I wonder if we could leverage all the tricks that uv has to do to achieve such performance), so for now I don't see an alternative.

I do like the general concept of "PyPI exposed as one more (name mapped) conda channel" because it allow us to leverage channel priority to navigate name conflicts. We'll see how we can get there in terms of implementation, but the idea is solid.