Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(pypi): implement PEP508 compliant marker evaluation #2692

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Mar 23, 2025

This implements the PEP508 compliant marker evaluation in starlark and
removes the need for the Python interpreter when evaluating requirements
files passed to pip.parse. This makes the evaluation faster and allows
us to fix a few known issues (#2690).

In the future the intent is to move the METADATA parsing to pure
starlark so that the RequiresDist could be parsed in starlark at the
macro evaluation or analysis phases. This should make it possible to
more easily solve the design problem that more and more things need to
be passed to whl_library as args to have a robust dependency parsing:

  • [pypi] Support python micro version in env marker based dependency evaluation #2319 needs the full Python version to have correct cross-platform
    compatible METADATA parsing and passing it to Python and back makes
    it difficult/annoying to implement.
  • Parsing the METADATA file requires the precise list of target
    platform or the list of available packages in the requirements.txt.
    This means that without it we cannot trim the dependency tree in the
    whl_library. Doing this at macro loading phase allows us to depend
    on .bzl files in the hub_repository and more effectively pass
    information.

I can remotely see that this could become useful in py_wheel or an building
wheels from sdists as the environment markers may be present in various source
metadata as well. What is more uv.lock file has the env markers as part of
the lock file information, so this might be useful there.

Work towards #2423
Work towards #260
Split from #2629

This implements the PEP508 compliant marker evaluation in starlark and
removes the need for the Python interpreter when evaluating requirements
files passed to `pip.parse`. This makes the evaluation faster and allows
us to fix a few known issues (bazel-contrib#2690).

In the future the intent is to move the `METADATA` parsing to pure
starlark so that the `RequiresDist` could be parsed in starlark at the
macro evaluation or analysis phases. This should make it possible to
more easily solve the design problem that more and more things need to
be passed to `whl_library` as args to have a robust dependency parsing:
* bazel-contrib#2319 needs the full Python version to have correct cross-platform
  compatible METADATA parsing and passing it to `Python` and back makes
  it difficult/annoying to implement.
* Parsing the `METADATA` file requires the precise list of target
  platform or the list of available packages in the `requirements.txt`.
  This means that without it we cannot trim the dependency tree in the
  `whl_library`. Doing this at macro loading phase allows us to depend
  on `.bzl` files in the `hub_repository` and more effectively pass
  information.

Fixes bazel-contrib#2423
@aignas aignas requested review from rickeylev and groodt as code owners March 23, 2025 09:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant