Skip to content

refactor(pypi): translate wheel METADATA parsing to starlark #2629

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

aignas
Copy link
Collaborator

@aignas aignas commented Feb 24, 2025

This PR starts using the newly introduced (#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (whl_installer) to the Starlark
in the whl_library repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the experimental_index_url (#260).

Refactor:

  • I have migrated all of the unit tests from Python to starlark for deps
    generation from METADATA Requires-Dist fields.
  • Read the METADATA file itself in Starlark.

Work towards #260, #2319, #2241
Fixes #2423

aignas added a commit to aignas/rules_python that referenced this pull request Apr 10, 2025
These are just bugfixes to already merged code:
* Fix nested bracket parsing in PEP508 marker parser.
* Fix the sys_platform constants, which I noticed in bazel-contrib#2629 but they got
  also pointed out in bazel-contrib#2766.
* Port some of python tests for requirement parsing and improve the
  implementation. Those tests will be removed in bazel-contrib#2629.
* Move the platform related code to a separate file.

All of the bug fixes have added tests.

Work towards bazel-contrib#2423.
aignas added a commit to aignas/rules_python that referenced this pull request Apr 10, 2025
These are just bugfixes to already merged code:
* Fix nested bracket parsing in PEP508 marker parser.
* Fix the sys_platform constants, which I noticed in bazel-contrib#2629 but they got
  also pointed out in bazel-contrib#2766.
* Port some of python tests for requirement parsing and improve the
  implementation. Those tests will be removed in bazel-contrib#2629.
* Move the platform related code to a separate file.
* Rename `pep508_req.bzl` to `pep508_requirement.bzl` to follow the
  convention.

All of the bug fixes have added tests.

Work towards bazel-contrib#2423.
github-merge-queue bot pushed a commit that referenced this pull request Apr 10, 2025
These are just bugfixes to already merged code:
* Fix nested bracket parsing in PEP508 marker parser.
* Fix the sys_platform constants, which I noticed in #2629 but they got
  also pointed out in #2766.
* Port some of python tests for requirement parsing and improve the
  implementation. Those tests will be removed in #2629.
* Move the platform related code to a separate file.
* Rename `pep508_req.bzl` to `pep508_requirement.bzl` to follow the
  convention.


All of the bug fixes have added tests.

Work towards #2423.
@aignas
Copy link
Collaborator Author

aignas commented Apr 12, 2025

OK, for today that is it.

Current progress:

  • Moved the METADATA file reading/parsing to starlark as well.
  • (TODO) add tests for this newly added code.

Out of scope all of the below:

entry_points.txt reading/parsing in starlark and generating
py_console_script_binary targets is out of scope. I think it would be still
good to generate the entry point python files, but those should be generated by
the py_console_script_binary rule. The good thing is that we only need to
know the script names (i.e. the starlark parser can be minimal).

Then there is the namespace_pkg_dirs generation. We should just straight out
move this to py_library and have a flag implicit_namespace_pkg_dir. I think
the code move to starlark should not be too bad, because we can do the
following in the build context:

  • Inspect the srcs list and deduct from the paths about which __init__.py
    files need to be added. This is only needed if we are not building venvs.
  • ctx.actions.template would be suitable here, because the contents of the
    file is static.

Lastly, unzipping of the wheel in starlark with repository_ctx.
The reason for that is that the unzipping does a few things:

  • Remap paths:
    • purelib: /site-packages
    • platlib: /site-packages
    • headers: /include
    • scripts: /bin
    • data: /data
  • Replace the interpreter in posix scripts
  • Set the installer metadata.

The idea that I have at the moment is to extract it to a sub-folder:
site-packages and then move the headers, scripts and data one folder up
if the rename_paths attribute does not work.

Once we move the pep508_deps calls to the analysis phase and these three
things are done, then we won't need any Python in the repository phase to
extract wheels and we could then create a whl_archive rule that could be
used to just specify the whl dependencies.

EDIT: Reworded what I said above. Actually I don't think it is good to add any code here, so PTAL.

@aignas aignas added this pull request to the merge queue Apr 13, 2025
Merged via the queue into bazel-contrib:main with commit 2cb920c Apr 13, 2025
3 checks passed
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423

fix(toolchain) Override coverage rc
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423

fix(toolchain) Override coverage rc
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
ewianda pushed a commit to ewianda/rules_python that referenced this pull request Apr 14, 2025
…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[pypi] PEP508 Remove the need for Python in env_marker evaluation in hub repos
2 participants