Can't install package tree-sitter-languages

[mikehearn]

pip install tree-sitter-languages

ERROR: Could not find a version that satisfies the requirement tree-sitter-languages (from versions: none)
ERROR: No matching distribution found for tree-sitter-languages

It's a binary only package that doesn't support source installs, maybe that's the cause? How does one compile such a thing for GraalPy? Perhaps there's a need to update their CI file to also build with GraalPy on each platform but I'm not sure how to do that.

[msimacek]

Yes, binary wheels for CPython only work on CPython. You can typically clone their repo and execute the same build steps as their CI does in a graalpy virtualenv to get a graalpy wheel.

Perhaps there's a need to update their CI file

setup-python supports graalpy, it should be enough to put python-version: graalpy-23.1 in there. But it might not be easy to convince them to add that. They run tests during the build and we don't know if graalpy would pass all of those.

It's not the only package that does binary only releases, it's a quite common problem for machine learning libraries. We have a patch in our pip that lets pip build from arbitrary tarball we specify in our patch metadata. But it also means that we need to provide a patch runs their build scripts within a standard pip build. Which is not always easy.

[mikehearn]

Is there a way to force the check? Looking at the code, it appears that the wheel contains ordinary C shared libraries that don't depend on Python. See here and at the build method above. It looks a lot like a GraalPython build would just be redundant work.

[msimacek]

What you say is true of the languages.so, but there is also core.cpython-311-x86_64-linux-gnu.so and that one is an ordinary python extension linked with CPython.

[msimacek]

To answer your question about the check - pip checks just the filename, so if you download the wheel and rename the it to have graalpy's ABI tag, then it should install. But the core extension won't work.

[msimacek]

I was able to install the package from a github release using:

pip install cython
pip install git+https://github.com/grantjenks/[email protected]

If you can confirm for me that it works for your use case, I can add a patch to GraalPy so that pip install tree-sitter-languages would work the same.

[laserjim]

The steps above worked for me, at least insofar as I was able to install the package on GraalPy 3.10.13 (Oracle GraalVM Native 24.0.2):

pip install cython
pip install git+https://github.com/grantjenks/[email protected]

However, on the GraalPy 3.11.7 (Oracle GraalVM Native 24.1.0-dev), the same steps didn't seem to work, crashing with the following error:

(.venv) root@ubuntu-s-2vcpu-4gb-sfo3-01:~# graalpy -m pip install cython
Looking in indexes: https://pypi.org/simple, https://www.graalvm.org/python/wheels/
Requirement already satisfied: cython in ./.venv/lib/python3.11/site-packages (3.0.10)
(.venv) root@ubuntu-s-2vcpu-4gb-sfo3-01:~# graalpy -m pip install git+https://github.com/grantjenks/[email protected]
Looking in indexes: https://pypi.org/simple, https://www.graalvm.org/python/wheels/
Collecting git+https://github.com/grantjenks/[email protected]
  Cloning https://github.com/grantjenks/py-tree-sitter-languages (to revision v1.10.2) to /tmp/pip-req-build-_igwh4ok
  Running command git clone --filter=blob:none --quiet https://github.com/grantjenks/py-tree-sitter-languages /tmp/pip-req-build-_igwh4ok
  Resolved https://github.com/grantjenks/py-tree-sitter-languages to commit 42f4baffec92848be4937b0cc52b2872201fe322
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "/root/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/root/.venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-gfcpv2k6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-gfcpv2k6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-gfcpv2k6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 502, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-gfcpv2k6/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
        File "<string>", line 5, in <module>
      ModuleNotFoundError: No module named 'Cython'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

[msimacek]

Ah, it seems that newer pip defaults to isolated builds. I was able to install it on master with

pip install wheel Cython
pip install --no-build-isolation git+https://github.com/grantjenks/[email protected]

[laserjim]

That worked! Amazing, thank you! You are so magical, consistently providing solutions to make everything work! Much appreciated!