Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python-install fails because of stubgen: #167

Open
dellaert opened this issue Sep 20, 2024 · 10 comments
Open

python-install fails because of stubgen: #167

dellaert opened this issue Sep 20, 2024 · 10 comments
Assignees

Comments

@dellaert
Copy link
Member

I am in a python3 conda environment, on MacOS 14.6.1,

(py310) (base) macbook-pro-8:build dellaert$ python --version
Python 3.10.14

Traceback (most recent call last):
  File "/Users/dellaert/miniforge3/envs/py310/bin/pybind11-stubgen", line 8, in <module>
    sys.exit(main())
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 319, in main
    run(
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 358, in run
    QualifiedName.from_str(module_name), importlib.import_module(module_name)
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
Traceback (most recent call last):
    return _bootstrap._gcd_import(name[level:], package, level)  File "/Users/dellaert/miniforge3/envs/py310/bin/pybind11-stubgen", line 8, in <module>

  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
    sys.exit(main())
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 319, in main
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
    run(
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/site-packages/pybind11_stubgen/__init__.py", line 358, in run
  File "/Users/dellaert/git/github/build/python/gtsam_unstable/__init__.py", line 1, in <module>
    QualifiedName.from_str(module_name), importlib.import_module(module_name)
  File "/Users/dellaert/miniforge3/envs/py310/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
    from .gtsam_unstable import *
ModuleNotFoundError: No module named 'gtsam_unstable.gtsam_unstable'
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/dellaert/git/github/build/python/gtsam/__init__.py", line 9, in <module>
    from gtsam import gtsam, utils
ImportError: cannot import name 'gtsam' from partially initialized module 'gtsam' (most likely due to a circular import) (/Users/dellaert/git/github/build/python/gtsam/__init__.py)
make[3]: *** [python/CMakeFiles/python-unstable-stubs] Error 1
make[3]: *** [python/CMakeFiles/python-stubs] Error 1
make[2]: *** [python/CMakeFiles/python-unstable-stubs.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
make[2]: *** [python/CMakeFiles/python-stubs.dir/all] Error 2
make[1]: *** [python/CMakeFiles/python-install.dir/rule] Error 2
make: *** [python-install] Error 2
@dellaert
Copy link
Member Author

Actually, even after disabling the stub generation, the library installs fine but the same circular import issue remains:

(py310) (base) macbook-pro-8:github dellaert$ /Users/dellaert/miniforge3/envs/py310/bin/python /Users/dellaert/git/github/python/gtsam/tests/test_Pose3.py
Traceback (most recent call last):
  File "/Users/dellaert/git/github/python/gtsam/tests/test_Pose3.py", line 16, in <module>
    from gtsam.utils.test_case import GtsamTestCase
  File "/Users/dellaert/.local/lib/python3.10/site-packages/gtsam/__init__.py", line 9, in <module>
    from gtsam import gtsam, utils
ImportError: cannot import name 'gtsam' from partially initialized module 'gtsam' (most likely due to a circular import) (/Users/dellaert/.local/lib/python3.10/site-packages/gtsam/__init__.py)

Did something about this change in the last PRs? It's weird as it still works in CI. I remember that this circular thing was an issue a long time ago, but it's weird it pops up again now.

@dellaert
Copy link
Member Author

ls /Users/dellaert/.local/lib/python3.10/site-packages/gtsam
Data				examples			imuBias.py			tests
__init__.py			gtsam.cpython-312-darwin.so	noiseModel.py			utils
__pycache__			gtsfm.py			symbol_shorthand.py

Seems the .so file was built for 3.12, but I made sure all cmake versions where 3.10... Does this ring a possible bell, @varunagrawal ?

@varunagrawal
Copy link
Collaborator

Check the GTSAM_PYTHON_VERSION and Cmake variables for the Python version in general. This looks like it is being set to different things in different variables.

This issue has bitten me before as well but it's a simple manual change hence I never tried to fix it.

@dellaert
Copy link
Member Author

They are all 3.10

@dellaert
Copy link
Member Author

And, I did:

cmake -DPYTHON_EXECUTABLE=/Users/dellaert/miniforge3/envs/py310/bin/python ..

And it installs in right directory. But somehow something is generating 3.12 binaries?

@dellaert
Copy link
Member Author

It works in a conda 3.12 environment (again with custom cmake)

@ProfFan
Copy link
Collaborator

ProfFan commented Sep 21, 2024

@dellaert I assume you nuked the entire build folder right? Could you paste your cmake log here?

@dellaert
Copy link
Member Author

Did you give it a try on Ubuntu with 3.10?

PS Should CI not test all python versions we support? It should also test that cmake works out of the box in a conda or virtualenv…

@dellaert
Copy link
Member Author

I could do a quick test with a devcontainer…

@ProfFan
Copy link
Collaborator

ProfFan commented Sep 22, 2024

I can test with a Docker in a sec

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants