undefined symbol: Py_SET_TYPE

[grisuthedragon]

After installing slycot using pip I got the following error, when I am trying to import it:

Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import slycot
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/env/lib/python3.8/site-packages/slycot/__init__.py", line 16, in <module>
    from .analysis import ab01nd, ab05md, ab05nd, ab07nd, ab08nd, ab08nz
  File "/home/user/env/lib/python3.8/site-packages/slycot/analysis.py", line 22, in <module>
    from . import _wrapper
ImportError: /home/user/env/lib/python3.8/site-packages/slycot/_wrapper.cpython-38-x86_64-linux-gnu.so: undefined symbol: Py_SET_TYPE

I am using Ubuntu 20.04 with Python 3.8.10 and created a new virtual environment for it:

$ virtualenv env 
$ source ./env/bin/activate
$ pip install --no-cache-dir slycot
Collecting slycot
  Downloading slycot-0.5.3.tar.gz (3.0 MB)
     |████████████████████████████████| 3.0 MB 6.3 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting numpy
  Downloading numpy-1.24.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
     |████████████████████████████████| 17.3 MB 100.9 MB/s 
Building wheels for collected packages: slycot
  Building wheel for slycot (PEP 517) ... done
  Created wheel for slycot: filename=slycot-0.5.3-cp38-cp38-linux_x86_64.whl size=1987476 sha256=53f161973476b615f3a07211877910eb89ad85d66428fc31d8b6e980a21a766a
  Stored in directory: /tmp/pip-ephem-wheel-cache-e2j324rb/wheels/f7/b1/ef/08bc53cdb0fe98fa3c4067f1e674ebc4fd32a965544b558450
Successfully built slycot
Installing collected packages: numpy, slycot
Successfully installed numpy-1.24.2 slycot-0.5.3
$ python -c 'import slycot'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/env/lib/python3.8/site-packages/slycot/__init__.py", line 16, in <module>
    from .analysis import ab01nd, ab05md, ab05nd, ab07nd, ab08nd, ab08nz
  File "/home/user/env/lib/python3.8/site-packages/slycot/analysis.py", line 22, in <module>
    from . import _wrapper
ImportError: /home/user/env/lib/python3.8/site-packages/slycot/_wrapper.cpython-38-x86_64-linux-gnu.so: undefined symbol: Py_SET_TYPE

[bnavigator]

Hi @grisuthedragon, thanks for the report.

The error message is interesting and would imply that the compilation step during pip install would have found the wrong python headers. Could you run and post the output of a verbose compilation please?

    pip wheel -v slycot

[roryyorke]

I can reproduce this (also Ubuntu 20.04).

Output of (rm -rf env/ && virtualenv env && source ./env/bin/activate && pip wheel -v slycot) |& tee slycot-build-v.txt attached.

This looks fine to me:

      /tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/cmake/data/bin/cmake /tmp/pip-wheel-rr0bha7n/slycot -G Ninja --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX:PATH=/tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-install -DPYTHON_VERSION_STRING:STRING=3.8.10 -DSKBUILD:INTERNAL=TRUE -DCMAKE_MODULE_PATH:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/skbuild/resources/cmake -DPYTHON_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPYTHON_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPYTHON_LIBRARY:PATH=/usr/lib/x86_64-linux-gnu/libpython3.8.so -DPython_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPython_ROOT_DIR:PATH=/home/rory/tmp/env -DPython_FIND_REGISTRY:STRING=NEVER -DPython_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include -DPython3_EXECUTABLE:PATH=/home/rory/tmp/env/bin/python -DPython3_ROOT_DIR:PATH=/home/rory/tmp/env -DPython3_FIND_REGISTRY:STRING=NEVER -DPython3_INCLUDE_DIR:PATH=/usr/include/python3.8 -DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include -DCMAKE_BUILD_TYPE:STRING=Release

specifically

-DPython_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include

and

-DPython3_NumPy_INCLUDE_DIRS:PATH=/tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include

followed by CMake reporting:

  -- Found NumPy: /tmp/pip-build-env-_qcqcl1f/overlay/lib/python3.8/site-packages/numpy/core/include (found version "1.24.3")
  -- Found F2PY: /tmp/pip-build-env-_qcqcl1f/overlay/bin/f2py3 (found version "1.24.3")

This compiler warning, however, suggests wrong Numpy (and Python?) headers are used (I have numpy installed in base system via apt)

  [684/1236] Building C object slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o
  /tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘int_from_pyobj’:
  /tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:271:14: warning: implicit declaration of function ‘Npy__PyLong_AsInt’ [-Wimplicit-function-declaration]
    271 |         *v = Npy__PyLong_AsInt(obj);
        |              ^~~~~~~~~~~~~~~~~
  /tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘PyInit__wrapper’:
  /tmp/pip-wheel-rr0bha7n/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:26712:5: warning: implicit declaration of function ‘Py_SET_TYPE’; did you mean ‘Py_TYPE’? [-Wimplicit-function-declaration]
  26712 |     Py_SET_TYPE(&PyFortran_Type, &PyType_Type);
        |     ^~~~~~~~~~~
        |     Py_TYPE

slycot-build-v.txt

I'm going to uninstall system numpy and python3-dev, and try again.

[roryyorke]

My comment about Python headers doesn't make sense: we're using the system Python in a venv, so we must use system headers in some way. Having said that I did uninstall those apt packages (and python3.8-dev), and I could successfully build. This makes sense i.t.o. numpy (see also below), but not, I think, python3-dev. I'm a bit confused about this.

I reinstalled python3-dev and numpy via apt, and did a build from git working tree branch from v0.5.4. The command python setup.py build -- -- -v gives this:

[680/1236] /usr/bin/cc -D_wrapper_EXPORTS -I/usr/include/python3.8 -I/home/rory/projects/slycot/gh193-buildbug/venv/lib/python3.8/site-packages/numpy/core/include -I/home/rory/projects/slycot/gh193-buildbug/venv/lib/python3.8/site-packages/numpy/f2py/src -O3 -DNDEBUG -fPIC -MD -MT slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o -MF slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o.d -o slycot/CMakeFiles/_wrapper.dir/_wrappermodule.c.o -c /home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘int_from_pyobj’:
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:271:14: warning: implicit declaration of function ‘Npy__PyLong_AsInt’ [-Wimplicit-function-declaration]
  271 |         *v = Npy__PyLong_AsInt(obj);
      |              ^~~~~~~~~~~~~~~~~
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c: In function ‘PyInit__wrapper’:
/home/rory/src/slycot/_skbuild/linux-x86_64-3.8/cmake-build/slycot/_wrappermodule.c:26712:5: warning: implicit declaration of function ‘Py_SET_TYPE’; did you mean ‘Py_TYPE’? [-Wimplicit-function-declaration]
26712 |     Py_SET_TYPE(&PyFortran_Type, &PyType_Type);
      |     ^~~~~~~~~~~
      |     Py_TYPE

If I look in /usr/include/python3.8:

$ ls -l /usr/include/python3.8|grep numpy
lrwxrwxrwx 1 root root    56 Dec  5  2022 numpy -> ../../lib/python3/dist-packages/numpy/core/include/numpy

My guess is gcc's include ordering gives preference to the system Numpy, since -I/usr/include/python3.8 is first in the arguments relative to the venv Numpy dirs. I think we rely on scikit-build to make that command-line, I don't know if we can affect the order of output.

So, I don't know how to fix this, but a workaround is to uninstall system numpy (apt remove python3-numpy), do the venv setup, then reinstall system numpy.

[roryyorke]

FWIW, the Ubuntu package for development is libpython3.8-dev; removing this prevents a successful build.

[bnavigator]

Nice digging @roryyorke. I am very surprised that this is an issue. We are certainly not the only package compiling stuff with both numpy and python headers, using the scikit-build cmake chain. What do we do different to other packages?

[ppisa]

I have problem to build actual slycot GIT mainline on Debian due this check

ls -l /usr/include/python3.11/numpy
lrwxrwxrwx 1 root root 56 Feb  9  2023 /usr/include/python3.11/numpy -> ../../lib/python3/dist-packages/numpy/core/include/numpy
ls -ld /usr/lib/python3/dist-packages/numpy/core/include/numpy
drwxr-xr-x 1 root root 842 Jul 17 20:40 /usr/lib/python3/dist-packages/numpy/core/include/numpy

This is the same directory mapped by symlink so this seems to be error to prevent build of the package.

When I remove check the Slycot builds and works with python-control and pysimCoder on my computer.