Support for numpy 2

[JupiterPucciarelli]

It has been a few months since release 1.13.2 which pinned numpy to <2. Is there any chance of updating to support numpy 2?

[MartinBubel]

Hi @JupiterPucciarelli
sorry for the late response. I dont have the time to keep track of gpy maintenance this year, but i guess that whole numpy 2.0 update may be a problem for more users.
if thats a top priority for you, I can get that done quite soon. just respond if this is still an issue for you.

[JupiterPucciarelli]

Hi @JupiterPucciarelli sorry for the late response. I dont have the time to keep track of gpy maintenance this year, but i guess that whole numpy 2.0 update may be a problem for more users. if thats a top priority for you, I can get that done quite soon. just respond if this is still an issue for you.

Yes, it is still an issue. Thank you for responding and looking into this.

[n3011]

We are also facing the same issue, it would be great to have a new release with numpy > 2.0 support.

[MartinBubel]

Thanks @n3011 for expressing support for this - it motivates!

[MartinBubel]

We have a PR now. I'll see if all of the github actions pass.
If they do, I'll move forward to a new minor release.

I naively lifted the upper limit on numpy and updated that of scipy.
However, the way we integrate the cython extensions into the GPy lib is highly sensitive w.r.t. the numpy and scipy combination and it is easy to screw it all up.

@n3011 @JupiterPucciarelli it would be great if you could test the update by using #1113 as a git-dependency or path-dependency if that is possible for you (it may be complicated depending on your setup or environment).

But I still have to do some updates to #1113 before testing it makes sense for you. I'll tag you once I have something to test (probably as early as tomorrow).

I don't want to release a version of GPy that is unstable just to fix it for some. However, I could think about releasing some special version in that case but let's see if this can be resolved more properly.

[MartinBubel]

I think I have something to test... @n3011 @JupiterPucciarelli could you please checkout the latest commit from #1113 and see if it works for you?

I'm curious if we can prevent a major maintenance - which becomes more and more overdue 🙈

I have had some issues getting the testing workflow running as it seems like with the current implementation of some of the tests, pytest>=8 won't work - which took me a while to identify...

[JupiterPucciarelli]

I use gpy via a dependency of elfi so I cannot test my pipelines with a dev release of gpy (afaik). I'll wait to see what @n3011 can do before figuring out a way to help.

[MartinBubel]

Little update from my side: I have to update the paramz dependency to also comply with numpy>2.

[BwL1289]

Chiming in here - excited to see this get merged as well.

[BwL1289]

@MartinBubel FWIW, I just tried compiling from commit 5def1c7110ec45b41008fd5e1d61c59435871897 and seeing the following.

It originates due to accessing PyArray_Descr.subarray, which has been deprecated for a long time and removed in NumPy 1.24+.

clang -cc1 version 20.0.0git based upon LLVM 20.0.0git default target aarch64-unknown-linux-gnu
  ignoring nonexistent directory "/include"
  ignoring duplicate directory "/usr/local/include"
    as it is a non-system directory that duplicates a system directory
  #include "..." search starts here:
  #include <...> search starts here:
   /usr/local/cuda/include
   /usr/local/lib/python3.12/site-packages/numpy/_core/include
   .
   /usr/local/include/python3.12
   /opt/llvm_toolchain/lib/clang/20/include
   /usr/local/include
   /usr/include
  End of search list.
  In file included from GPy/kern/src/stationary_cython.c:1260:
  In file included from /usr/local/lib/python3.12/site-packages/numpy/_core/include/numpy/arrayobject.h:5:
  In file included from /usr/local/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12:
  In file included from /usr/local/lib/python3.12/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1909:
  /usr/local/lib/python3.12/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with "          "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
     17 | #warning "Using deprecated NumPy API, disable it with " \
        |  ^
  GPy/kern/src/stationary_cython.c:18565:41: error: no member named 'subarray' in 'struct _PyArray_Descr'
   18565 |     __Pyx_INCREF(((PyObject*)__pyx_v_d->subarray->shape));
         |                              ~~~~~~~~~  ^
  GPy/kern/src/stationary_cython.c:2117:37: note: expanded from macro '__Pyx_INCREF'
   2117 |   #define __Pyx_INCREF(r) Py_INCREF(r)
        |                                     ^
  /usr/local/include/python3.12/object.h:661:50: note: expanded from macro 'Py_INCREF'
    661 | #  define Py_INCREF(op) Py_INCREF(_PyObject_CAST(op))
        |                                                  ^~
  /usr/local/include/python3.12/object.h:194:49: note: expanded from macro '_PyObject_CAST'
    194 | #define _PyObject_CAST(op) _Py_CAST(PyObject*, (op))
        |                                                 ^~
  /usr/local/include/python3.12/pyport.h:24:38: note: expanded from macro '_Py_CAST'
     24 | #define _Py_CAST(type, expr) ((type)(expr))
        |                                      ^~~~
  GPy/kern/src/stationary_cython.c:18566:38: error: no member named 'subarray' in 'struct _PyArray_Descr'
   18566 |     __pyx_r = ((PyObject*)__pyx_v_d->subarray->shape);
         |                           ~~~~~~~~~  ^
  1 warning and 2 errors generated.
  error: command '/opt/llvm_toolchain/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/local/bin/python3.12 -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/tmp/eugo/GPy-5def1c7110ec45b41008fd5e1d61c59435871897/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /tmp/pip-wheel-fwihxvnn
  cwd: /tmp/eugo/GPy-5def1c7110ec45b41008fd5e1d61c59435871897/
  Building wheel for GPy (setup.py) ... error
  ERROR: Failed building wheel for GPy
  Running setup.py clean for GPy
  Running command python setup.py clean
  /usr/local/lib/python3.12/site-packages/setuptools/__init__.py:94: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
  !!

          ********************************************************************************
          Requirements should be satisfied by a PEP 517 installer.
          If you are using pip, you can try `pip install --use-pep517`.
          ********************************************************************************

  !!
    dist.fetch_build_eggs(dist.setup_requires)
  /usr/local/lib/python3.12/site-packages/setuptools/_distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_suite'
    warnings.warn(msg)
  running clean
  removing 'build/temp.linux-aarch64-cpython-312' (and everything under it)
  removing 'build/lib.linux-aarch64-cpython-312' (and everything under it)
  'build/bdist.linux-aarch64' does not exist -- can't clean it
  'build/scripts-3.12' does not exist -- can't clean it
  removing 'build'

  GPy: User configuration file at location /root/.config/GPy/user.cfg
Failed to build GPy
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (GPy)

[MartinBubel]

That's the sort of stuff that keeps complicating bug fixes for the last two years...

Sorry for the silence here, it's been busy weeks lately.
I should be able to find some time and get this done over the next couple of days.

[casenave]

Just mentioning that this matter is important for me and my colleagues. Thanks a lot for your efforts

[MartinBubel]

Thanks for expressing interest in this - it helps.

However, bad news so far.

I've been digging into the paramz issues, which are the main resaon for the numpy>=2 incompatibitliy here and after spending much more time on it that I initially wanted, I feel like this is going to be really difficult to fix.

I've started sods/paramz#48, which fixes all numpy deprecations except for one, the .data accessor.
This is quite a big deal, unfortunately, because it seems like paramz was making use of a concept that numpy deemed unsafe over the years which led to the deprecation of the .data accessor without a clear replacement.
It seems like numpy does not want to support swapping the memory addresses pointers of arrays - which is exactly what's done in paramz atm.

I could not yet find a workaround, fix or re-write to it and I'm afraid that this may take some time.
I'm currently unable to put in much time, especially not contiguous time...

If anyone has time and experience or even joy in these things, feel free to express your interest and join :)

Best, Martin