Build wheels for free-threading Python (3.15?)

[blankjul]

Hey,

I have just wanted to build the binaries in pymoo for a new release and ran into the following issue shown below.
(see https://github.com/anyoptimization/pymoo/actions/runs/19623799509/job/56188870983#logs)

Can extend your build to include the new version as well? At least with a quick look at it I think that is the issue here.

+ uv venv /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/venv-test-arm64 --python /Library/Frameworks/PythonT.framework/Versions/3.14/bin/python3.14t
  Using CPython 3.14.0+freethreaded interpreter at: /Library/Frameworks/PythonT.framework/Versions/3.14/bin/python3.14t
  Creating virtual environment at: /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/venv-test-arm64
  Activate with: source /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/venv-test-arm64/bin/activate
  + which python
  /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/venv-test-arm64/bin/python
  + /Users/runner/hostedtoolcache/uv/0.9.11/aarch64/uv pip install /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/repaired_wheel/pymoo-0.6.1.6-cp314-cp314t-macosx_11_0_arm64.whl
  Using Python 3.14.0 environment at: /private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/venv-test-arm64
  Resolved 24 packages in 162ms
  Downloading fonttools (2.8MiB)
  Downloading pillow (4.4MiB)
  Downloading scipy (20.4MiB)
  Downloading matplotlib (7.8MiB)
     Building moocore==0.1.9
   Downloaded pillow
   Downloaded fonttools
   Downloaded matplotlib
   Downloaded scipy
    × Failed to build `moocore==0.1.9`
    ├─▶ The build backend returned an error
    ╰─▶ Call to `setuptools.build_meta.build_wheel` failed (exit status: 1)
  
        [stdout]
        Ignoring py_limited_api=True for free-threaded build.
        running bdist_wheel
  
        [stderr]
        /Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/config/pyprojecttoml.py:72:
        _ExperimentalConfiguration: `[tool.distutils]` in `pyproject.toml` is
        still *experimental* and likely to change in future releases.
          config = read_configuration(filepath, True, ignore_option_errors,
        dist)
        Traceback (most recent call last):
          File "<string>", line 11, in <module>
            wheel_filename =
        backend.build_wheel("/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmpperhId",
        {}, None)
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/build_meta.py",
        line 432, in build_wheel
            return _build(['bdist_wheel'])
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/build_meta.py",
        line 423, in _build
            return self._build_with_temp_dir(
                   ~~~~~~~~~~~~~~~~~~~~~~~~~^
                cmd,
                ^^^^
            ...<3 lines>...
                self._arbitrary_args(config_settings),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            )
            ^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/build_meta.py",
        line 404, in _build_with_temp_dir
            self.run_setup()
            ~~~~~~~~~~~~~~^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/build_meta.py",
        line 317, in run_setup
            exec(code, locals())
            ~~~~^^^^^^^^^^^^^^^^
          File "<string>", line 4, in <module>
            sys.path.pop(0)
            ~~~~~~~~~~~~^^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/__init__.py",
        line 115, in setup
            return distutils.core.setup(**attrs)
                   ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/_distutils/core.py",
        line 186, in setup
            return run_commands(dist)
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/_distutils/core.py",
        line 202, in run_commands
            dist.run_commands()
            ~~~~~~~~~~~~~~~~~^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/_distutils/dist.py",
        line 1002, in run_commands
            self.run_command(cmd)
            ~~~~~~~~~~~~~~~~^^^^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/dist.py",
        line 1102, in run_command
            super().run_command(command)
            ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/_distutils/dist.py",
        line 1020, in run_command
            cmd_obj.ensure_finalized()
            ~~~~~~~~~~~~~~~~~~~~~~~~^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/_distutils/cmd.py",
        line 135, in ensure_finalized
            self.finalize_options()
            ~~~~~~~~~~~~~~~~~~~~~^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/command/bdist_wheel.py",
        line 250, in finalize_options
            self._validate_py_limited_api()
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
          File
        "/Users/runner/work/_temp/setup-uv-cache/builds-v0/.tmp9nqxn6/lib/python3.14t/site-packages/setuptools/command/bdist_wheel.py",
        line 285, in _validate_py_limited_api
            raise ValueError(
            ...<4 lines>...
            )
        ValueError: `py_limited_api='cp310'` not supported. `Py_LIMITED_API`
        is currently incompatible with `Py_GIL_DISABLED`. See
        https://github.com/python/cpython/issues/111506.
  
        hint: This usually indicates a problem with the package or the build
        environment.
    help: `moocore` (v0.1.9) was included because `pymoo` (v0.6.1.6) depends
          on `moocore`
                                                              ✕ 2.46s
Error: cibuildwheel: Command ['/Users/runner/hostedtoolcache/uv/0.9.11/aarch64/uv', 'pip', 'install', '/private/var/folders/6c/pzd640_546q6_yfn24r65c_40000gn/T/cibw-run-0n992_zu/cp314t-macosx_arm64/repaired_wheel/pymoo-0.6.1.6-cp314-cp314t-macosx_11_0_arm64.whl'] failed with code 1. 

Error: Process completed with exit code 1.

[MLopez-Ibanez]

The problem is that you are building for Python 3.14t (the free-threading version): https://docs.python.org/3/howto/free-threading-python.html

Currently, I skip building wheels for free-threading python:

moocore/python/pyproject.toml

Line 107 in 89a24a9

"cp314t-*", # Skip free-threaded builds

moocore wheels are compatible with all python versions included 3.14 that are non-free-threading. The free-threading versions are still experimental so each new free-threading python version will need its own wheels.

Moreover, since free-threading is still experimental and not the default, there may be bugs that only the Python developers can fix. I would expect most users of moocore and pymoo to use the default non-free-threaded version of Python 3.14.

I am happy to create version-specific wheels for 3.14t for you to test if you think pymoo should work in free-threading mode, but you may wish to check all your CPython code to make sure you do support free-threading: https://docs.python.org/3/howto/free-threading-extensions.html#freethreading-extensions-howto

Please let me know what you plan to do. I don't want to build for 3.14 free-threading unless you really need it.

[MLopez-Ibanez]

By the way, python/cpython#111506 is about extending the stable ABI to support free-threaded Python. If they do that, then the builds of moocore will work in both classical and free-threaded Python versions, without needing to build new wheels for each Python version.

[blankjul]

actually fair point that it is currently flagged as experimental. I saw you already commented on the corresponding issue in pymoo as well.

let me follow the same build pattern as moocore for now.

[MLopez-Ibanez]

OK, I'll keep this open in case somebody else wants to work on it. I will revisit this again once Python 3.15 is released. Hopefully the ABI will be stable (or even better, free-threading will be the default).

[tpgillam]

There appears to be appetite for free-threaded support btw; see user request here:

• [BUG] version 0.6.1.6 introduced moocore dependecy imcompatible with python 3.14t anyoptimization/pymoo#765

[Deathn0t]

Hello,

My appetite is mostly experimental 😄 I had not found this discussion yet! So no rush for me at least.

I found out with a simple experiments using Python 3.14t that it seems to scale poorly w.r.t. threads for now.
From 1 to 2 threads I get a speed-up significantly less than x2 (x1.45, from 84sec. to 58sec.). Looking at htop in same time clearly shows that the correct corresponding number of cores are busy!

I used the following script if you want to play with that! I have not yet tested the same test without joblib but directly through threading module...

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "joblib",
#   "numpy",
# ]
# ///

# $ uv run -p 3.14t test_joblib.py 1
# N-Jobs: 1
# Duration: 84.40 sec.

# $ uv run -p 3.14t test_joblib.py 2
# N-Jobs: 2
# Duration: 58.01 sec.

# $ uv run -p 3.14t test_joblib.py 5
# N-Jobs: 5
# Duration: 77.45 sec.

# $ uv run -p 3.14t test_joblib.py 10
# N-Jobs: 10
# Duration: 76.93 sec.

import time
import sys
import numpy as np
from joblib import Parallel, delayed


def do_some_work(chunk_size: int, n_repetition: int):
    for _ in range(n_repetition):
        # I was just checking if the random generator would imply some
        # additional mutex overhead...
        # X = np.random.random((chunk_size, chunk_size))
        X = np.ones((chunk_size, chunk_size))
        Y = X @ X
        y = np.sum(Y)
    return y


if __name__ == "__main__":
    chunk_size = 1000
    n_repetition = 1000
    n_tasks = 10
    n_jobs = int(sys.argv[-1])
    t1 = time.monotonic()
    results = Parallel(n_jobs=n_jobs, backend="threading")(
        delayed(do_some_work)(chunk_size, n_repetition) for _ in range(n_tasks)
    )
    d = time.monotonic() - t1
    print(f"N-Jobs: {n_jobs}")
    print(f"Duration: {d:.2f} sec.")

[MLopez-Ibanez]

I would also guess that neither pymoo nor moocore will benefit from free-threaded python because expensive computations are done in C already for speed. It is relatively easy to parallelize the C code using OpenMP if desired.

[tpgillam]

I would also guess that neither pymoo nor moocore will benefit from free-threaded python because expensive computations are done in C already for speed. It is relatively easy to parallelize the C code using OpenMP if desired.

I presume the benefit would be that if users wish to use free-threaded python for other parts of their code, then it'll be convenient if they can use other libraries from within the same interpreter. Even if e.g. moocore or pymoo wouldn't go any faster, other parts of the codebase might.