how to find PythonExtensions

[ofloveandhate]

I'm trying to use scikit-build-core for a C++ project that generates Python bindings, and I'm stuck at an early step. I've tried to follow the tutorial, and I can't even get that to work because I can't get CMake to find FindPythonExtensions.cmake. I'm writing to ask for help making sure this .cmake file is available.

---

I've seen a number of projects online that use your tool, and their CMakeLists.txt files contain the line

find_package(PythonExtensions REQUIRED)

I just cannot get this step to succeed. All I can get is

        By not providing "FindPythonExtensions.cmake" in CMAKE_MODULE_PATH this
        project has asked CMake to find a package configuration file provided by
        "PythonExtensions", but CMake did not find one.
      
        Could not find a package configuration file provided by "PythonExtensions"
        with any of the following names:
      
          PythonExtensionsConfig.cmake
          pythonextensions-config.cmake
      
        Add the installation prefix of "PythonExtensions" to CMAKE_PREFIX_PATH or
        set "PythonExtensions_DIR" to a directory containing one of the above
        files.  If "PythonExtensions" provides a separate development package or
        SDK, be sure it has been installed.

I can see the necessary file FindPythonExtensions.cmake as part of my scikit-build installation at site-packages/skbuild/resources/cmake. But I do not know how to programmatically get the location of this file to add it to my CMAKE_MODULE_PATH. To that end, I've tried running a Python script via EXECUTE_PROCESS a la this forum post, but I cannot get it to succeed for the life of me.

---

Am I supposed to take a copy of FindPythonExtensions.cmake into my repo? Why doesn't scikit-build-core find it when I invoke pip install . to build and install my library?

[henryiii]

PythonExtensions is the old tool that is part of scikit-build (classic) and is based on the deprecated (and partially removed in CMake 3.27+) FindPythonInterp/FindPythonLibs modules in CMake. Scikit-build-core is designed to be used with the modern FindPython, which doesn't need the extra helpers in PythonExtensions.

If you use scikit-build (classic), then that tutorial will work, and PythonExtensions will automatically be available. If you use scikit-build-core (following https://scikit-build-core.readthedocs.io/en/latest/getting_started.html), then you don't need additional modules and can just use what CMake provides.

[ofloveandhate]

Ok, thank you.

Where can I find documentation on the CMake things that scikit-build and scikit-build-core provide? I want to be as independent as possible! My goal is to redo my build system for a C++/Python library using Boost.Python (so I can finally hopefully distribute pre-compiled binaries, a goal that's long challenged me), and I'm working to build a minimal example since the examples don't include such an example. For example, my question next is about linking needed libraries to my python_add_library. I get an error on my next line, where i have

python_add_library(slartibartfast MODULE ${SOURCES} ${HEADERS} WITH_SOABI)  
#ok, cool, i'm making a python library
target_link_libraries(slartibartfast ${Boost_LIBRARIES})  # error

The error I get is

        The keyword signature for target_link_libraries has already been used with
        the target "slartibartfast".  All uses of target_link_libraries with a
        target must be either all-keyword or all-plain.
      
        The uses of the keyword signature are here:
      
         * /opt/homebrew/share/cmake/Modules/FindPython/Support.cmake:4249 (target_link_libraries)

I can get it to work by adding the PRIVATE keyword to the target_link_libraries call, which I knew to do from pybind11's example, not from documentation. How could I have known to use PRIVATE to get around this?

Another thing vexing me is how to get scikit-build-core / CMake / pip to install both a built C++/.so library AND a pure-python library that imports the .so and makes calls into it. I haven't found an example of that yet. Is there one in the documentation for scikit-build-core?

Thanks for your time!

[LecrisUT]

scikit-build-core does not provide any special CMake functions, other than backporting find_python when needed ¹. You can (should) design the build using scikit-build-core as you would a plain CMake project. On top of that, you just add packaging instructions, e.g. install.

Regarding dependencies though, you can bundle them and download them as needed using FetchContent (and I believe boost is actually FetchContent compliant, but not sure if all components), but you need to take care of bundling all the necessary dependencies to either have them all statically linked and included or appropriate RPATH. Usually cibuildwheel can help with such bundling.

Another thing vexing me is how to get scikit-build-core / CMake / pip to install both a built C++/.so library AND a pure-python library that imports the .so and makes calls into it. I haven't found an example of that yet. Is there one in the documentation for scikit-build-core?

This is a rather more complex design question, and it can be handled in various ways depending on what are your requirements. I generally ask the question where do you want to package and support your project (PyPI, conda, spack, distro packages)? Each of these will put some constraints on the design and guide you to some good practices for each environment. Another important aspect is what is your intended user experience after pip install, e.g. do you want the project to try to use some system libraries or always use the bundled libraries?

As for reference projects, I do not believe we have any official ones to recommend, but maybe @henryiii has some good references. My design in spglib that is trying to address some of your issues, but it is slightly blocked by #880. At some point though, I will post a hello-world project derived from it. The main aspect that I tackle there is integrating compatibility for distro-packaging, PyPI distribution and FetchContent compatibility.

Footnotes

1. https://github.com/scikit-build/scikit-build-core/tree/main/src/scikit_build_core/resources/find_python ↩

[ofloveandhate]

Thank you for your thorough reply. I have a working CMake build setup already for my library, but it still feels piecemeal, in that pip installing from the repo isn't connected to compilation (yet). My tool consists of three pieces:

• C++ core library, can be installed without Python bindings
• Boost.Python bindings, depend on core being installed
• Pure Python wrapper around the compiled bindings

Getting all three built and installed currently takes three steps, and they feel disconnected, and it's a barrier to people using my tools. I desperately need it to be pip- or conda-installable.

I certainly would welcome a complete model example for packaging and distribution of a library built using C++ and Python; not just building, but how to get it all the way into the distribution pipeline. My highest goal is pip over conda, but both would be best (together with homebrew). I think galsim seems pretty close to what I'm doing.

I think we're close to the end of this thread, which has now wandered from the title of this thread. To conclude, do you have advice or resources for closing the gap between a public repo on github and installation via pip, for such a library as I'm working with? That's what brought me to scikit-build in the first place. I keep finding examples that do one step or another, but nothing that's end-to-end (and maybe such an example would be too specific, but here I am, needing to solve this problem, and it continues to be challenging to me).

Thanks again, I really appreciate your time.