Add nanobind

[WillAyd]

cc @wjakob

[eli-schwartz]

CI fails due to missing releases.json:

https://github.com/mesonbuild/wrapdb?tab=readme-ov-file#wrap-release-meta

Aside: I would suggest squashing fixup commits (git commit --amend && git push --force or alternatively, git commit --fixup=HEAD followed later by git rebase -i --autosquash).

[eli-schwartz]

nanobind's CMake files have an option whether to install it, do we also want that? If users can choose not to install it, we might want to use:

build_target(...,
    target_type: should_install ? 'library' : 'static_library',
)

[WillAyd]

Nice catch. If I am reading the CMake configuration correctly it seems like it is only intended to be installed when not used as a subproject - so maybe I should just remove install altogether from this configuration?

[eli-schwartz]

Entirely possible. In that case, also change library to static_library, I think...

[eli-schwartz]

Add (for python module including embed: kwarg support):

    meson_version: '>=0.53.0',

[eli-schwartz]

IIRC releases.json has to be in alphabetical order, nanobind is right before nanoarrow but should instead be right after.

[WillAyd]

That makes sense. Looks like some of the build failures have to do with the robin-map depdendency version as well - may need to update that as a precursor

[wjakob]

@eli-schwartz @WillAyd : If I understand correctly, the goal of this PR is to build using pure Meson without depending on CMake? This would be awesome, but the recipe here looks somehow too easy to me.

Please take a look at some of the steps explained in the comment at the top of https://github.com/wjakob/nanobind/blob/master/src/nb_combined.cpp. And this is the "ground truth" reference for the extension build process: https://github.com/wjakob/nanobind/blob/master/cmake/nanobind-config.cmake

For example, does this wrapper

1. Avoid linking against a specific libpython.so/dylib? Basically, we don't want to bake a libpython path into the compiled shared library—the interpreter loading the extension will already have all of the libpython symbols.
2. Compute the extension suffix suitable for the desired Python ABI and stable/non-stable convention?
3. Set compilation flags required for correctness (-fno-strict-aliasing) in the static library part only?
4. Support stable ABI builds?
5. Set flags that are a good idea to obtain small binaries when making release builds ('-DNDEBUG' and '-DNB_COMPACT_ASSERTIONS', '-ffunction-sections/-fdata-sections/--gc-sections', various stripping flags -- see nanobind's CMake build system for details on these steps)
6. Use the chained fixups optimization for library builds on macOS? Nanobind ships a custom linker response file to enable this.
7. Support isolating extensions from others by specifying the NB_DOMAIN parameter? This is in general something that almost every extension should do to avoid cross-contamination.

Quite a lot of work has gone into the CMake interface produce nice output—I would be concerned about alternatives that produce subpar extensions by missing some of these steps. A good recipe that helped to reproduce the behavior in the Bazel build system (https://github.com/nicholasjng/nanobind-bazel) has been to run nanobind's CMake build sytem on a simple extension in verbose mode and then try to exactly reproduce all of the compiler and linker invocations with their flags on macOS+Windows+Linux (some subtle differences there as well).

I'm happy to provide further detail on anything with the caveat that I'll be traveling in July.

[eli-schwartz]

For example, does this wrapper

1. Avoid linking against a specific libpython.so/dylib? Basically, we don't want to bake a libpython path into the compiled shared library—the interpreter loading the extension will already have all of the libpython symbols.
2. Compute the extension suffix suitable for the desired Python ABI and stable/non-stable convention?

These questions are related.

That's something every python module already has to handle. Setuptools handles this automatically, as does maturin. Meson handles it automatically too! CMake doesn't support it, which is why scikit-build and I assume nanobind as well, provide their own cmake modules that define functions which override cmake add_library.

Meson calls this pymod.extension_module().

3. Set compilation flags required for correctness (-fno-strict-aliasing) in the static library part only?

This is a legacy of python 2 which violates the C/C++ language standards. It's not valid code if it has to compile with disabling the mandatory aliasing rules of the C++ language.

Since many people get aliasing wrong even though it breaks the language (and so did CPython 2.7!) -- and because sometimes the fact that your code breaks the requirements is difficult to spot -- compiler vendors offer a compiler extension that tells the compiler to cripple its codegen and produce really slow code that tiptoes around aliasing, adding redundant checks here there and everywhere to check at runtime whether the code actually followed the aliasing rules, and recover if it didn't follow the rules.

Aliasing violations can't come from CPython since CPython fixed that. They can still come from projects' own code, or from their nanobind dependency.

I think you're saying that nanobind needs this extra protection as the cpp_args kwarg to the library() call for nanobind itself? That can be added.

4. Support stable ABI builds?

Meson handles this automatically as part of pymod.extension_module(). If nanobind needs to be built with the limited API define then it should be manually calculated based on the -Dpython.allow_limited_api=true builtin option, as meson doesn't handle this for static libraries that use CPython and get linked into extension modules.

It may make sense to extend the meson language to handle static libraries here somehow. I'll think about this...

5. Set flags that are a good idea to obtain small binaries when making release builds ('-DNDEBUG' and '-DNB_COMPACT_ASSERTIONS', '-ffunction-sections/-fdata-sections/--gc-sections', various stripping flags -- see nanobind's CMake build system for details on these steps)

This can be added to the declare_dependency() function call in this PR. Meson does have a builtin -Db_ndebug=true option which should be respected, though.

6. Use the chained fixups optimization for library builds on macOS? Nanobind ships a custom linker response file to enable this.
7. Support isolating extensions from others by specifying the NB_DOMAIN parameter? This is in general something that almost every extension should do to avoid cross-contamination.

Those are interesting points. The linker response file can probably be handled just like point 5. I'm not sure how to handle point 7 (users could I suppose add that to their own extensions).

Quite a lot of work has gone into the CMake interface produce nice output—I would be concerned about alternatives that produce subpar extensions by missing some of these steps. A good recipe that helped to reproduce the behavior in the Bazel build system (https://github.com/nicholasjng/nanobind-bazel) has been to run nanobind's CMake build sytem on a simple extension in verbose mode and then try to exactly reproduce all of the compiler and linker invocations with their flags on macOS+Windows+Linux (some subtle differences there as well).

I'm happy to provide further detail on anything with the caveat that I'll be traveling in July.

Thanks for your insights!

[WillAyd]

9. Set flags that are a good idea to obtain small binaries when making release builds ('-DNDEBUG' and '-DNB_COMPACT_ASSERTIONS', '-ffunction-sections/-fdata-sections/--gc-sections', various stripping flags -- see nanobind's CMake build system for details on these steps)

This can be added to the declare_dependency() function call in this PR. Meson does have a builtin -Db_ndebug=true option which should be respected, though.

Does Meson not set most of these in the first place for a release build? I added -DNB_COMPACT_ASSERTIONS but from previous testing the rest were automatically added - so I don't think need to try and manually add here?

[WillAyd]

Ignore my previous comment - those flags probably came from the CMake wrap I tried before. Will take another look

[wjakob]

Two quick responses to @eli-schwartz:

CMake doesn't support it, which is why scikit-build and I assume nanobind as well, provide their own cmake modules that define functions which override cmake add_library.

Incorrect -- CMake handles it too (via Python_add_library). But the way that CMake does this on macOS (by specifying the legacy -undefined dynamic_lookup linker flag) is incompatible with the recently added chained fixups linker optimization, so nanobind uses a custom solution. There is lots of discussion about it here.

Aliasing violations can't come from CPython since CPython fixed that.

To my knowledge (and I would be really curious to be pointed to something saying otherwise) Core CPython C code to the cannot be compiled as regular C++ code. It heavily relies on macros that cast and reinterpret data structures that aren't related to each other through inheritance, which violates C++ type punning rules. The -fno-strict-aliasing is therefore required when compiling CPython-style C code in a C++ context. The reason why nanobind specifies this flag to the "libnanobind" utility library component is because it is written like regular CPython code, making plentiful use of such data structure reinterpretation that would be suspect in C++. But this flag isn't imposed on the remainder of the (user-provided) bindings, so aliasing analysis remains useful there.

[thesamesam]

To my knowledge (and I would be really curious to be pointed to something saying otherwise) Core CPython C code to the cannot be compiled as regular C++ code. It heavily relies on macros that cast and reinterpret data structures that aren't related to each other through inheritance, which violates C++ type punning rules.

I'm not sure about this. C of course has its own type punning rules. CPython 2.x did have an infamous problem with it (https://peps.python.org/pep-3123/). I'm not aware of any issue in modern CPython.

numpy also has a bunch of C++ which is built without -fno-strict-aliasing (discussed at numpy/numpy#25004) and nothing has gone wrong there.

Further, I'm not aware of any construct used in CPython which is legal under C aliasing rules but suddenly illegal under C++ rules. Could you point me to it?

[SoapGentoo]

@wjakob please link where in the Python documentation C++ compilation requires -fno-strict-aliasing. C and C++ literally have pretty much the exact same strict aliasing rules. Specifically, the rules that PEP 3123 relies are the same in C++, so your claims require actual evidence.

[eli-schwartz]

I would use the string method .version_compare('<3.12')

[WillAyd]

Any guidance on how to handle the MSVC errors about a debug build with a release Python interpreter?

nanobind| WARNING: Using a debug build type with MSVC or an MSVC-compatible compiler
nanobind| when the Python interpreter is not also a debug build will almost
nanobind| certainly result in a failed build. Prefer using a release build
nanobind| type or a debug Python interpreter.

Should I just force a release build on msvc?

[eli-schwartz]

Setting the release buildtype in ci_config.json is definitely a reasonable way to deal with this.

[eli-schwartz]

The alpine failure is probably because alpine uses -dev packages just like Ubuntu.

[WillAyd]

Temporarily including to make CI verification easier, as pointed out by @benoit-pierre

[eli-schwartz]

Shouldn't these also be part of dep_link_args?

[WillAyd]

The current CMake configuration sets these as PRIVATE on the target, so just copying that over. Happy to move this to the dependency interface if you think that makes more sense

[WillAyd]

Ah nevermind - I take that back. I see it is called by both the internal function to build the nanobind library as well as the nanobind_add_module function that end users are expected to use.

I think what you are proposing makes sense

[eli-schwartz]

if is_pypy # how to check this?
    resp_file = meson.project_source_root() / 'cmake/darwin-ld-pypy.sym'
else
    resp_file = meson.project_source_root() / 'cmake/darwin-ld-cpython.sym'
fi
dep_link_args += ['-Wl,@' + resp_file]

@rgommers might remember if we've discussed how to detect pypy.

[eli-schwartz]

static_library (and library, to cover static ones) has the pic: <bool> option. By default it is unset, which means that the value of -Db_staticpic is used instead, so builders can make this choice. By default, the project option is true: all static libraries will build with PIC.

You can explicitly force this with pic: true as a kwarg to static_library to prevent anyone from overriding it on the command line. You can also explicitly force it off with pic: false if you're positive you don't want to build with PIC.

In general, static libraries that may later be used by shared libraries need this, disabling PIC is a niche use case.

[eli-schwartz]

get_option('b_ndebug') has 3 values: true, false, if-release

Probably, projects should simply specify if-release for their own use. There's no need to manually pass -DNDEBUG IMO.

Opinions vary, apparently. 🤷

[WillAyd]

Seems like the current VisualStudio x64 failure is reported upstream and a bug with the latest msvc compiler version

wjakob/nanobind#613

[benoit-pierre]

And for the x32 job: #1562. With that get similar errors.

[WillAyd]

Thanks @benoit-pierre . My assumption is those two jobs will continue to fail until a newer msvc release comes out (or we can try to test against an older compiler version)

[eli-schwartz]

embed: true is used for cases where you are building an application that contains its own main() function and invokes Py_Initialize().

[eli-schwartz]

When a project uses:

py.extension_module(
    'foomod',
    'foomod.c',
    limited_api: '3.12',
)

meson links to python3.dll rather than python312.dll. I assume that's what "SABIModule" is.

[WillAyd]

Seems reasonable. I assume it would be more idiomatic to remove the code that defines Py_LIMITED_API in this configuration and just let Meson handle that

[WillAyd]

Converted to draft to try out the nanobind commit that works around the MSVC compiler bug and get things green. Probably need to wait until the next nanobind release or MSVC fix, whichever comes first

However, I think the configuration itself is ready for any more review comments

[WillAyd]

Is the MSVC bug affecting nanobind in CI a blocker for this? This is the upstream report:

https://developercommunity.visualstudio.com/t/Internal-compiler-error-w-nanobind-v19/10662883?sort=newest

If not and if there is no further feedback on this, I'd be happy to squash and skip MSVC builds for now to push through

Requested changes were implemented

[eli-schwartz]

Looks ready to me...

[WillAyd]

When using that I notice that I have to do both:

meson wrap install robin-map
meson wrap install nanobind

Is it expected that the user has to specify this, or did I maybe miss something here that ensures when a user installs nanobind that robin-map gets pulled in as well?

[eli-schwartz]

You aren't missing anything. meson wrap install doesn't know about recursive dependencies, and you might not necessarily care about them anyway -- some dependencies are harder than others, so e.g. people might not bother adding a wrap for zlib, but would add a wrap for curl which depends on zlib.