libomp: xgboost incorrect LC_RPATH referenced in libxgboost.dylib

[Ubiquitous]

Upon installing xgboost (2.1.2) with pip on macOS (15.0.1) I noticed that there was an error related to not finding libomp once installed. In the installed libxgboost.dylib the rcpath references an absolute 'homebrew' directory (I don't use homebrew).

% otool -l /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib

Load command 15
          cmd LC_RPATH
      cmdsize 48
         path /opt/homebrew/opt/libomp/lib (offset 12)

It should likely be changed to use: @rpath/libomp unless you expect anyone who doesn't use homebrew to fix the hardcoded path anytime they install xgboost with pip.

[trivialfis]

@Ubiquitous Thank you for opening an issue. Would you like to help fix it? We are indeed having a lot of headaches with macos distribution around openmp, your expertise is appreciated.

[hcho3]

We merged #10440 to patch libxgboost.dylib so that it depends on @rpath/libomp.dylib, by running

install_name_tool -change /opt/homebrew/opt/libomp/lib/libomp.dylib "@rpath/libomp.dylib" libxgboost.dylib

It looks like we need to run additional command(s) to patch the rcpath as well.

[hcho3]

@Ubiquitous Can you try running the following command and see if that fixes the issue?

install_name_tool -add_rpath "@rpath/libomp" /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib

[hcho3]

I also want to hear from @jameslamb about how LightGBM solved this issue.

[jameslamb]

It should likely be changed to use: @rpath/libomp unless you expect anyone who doesn't use homebrew to fix the hardcoded path anytime they install xgboost with pip.

This doesn't sound quite right to me. Having an RPATH pointing to Homebrew's preferred location is just a hint to the loader. If you don't have OpenMP installed via Homebrew, then there shouldn't be anything found at that path, and the loader should precede to search standard system paths like /usr/lib.

Before we go any further... @Ubiquitous can you please share precisely what problem you encountered? How are you getting OpenMP and where is it installed?

You just mentioned "an error" and then prescribed a solution... but the solution will depend on the specific problem you faced.

Important to note that the install name for libxgboost.dylib's OpenMP dependency is @rpath/libomp.dylib (as @hcho3 alluded to in #10946 (comment)), so it's not true that using Homebrew is absolutely required.

mkdir -p ./delete-me
pip download \
	-d ./delete-me \
	--no-deps \
	--prefer-binary \
	'xgboost==2.1.2'

cd ./delete-me
unzip ./xgboost*.whl

otool -l xgboost/lib/libxgboost.dylib

Excerpt from that:

Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libomp.dylib (offset 24)
   time stamp 2 Wed Dec 31 18:00:02 1969
      current version 5.0.0
compatibility version 5.0.0
...
Load command 15
          cmd LC_RPATH
      cmdsize 48
         path /opt/homebrew/opt/libomp/lib (offset 12)

And output of otool -L ./xgboost/lib/libxgboost.dylib

./xgboost/lib/libxgboost.dylib:
        @rpath/libxgboost.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.5)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)

[Ubiquitous]

@jameslamb, I apologize for not being more clear; I installed the libomp via MacPorts, succeeded by pip install xgboost. An error came back after import xgboost threw an exception: libomp.dylib no such file. The exact error is shown below and none of the locations searched is where MacPorts installs libomp.dylib.

% pip install xgboost 
...
Successfully installed xgboost-2.1.2

% python
>>> import xgboost as xgb

Error message(s): ["dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib, 0x0006): Library not loaded: @rpath/libomp.dylib\n
Referenced from: <F2F42313-BF4F-3B95-A853-AE1DE94D4C87> /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib\n
Reason: tried: '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/opt/libomp/lib/libomp.dylib' (no such file)"]

I used otool to check the paths of libxgboost.dylib:

% otool -l /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib

...
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name @rpath/libomp.dylib (offset 24)
   time stamp 2 Wed Dec 31 17:00:02 1969
      current version 5.0.0
compatibility version 5.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 17:00:02 1969
      current version 1700.255.5
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 17:00:02 1969
      current version 1345.120.2
compatibility version 1.0.0
Load command 15
          cmd LC_RPATH
      cmdsize 48
         path /opt/homebrew/opt/libomp/lib (offset 12)
...

I used install_name_tool to change the path of Load command 15:

% sudo install_name_tool -rpath /opt/homebrew/opt/libomp/lib /opt/local/lib/libomp /opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xgboost/lib/libxgboost.dylib

xgboost worked fine after that.

---

I searched here in the repo for references to homebrew, which found 12 files, although it's unclear exactly where the path is getting set when pip builds libxgboost.dylib

My recommendation would likely be to create a different build script that takes into account possible other locations where libomp.dylib might be installed and set the @rpath that way. Possible issues with this approach is that there could be more than one version or installed in a location that's uncommon.

#!/bin/bash

# Common base paths where libomp might be installed
BASE_PATHS=(
    "/opt/local/lib/libomp"           # MacPorts
    "/opt/homebrew/opt/libomp/lib"    # Homebrew Apple Silicon
    "/usr/local/opt/libomp/lib"       # Homebrew Intel
    "/usr/local/lib/libomp"           # Custom install
    "/usr/lib/libomp"                 # System
)

# Check each base path for libomp.dylib
for base in "${BASE_PATHS[@]}"; do
    if [ -f "$base/libomp.dylib" ]; then
        echo "Found: $base/libomp.dylib"
        exit 0
    fi
done

# If not found in common paths, do a restricted but deeper check
for dir in /opt /usr/local; do
    find "$dir" -name "libomp.dylib" 2>/dev/null
done

Since I'm running MacPorts the path where libomp.dylib resides is /opt/local/lib/libomp.

I can certainly help you resolve the issue, although there's multiple ways to attack here. Having what appears to be an exclusive dependency on Homebrew is likely the wrong approach.

[jameslamb]

Thanks for that detail... it makes the problem and relative preference for solutions much clearer.

It's not quite accurate to frame this problem as XGBoost having an "exclusive dependency on Homebrew". I strongly suspect that if you set environment variable LD_LIBRARY_PATH=/opt/local/lib/libomp or symlinked the macports install of OpenMP into a system directory like /usr/local/lib, import xgboost would find that library without any modifications to its binary.

But anyway... @hcho3 I think xgboost should add something like microsoft/LightGBM#6489, to take on an additional RPATH of /opt/local/lib/libomp for macOS builds. That'd allow libomp.dylib installed via macPorts to be found at runtime.

No shell script like #10946 (comment) (or equivalent code in any other language) should be required... just adding another RPATH entry here:

xgboost/cmake/FindOpenMPMacOS.cmake

Line 121 in c502bb8

INSTALL_RPATH "${HOMEBREW_LIBOMP_PREFIX}/lib;${__OpenMP_LIBRARY_DIR}"

How LightGBM does this: https://github.com/microsoft/LightGBM/blob/13f2e92bb0ac64f94d9b5016a33b5c34d2134204/CMakeLists.txt#L777-L782

[Ubiquitous]

@jameslamb You're welcome, and thank you for your attention on this issue.

Why though does the LC_RPATH get set to /opt/homebrew/opt/libomp/lib? Sure, I can set the environment variable or symlink as you said, although the error seemed to indicate libxgboost.dylib was relying on the homebrew path to find libomp.dylib.

[hcho3]

@Ubiquitous We can do what LightGBM does, which is to specify both /opt/homebrew/opt/libomp/lib and /opt/local/lib/libomp as candidates for the rpath. I will prepare a pull request.

[jameslamb]

Why though does the LC_RPATH get set to /opt/homebrew/opt/libomp/lib?

I think you are maybe misunderstanding how LC_RPATH works. You can provide multiple such paths and the loader will search them in order. Instead of saying "set to", it would be more accurate to say "has one entry pointing to". On your system, where /opt/homebrew/opt/libomp/lib does not exist, libxgboost.dylib having an RPATH entry pointing at that Homebrew-specific directory is not a problem.

And xgboost provides that path as a hint because it uses Homebrew to provide OpenMP in its wheel builds,

xgboost/.github/workflows/python_wheels.yml

Lines 33 to 34 in 3a9ce48

	- name: Install libomp
	run: brew install libomp

and because its docs recommend that users install OpenMP with homebrew.

xgboost/doc/install.rst

Lines 136 to 140 in 3a9ce48

	If you are using Mac OSX, you should first install OpenMP library (``libomp``) by running
	.. code-block:: bash
	brew install libomp

I'd support libxgboost.dylib picking up an additional RPATH entry suggesting that the loader should look wherever macports puts libomp.dylib (/opt/local/lib/libomp), as we do in lightgbm and as I linked above.

[Ubiquitous]

@Ubiquitous We can do what LightGBM does, which is to specify both /opt/homebrew/opt/libomp/lib and /opt/local/lib/libomp as candidates for the rpath. I will prepare a pull request.

Yes, that would work fine, thank you.

Why though does the LC_RPATH get set to /opt/homebrew/opt/libomp/lib?

I think you are maybe misunderstanding how LC_RPATH works. You can provide multiple such paths and the loader will search them in order. Instead of saying "set to", it would be more accurate to say "has one entry pointing to". On your system, where /opt/homebrew/opt/libomp/lib does not exist, libxgboost.dylib having an RPATH entry pointing at that Homebrew-specific directory is not a problem.

And xgboost provides that path as a hint because it uses Homebrew to provide OpenMP in its wheel builds,

xgboost/.github/workflows/python_wheels.yml

Lines 33 to 34 in 3a9ce48

	- name: Install libomp
	run: brew install libomp

and because its docs recommend that users install OpenMP with homebrew.

xgboost/doc/install.rst

Lines 136 to 140 in 3a9ce48

	If you are using Mac OSX, you should first install OpenMP library (``libomp``) by running
	.. code-block:: bash
	brew install libomp

I'd support libxgboost.dylib picking up an additional RPATH entry suggesting that the loader should look wherever macports puts libomp.dylib (/opt/local/lib/libomp), as we do in lightgbm and as I linked above.

Including the additional path(s) would be great. I wasn't misunderstanding how rpath works, it just came across as if homebrew was all that was supported since it was the only rpath included. One thing to note is the homebrew path for apple intel might differ from apple silicon, although I can't confirm that, so you might want to include both of those as well perhaps if that's the case ~ tc.