Missing library alias on Mac

[peastman]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

The Mac packages distributed through the default channel include an alias with the name libopenblas.dylib pointing to the library. That allows it to be referenced with standard naming. The conda-forge packages are missing that alias. Instead you have to reference it with the name libopenblas.0.dylib. This causes problems for software that expects the standard name to work. See, for example, numpy/numpy#14165 and https://gitlab.kitware.com/cmake/cmake/-/issues/23291.

Would it be possible to add the missing alias?

Installed packages

# packages in environment at /Users/peastman/miniconda3/envs/test:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h3422bc3_4    conda-forge
ca-certificates           2021.10.8            h4653dfc_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0.dev0      11_0_1_hf114ba7_23    conda-forge
libgfortran5              11.0.1.dev0         hf114ba7_23    conda-forge
libopenblas               0.3.18          openmp_h5dd58f0_0    conda-forge
libzlib                   1.2.11            hee7b306_1013    conda-forge
llvm-openmp               13.0.1               hf3c4609_0    conda-forge
ncurses                   6.3                  hc470f4d_0    conda-forge
openssl                   3.0.0                h3422bc3_2    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
python                    3.9.10          h38ef502_2_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
readline                  8.1                  hedafd6a_0    conda-forge
setuptools                60.9.3           py39h2804cbe_0    conda-forge
sqlite                    3.37.0               h72a2b83_0    conda-forge
tk                        8.6.12               he1e0b03_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h642e427_1    conda-forge
zlib                      1.2.11            hee7b306_1013    conda-forge

Environment info

active environment : test
    active env location : /Users/peastman/miniconda3/envs/test
            shell level : 2
       user config file : /Users/peastman/.condarc
 populated config files : /Users/peastman/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.8.11.final.0
       virtual packages : __osx=12.2=0
                          __unix=0=0
                          __archspec=1=arm64
       base environment : /Users/peastman/miniconda3  (writable)
      conda av data dir : /Users/peastman/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/omnia/osx-arm64
                          https://conda.anaconda.org/omnia/noarch
                          https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/peastman/miniconda3/pkgs
                          /Users/peastman/.conda/pkgs
       envs directories : /Users/peastman/miniconda3/envs
                          /Users/peastman/.conda/envs
               platform : osx-arm64
             user-agent : conda/4.11.0 requests/2.25.1 CPython/3.8.11 Darwin/21.3.0 OSX/12.2
                UID:GID : 503:20
             netrc file : None
           offline mode : False

[isuruf]

It's in openblas package along with the headers.

[peastman]

I'm trying to track down a build failure. Perhaps you have insight into what the real problem is? My library uses PyTorch and links to libtorch.dylib. It does not directly use OpenBLAS, but apparently PyTorch does at some level, and I get an error trying to configure with CMake, telling me that libopenblas.dylib doesn't exist. Installing the PyTorch package installs the libopenblas package but not the openblas package. So what is the real problem?

• Should the PyTorch package list openblas as a dependency?
• Should the libopenblas package include the alias (the same way the corresponding package on the default channel does)?
• Should CMake recognize that the correct library name is libopenblas.0.dylib rather than libopenblas.dylib?
• Something else?

One way or another, the current behavior is that if you install the PyTorch conda package and then use CMake to build something that links to libtorch, you get an error. What is the correct place to fix this?

[isuruf]

What's the error exactly?

[peastman]

make[2]: *** No rule to make target '/Users/peastman/miniconda3/envs/openmm/lib/libopenblas.dylib', needed by 'libOpenMMTorch.dylib'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:190: CMakeFiles/OpenMMTorch.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

[peastman]

Does anyone have insight on what the correct fix is? I don't know whether the change should be in PyTorch, OpenBLAS, or CMake, but something needs to be changed somewhere.

[isuruf]

See #134 (comment)

[peastman]

See #134 (comment) above. If a user simply installs PyTorch and tries to compile against it, they get a compilation error. The package includes libtorch, which contains the C++ API and is designed to be a library users link their own code to. Using the package in the intended way leads to errors. The fact that it uses OpenBLAS is an internal implementation detail of PyTorch, exactly the sort of thing conda is supposed to take care of for you.

Is there any good reason not to include the symlink in libopenblas? Omitting it creates problems for users. The version of the package on the defaults channel includes it.

[peastman]

Can anyone else comment on this? The problem is still not resolved. I believe the solution I originally proposed (including the alias in libopenblas) is the correct solution. I'm open to other possibilities, but no one has suggested a better one or pointed out any downside to including the alias. It's a tiny change that saves a lot of problems for users and makes the conda-forge version of the package consistent with the default channel version.

cc @jakirkham @h-vetinari

[isuruf]

Is there any good reason not to include the symlink in libopenblas?

Yes, to prevent downstream packages from accidentally linking to them. The symlink and headers are in the development package just like debian, fedora and countless other package managers. Are you suggesting that debian should package the symlink and the headers in the non dev package?

[isuruf]

For eg: see the difference between
https://packages.debian.org/stretch/amd64/libopenblas-base/filelist
and
https://packages.debian.org/stretch/amd64/libopenblas-dev/filelist

[peastman]

Again, I'm happy to consider other options. But you still haven't suggested any solution at all to the problem. Let me repeat one more time exactly what the steps are.

1. Install the pytorch package from CondaForge.
2. Try to compile a program that links to libtorch.
3. It fails with a linker error saying that libopenblas.dylib can't be found.

Notice that the user has not installed OpenBLAS. Until they hit this error message, they didn't even know that PyTorch used OpenBLAS. They certainly don't know there are two different packages related to OpenBLAS. Further, this error only happens with recent versions of PyTorch. With older versions the above steps work perfectly, because they require an older version of OpenBLAS than is available on CondaForge. Instead conda installs a package from the defaults channel which does include the alias.

Are you suggesting that debian should package the symlink and the headers in the non dev package?

What does this have to do with Debian? This is an error on Mac.

[isuruf]

But you still haven't suggested any solution at all to the problem.

Ask pytorch-cpu-feedstock maintainers to patch the cmake file. It should not be necessary to link to libopenblas.dylib.

What does this have to do with Debian? This is an error on Mac.

It's not unique to Mac. This doesn't manifest in Linux x86, because pytorch uses MKL in Linux x86.

Anyways, this is a pytorch-cpu-feedstock issue and not this feedstock.