Added nomkl package installation causes issues with pytorch downstream

[holzman]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

@isuruf

numexpr now installs nomkl by default, which can cause problems with downstream packages that are mkl-only.

Installing pytorch after numexpr fails with

Encountered problems while solving:
  - package pytorch-1.6.0-cuda100py39h2b73809_1 requires mkl >=2020.4,<2021.0a0, but none of the providers can be installed

Reproducer Dockerfile:

FROM centos:centos7
ARG MAMBAFORGE=https://github.com/conda-forge/miniforge/releases/download/4.11.0-4/Mambaforge-Linux-x86_64.sh
RUN curl -sSL $MAMBAFORGE -o /tmp/mambaforge.sh && bash /tmp/mambaforge.sh -bfp /usr/local
RUN mamba install --yes numexpr
RUN mamba install --yes pytorch

Installed packages

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
brotlipy                  0.7.0           py39h3811e60_1003    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
certifi                   2021.10.8        py39hf3d152e_1    conda-forge
cffi                      1.15.0           py39h4bc2ebd_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.11.0           py39hf3d152e_0    conda-forge
conda-package-handling    1.7.3            py39h3811e60_1    conda-forge
cryptography              36.0.1           py39h95dcef6_0    conda-forge
icu                       69.1                 h9c3ff4c_0    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
krb5                      1.19.2               hcc1bbae_3    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libarchive                3.5.2                hccf745f_1    conda-forge
libblas                   3.9.0           13_linux64_openblas    conda-forge
libcblas                  3.9.0           13_linux64_openblas    conda-forge
libcurl                   7.81.0               h2574ce0_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgfortran-ng            11.2.0              h69a702a_14    conda-forge
libgfortran5              11.2.0              h5c6108e_14    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           13_linux64_openblas    conda-forge
libmamba                  0.21.2               h3985d26_0    conda-forge
libmambapy                0.21.2           py39h8bfa403_0    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.18          pthreads_h8fe5266_0    conda-forge
libsolv                   0.7.19               h780b84a_5    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_12    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libxml2                   2.9.12               h885dcf4_1    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mamba                     0.21.2           py39hfa8f2c8_0    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numexpr                   2.8.0           py39hbd72853_101    conda-forge
numpy                     1.22.3           py39h18676bf_0    conda-forge
openssl                   1.1.1l               h7f98852_0    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.3           py39h3811e60_1009    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1            py39hf3d152e_4    conda-forge
python                    3.9.10          h85951f9_2_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
reproc                    14.2.3               h7f98852_0    conda-forge
reproc-cpp                14.2.3               h9c3ff4c_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
ruamel_yaml               0.15.80         py39h3811e60_1006    conda-forge
setuptools                60.9.3           py39hf3d152e_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tqdm                      4.62.3             pyhd8ed1ab_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
urllib3                   1.26.8             pyhd8ed1ab_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yaml-cpp                  0.6.3                he1b5a44_4    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge

Environment info

active environment : None
       user config file : /root/.condarc
 populated config files : /usr/local/.condarc
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.9.10.final.0
       virtual packages : __linux=3.10.0=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /usr/local  (writable)
      conda av data dir : /usr/local/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /usr/local/pkgs
                          /root/.conda/pkgs
       envs directories : /usr/local/envs
                          /root/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.9.10 Linux/3.10.0-1160.41.1.el7.x86_64 centos/7.9.2009 glibc/2.17
                UID:GID : 0:0
             netrc file : None
           offline mode : False

[isuruf]

I guess https://github.com/conda-forge/numexpr-feedstock/blob/main/recipe/meta.yaml#L21-L22 is the issue. Maybe we can remove that and see if it works.

[hmaarrfk]

Yeah, i noticed that.

I think that the reason why these track features have been added more recently is because they downweigh the mamba solver.

@wolfv maybe this kind of issue can help you better understand the usecases encountered.

[beckermr]

Picking this back up. Is there a reason we need nomkl in run? I'd think if numexpr is not built against mkl, it wouldn't care either way if mkl happens to be in the environment.

I suppose this is to ensure numexpr is using mkl if it is there?

Could we add a downweighted output that is neutral with respect to mkl?

[isuruf]

@beckermr, can you send a PR to remove track_features?

[beckermr]

Yep. I am happy to do that but it won't solve my specific problem.

[beckermr]

xref: #47

[isuruf]

What's your specific problem?

[beckermr]

PyTorch requires mkl on Linux.

[beckermr]

I made a pr adding Openblas.

[isuruf]

PyTorch requires mkl on Linux.

Do you not want MKL or do you want to be able to install numexpr with pytorch?

[beckermr]

I want to use pytorch with openblas.

[isuruf]

Ok, so it's not related to numexpr at all.

[beckermr]

Yes. My apologies for not being clear. I thought it was but it turns out not. I am happy to have fixed this issue hopefully though! Thanks for your time!

[hmaarrfk]

So I'm still experiencing issues with this:

Generally speaking, it seems that numpexr REQUIRES nomkl, but there is no way to "install a different package build string" while simultaneously uninstalling nomkl.

With:

• numexpr preferring nomkl (via build numbers)
• pytorch preferring cuda (via build numbers)

it somewhat puts the solver in a difficult "bind" to try to find a good solution. I often find myself having conflicts when I try to kick pytorch back into the cuda variant.

So if i have pytorch=*=cpu* and numpexpr (nomkl variant)

 $ mamba install pytorch=*=cuda120*

Looking for: ['pytorch=[build=cuda120*]'

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Pinned packages:
  - python 3.11.*


warning  libmamba Added empty dependency for problem type SOLVER_RULE_UPDATE
Could not solve for environment specs
The following packages are incompatible
├─ nomkl is installable and it requires
│  └─ mkl <0.a0 , which can be installed;
├─ numpexr * mkl* does not exist (perhaps a typo or a missing channel);
├─ pyarrow-core is installable with the potential options
│  ├─ pyarrow-core [16.0.0|16.1.0|17.0.0] would require
│  │  └─ libarrow [16.0.0.* *cpu|16.0.0.* *cuda|...|17.0.0.* *cuda], which requires
│  │     └─ libabseil >=20240116.2,<20240117.0a0 , which can be installed;
│  ├─ pyarrow-core [16.0.0|16.1.0|17.0.0] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ pyarrow-core [16.0.0|16.1.0|17.0.0] would require
│  │  └─ python >=3.12,<3.13.0a0 , which can be installed;
│  ├─ pyarrow-core [16.0.0|16.1.0|17.0.0] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  └─ pyarrow-core [16.0.0|16.1.0|17.0.0] would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ pytorch * cuda120* is installable with the potential options
   ├─ pytorch [2.0.0|2.1.0|...|2.4.0] would require
   │  └─ python >=3.10,<3.11.0a0 , which can be installed;
   ├─ pytorch [2.0.0|2.1.0|2.1.2] would require
   │  └─ libprotobuf [>=4.24.4,<4.24.5.0a0 |>=4.25.1,<4.25.2.0a0 ], which requires
   │     └─ libabseil >=20230802.1,<20230803.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytorch [2.0.0|2.1.0|...|2.4.0] would require
   │  └─ python >=3.8,<3.9.0a0 , which can be installed;
   ├─ pytorch [2.0.0|2.1.0|...|2.4.0] would require
   │  └─ python >=3.9,<3.10.0a0 , which can be installed;
   ├─ pytorch [2.1.0|2.1.2|2.3.0|2.3.1|2.4.0] would require
   │  └─ python >=3.12,<3.13.0a0 , which can be installed;
   ├─ pytorch 2.1.2 would require
   │  └─ libtorch 2.1.2.*  with the potential options
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_generic_*_4, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_generic_*_0, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_generic_*_1, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_generic_*_3, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_mkl_*_100, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_mkl_*_101, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_mkl_*_103, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cpu_mkl_*_104, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda112_*_300, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda112_*_301, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda118_*_301, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda118_*_303, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda118_*_300, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda118_*_304, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda120_*_301, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda120_*_303, which can be installed;
   │     ├─ libtorch 2.1.2 would require
   │     │  └─ pytorch 2.1.2 cuda120_*_300, which can be installed;
   │     └─ libtorch [2.1.2|2.3.0|2.3.1|2.4.0] would require
   │        └─ mkl >=2023.2.0,<2024.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytorch 2.3.0 would require
   │  └─ libtorch 2.3.0.*  with the potential options
   │     ├─ libtorch [2.1.2|2.3.0|2.3.1|2.4.0], which cannot be installed (as previously explained);
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cpu_generic_*_0, which can be installed;
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cpu_generic_*_1, which can be installed;
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cpu_mkl_*_101, which can be installed;
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cpu_mkl_*_100, which can be installed;
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cuda118_*_301, which can be installed;
   │     ├─ libtorch 2.3.0 would require
   │     │  └─ pytorch 2.3.0 cuda118_*_300, which can be installed;
   │     └─ libtorch 2.3.0 would require
   │        ├─ mkl >=2023.2.0,<2024.0a0 , which conflicts with any installable versions previously reported;
   │        └─ pytorch 2.3.0 cuda120_*_300, which can be installed;
   ├─ pytorch 2.3.1 would require
   │  └─ libtorch 2.3.1.*  with the potential options
   │     ├─ libtorch [2.1.2|2.3.0|2.3.1|2.4.0], which cannot be installed (as previously explained);
   │     ├─ libtorch 2.3.1 would require
   │     │  └─ pytorch 2.3.1 cpu_generic_*_0, which can be installed;
   │     ├─ libtorch 2.3.1 would require
   │     │  └─ pytorch 2.3.1 cpu_mkl_*_100, which can be installed;
   │     └─ libtorch 2.3.1 would require
   │        └─ pytorch 2.3.1 cuda118_*_300, which can be installed;
   └─ pytorch 2.4.0 would require
      └─ libtorch 2.4.0.*  with the potential options
         ├─ libtorch [2.1.2|2.3.0|2.3.1|2.4.0], which cannot be installed (as previously explained);
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cpu_generic_*_1, which can be installed;
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cpu_generic_*_0, which can be installed;
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cpu_mkl_*_100, which can be installed;
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cpu_mkl_*_101, which can be installed;
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cuda118_*_300, which can be installed;
         ├─ libtorch 2.4.0 would require
         │  └─ pytorch 2.4.0 cuda118_*_301, which can be installed;
         └─ libtorch 2.4.0 would require
            ├─ mkl >=2023.2.0,<2024.0a0 , which conflicts with any installable versions previously reported;
            └─ pytorch 2.4.0 cuda120_*_300, which can be installed.

My workaround: conda uninstall --force --offline nomkl --yes

conda uninstall --force --offline nomkl --yes

## Package Plan ##

  environment location: /home/mark/miniforge3/envs/test

  removed specs:
    - nomkl


The following packages will be REMOVED:

  nomkl-1.0-h5ca1d4c_0


Preparing transaction: done
Verifying transaction: done
Executing transaction: done

I can then run mamba install pytorch=*=cuda120* numexpr=*=mkl* and then things will solve....

Could we perhaps remove the nomkl from the runtime (@beckermr had suggested something like this a while back #38 (comment))