A warning message showing that MultiScaleDeformableAttention.so is not found in /root/.cache/torch_extensions if ninja is installed with transformers

[cainmagi]

System Info

• transformers: 4.47.1
• torch: 2.5.1
• timm: 1.0.12
• ninja: 1.11.1.3
• python: 3.10.14
• pip: 23.0.1
• CUDA runtime installed by torch: nvidia-cuda-runtime-cu12==12.4.127
• OS (in container): Debian GNU/Linux 12 (bookworm)
• OS (native device): Windows 11 Enterprise 23H2 (10.0.22631 Build 22631)
• Docker version: 27.3.1, build ce12230
• NVIDIA Driver: 565.57.02

Who can help?

I am asking help for DeformableDetrModel
vision models: @amyeroberts, @qubvel

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Start a new docker container by

   docker run --gpus all -it --rm --shm-size=1g python:3.10-slim bash

2. Install dependencies

   pip install transformers[torch] requests pillow timm

3. Run the following script (copied from the document), it works fine and does not show any message.

   from transformers import AutoImageProcessor, DeformableDetrModel
   from PIL import Image
   import requests
   
   url = "http://images.cocodataset.org/val2017/000000039769.jpg"
   image = Image.open(requests.get(url, stream=True).raw)
   
   image_processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr")
   model = DeformableDetrModel.from_pretrained("SenseTime/deformable-detr")
   
   inputs = image_processor(images=image, return_tensors="pt")
   
   outputs = model(**inputs)
   
   last_hidden_states = outputs.last_hidden_state
   list(last_hidden_states.shape)

4. Install ninja:

   pip install ninja

5. Run the same script again, this time, the following warning messages will show

       
                                  !! WARNING !!
   
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   Your compiler (c++) is not compatible with the compiler Pytorch was
   built with for this platform, which is g++ on linux. Please
   use g++ to to compile your extension. Alternatively, you may
   compile PyTorch from source using c++, and then you can also use
   c++ to compile your extension.
   
   See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
   with compiling PyTorch from source.
   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   
                                 !! WARNING !!
   
     warnings.warn(WRONG_COMPILER_WARNING.format(
   Could not load the custom kernel for multi-scale deformable attention: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   Could not load the custom kernel for multi-scale deformable attention: /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/MultiScaleDeformableAttention.so: cannot open shared object file: No such file or directory
   

   Certainly, /root/.cache/torch_extensions/py310_cu124/MultiScaleDeformableAttention/ is empty.

The issue happens only when both ninja and transformers are installed. I believe that the following issue may be related to this issue:

https://app.semanticdiff.com/gh/huggingface/transformers/pull/32834/overview

Expected behavior

It seems that ninja will let DeformableDetrModel throw unexpected error messages (despite that the script still works). That's may be because I am using a container without any compiler or CUDA preinstalled (the CUDA run time is installed by pip).

I think there should be a check that automatically turn of the ninja related functionalities even if ninja is installed by pip, as long as the requirements like compiler version, CUDA path, or something, are not fulfilled.

[Rocketknight1]

This seems like an interaction between ninja and the DeformableDETR library, rather than transformers specifically.