./TensorNVMe/include/aio.h:3:10: fatal error: libaio.h: No such file or directory

[marsggbo]

how to solve this problem without root priority when installing tensornvme?

[Angelconan]

mark，I have the same problem！

[Angelconan]

I solved this problem by compiling locally

[marsggbo]

I solved this problem by compiling locally

Hi @Angelconan , could you share your installation steps?

• cmake version: 3.22.1
• gcc version: 7.3.0

I tried to install liburing and libaio locally by using their source code, but when I ran pip install tensornvme, I got the error below:

Processing /home/xihe/.tensornvme/TensorNVMe
  Running command python setup.py egg_info
  running egg_info
  creating /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info
  writing /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/entry_points.txt
  writing requirements to /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/requires.txt
  writing top-level names to /tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/top_level.txt
  writing manifest file '/tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/SOURCES.txt'
  reading manifest file '/tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file '/tmp/pip-pip-egg-info-hbii2su8/tensornvme.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Requirement already satisfied: packaging in /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages (from tensornvme==0.1.0) (21.3)
Requirement already satisfied: click in /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages (from tensornvme==0.1.0) (8.1.3)
Requirement already satisfied: torch in /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages (from tensornvme==0.1.0) (1.12.1+cu113)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages (from packaging->tensornvme==0.1.0) (3.0.9)
Requirement already satisfied: typing-extensions in /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages (from torch->tensornvme==0.1.0) (4.3.0)
Building wheels for collected packages: tensornvme
  Running command python setup.py bdist_wheel
  -- The CXX compiler identification is GNU 7.3.0
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  liburing is found in /home/xihe/.tensornvme/libs/liburing/lib/liburing.so
  libaio is found in /home/xihe/.tensornvme/libs/libaio/lib/libaio.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/xihe/.tensornvme/TensorNVMe/cmake-build
  make: *** No rule to make target `extern_aio'.  Stop.
  No CUDA runtime is found, using CUDA_HOME='/cm/extra/Utils/CUDA/11.1.0.0_455.23.05'
  running bdist_wheel
  running build
  running build_py
  running build_ext
  building 'tensornvme._C' extension
  Emitting ninja build file /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  ninja: no work to do.
  g++ -pthread -B /datasets/xihe/miniconda3/envs/colossal/compiler_compat -shared -Wl,-rpath,/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath-link,/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath,/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath-link,/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/aio.o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/offload.o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/space_mgr.o -L/home/xihe/.tensornvme/lib -L/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/lib -laio -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-39/tensornvme/_C.cpython-39-x86_64-linux-gnu.so
  /datasets/xihe/miniconda3/envs/colossal/compiler_compat/ld: cannot find -laio: No such file or directory
  collect2: error: ld returned 1 exit status
  error: command '/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/bin/g++' failed with exit code 1
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /datasets/xihe/miniconda3/envs/colossal/bin/python -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/home/xihe/.tensornvme/TensorNVMe/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' bdist_wheel -d /tmp/pip-wheel-dhb1gpbd
  cwd: /home/xihe/.tensornvme/TensorNVMe/
  Building wheel for tensornvme (setup.py) ... error
  ERROR: Failed building wheel for tensornvme
  Running setup.py clean for tensornvme
  Running command python setup.py clean
  running clean
  removing 'build/temp.linux-x86_64-cpython-39' (and everything under it)
  'build/lib' does not exist -- can't clean it
  'build/bdist.linux-x86_64' does not exist -- can't clean it
  'build/scripts-3.9' does not exist -- can't clean it
Failed to build tensornvme
Installing collected packages: tensornvme
  Running command Running setup.py install for tensornvme
  liburing is found in /home/xihe/.tensornvme/libs/liburing/lib/liburing.so
  libaio is found in /home/xihe/.tensornvme/libs/libaio/lib/libaio.so
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/xihe/.tensornvme/TensorNVMe/cmake-build
  make: *** No rule to make target `extern_aio'.  Stop.
  No CUDA runtime is found, using CUDA_HOME='/cm/extra/Utils/CUDA/11.1.0.0_455.23.05'
  running install
  /datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  running build
  running build_py
  running build_ext
  building 'tensornvme._C' extension
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe
  creating /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc
  Emitting ninja build file /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/build.ninja...
  Compiling objects...
  Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
  [1/3] c++ -MMD -MF /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/space_mgr.o.d -pthread -B /datasets/xihe/miniconda3/envs/colossal/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -I/datasets/xihe/miniconda3/envs/colossal/include -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -fPIC -DDISABLE_URING -I/home/xihe/.tensornvme/TensorNVMe/csrc -I/home/xihe/.tensornvme/TensorNVMe/include -I/home/xihe/.tensornvme/include -I/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/TH -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/THC -I/datasets/xihe/miniconda3/envs/colossal/include/python3.9 -c -c /home/xihe/.tensornvme/TensorNVMe/csrc/space_mgr.cpp -o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/space_mgr.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  [2/3] c++ -MMD -MF /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/aio.o.d -pthread -B /datasets/xihe/miniconda3/envs/colossal/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -I/datasets/xihe/miniconda3/envs/colossal/include -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -fPIC -DDISABLE_URING -I/home/xihe/.tensornvme/TensorNVMe/csrc -I/home/xihe/.tensornvme/TensorNVMe/include -I/home/xihe/.tensornvme/include -I/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/TH -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/THC -I/datasets/xihe/miniconda3/envs/colossal/include/python3.9 -c -c /home/xihe/.tensornvme/TensorNVMe/csrc/aio.cpp -o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/aio.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  [3/3] c++ -MMD -MF /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/offload.o.d -pthread -B /datasets/xihe/miniconda3/envs/colossal/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -I/datasets/xihe/miniconda3/envs/colossal/include -fPIC -O2 -isystem /datasets/xihe/miniconda3/envs/colossal/include -fPIC -DDISABLE_URING -I/home/xihe/.tensornvme/TensorNVMe/csrc -I/home/xihe/.tensornvme/TensorNVMe/include -I/home/xihe/.tensornvme/include -I/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/TH -I/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/include/THC -I/datasets/xihe/miniconda3/envs/colossal/include/python3.9 -c -c /home/xihe/.tensornvme/TensorNVMe/csrc/offload.cpp -o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/offload.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
  g++ -pthread -B /datasets/xihe/miniconda3/envs/colossal/compiler_compat -shared -Wl,-rpath,/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath-link,/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath,/datasets/xihe/miniconda3/envs/colossal/lib -Wl,-rpath-link,/datasets/xihe/miniconda3/envs/colossal/lib -L/datasets/xihe/miniconda3/envs/colossal/lib /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/aio.o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/offload.o /home/xihe/.tensornvme/TensorNVMe/build/temp.linux-x86_64-cpython-39/home/xihe/.tensornvme/TensorNVMe/csrc/space_mgr.o -L/home/xihe/.tensornvme/lib -L/datasets/xihe/miniconda3/envs/colossal/lib/python3.9/site-packages/torch/lib -laio -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-39/tensornvme/_C.cpython-39-x86_64-linux-gnu.so
  /datasets/xihe/miniconda3/envs/colossal/compiler_compat/ld: cannot find -laio: No such file or directory
  collect2: error: ld returned 1 exit status
  error: command '/cm/extra/apps/GCC/7.3.0/GCC-4.8.5_GMP-6.1.2_MPFR-4.0.1_MPC-1.1.0/bin/g++' failed with exit code 1
  error: subprocess-exited-with-error
  
  × Running setup.py install for tensornvme did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /datasets/xihe/miniconda3/envs/colossal/bin/python -u -c '
  exec(compile('"'"''"'"''"'"'
  # This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
  #
  # - It imports setuptools before invoking setup.py, to enable projects that directly
  #   import from `distutils.core` to work with newer packaging standards.
  # - It provides a clear error message when setuptools is not installed.
  # - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
  #   setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
  #     manifest_maker: standard file '"'"'-c'"'"' not found".
  # - It generates a shim setup.py, for handling setup.cfg-only projects.
  import os, sys, tokenize
  
  try:
      import setuptools
  except ImportError as error:
      print(
          "ERROR: Can not execute `setup.py` since setuptools is not available in "
          "the build environment.",
          file=sys.stderr,
      )
      sys.exit(1)
  
  __file__ = %r
  sys.argv[0] = __file__
  
  if os.path.exists(__file__):
      filename = __file__
      with tokenize.open(__file__) as f:
          setup_py_code = f.read()
  else:
      filename = "<auto-generated setuptools caller>"
      setup_py_code = "from setuptools import setup; setup()"
  
  exec(compile(setup_py_code, filename, "exec"))
  '"'"''"'"''"'"' % ('"'"'/home/xihe/.tensornvme/TensorNVMe/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' install --record /tmp/pip-record-v1xk0mb3/install-record.txt --single-version-externally-managed --compile --install-headers /datasets/xihe/miniconda3/envs/colossal/include/python3.9/tensornvme
  cwd: /home/xihe/.tensornvme/TensorNVMe/
  Running setup.py install for tensornvme ... error
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> tensornvme

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.