v 0.3 fails to install

[ausstein]

Describe the bug
Installing the newest version gives errors

I am in a slurm environment without sudo rights

first I got an error since the /tmp folder only has 5GB and the installation process needed more, so i specified a custom tmp folder in my home directory. then I tried again and got another error. the full log is attached at the bottom. the relevant message seems to be

from /home/ausstein/github/torchTT/cpp/cpp_ext.cpp:1:
/usr/include/features-time64.h:20:10: fatal error: bits/wordsize.h: No such file or directory

for now I went back to 886ef89 which I had installed previously and still works fine

To Reproduce
Steps to reproduce the behavior:

1. create clean environment conda create -n torchTT pytorch pytest numpy opt_einsum scipy ninja numba tqdm matplotlib snakeviz pytorch-cuda=12.4 -c pytorch -c nvidia

2. conda activate torchTT

3. git clone https://github.com/ion-g-ion/torchTT

4. TMPDIR=~/my_tmp pip install ./torchTT/

5. See error

Expected behavior
Installation runs smoothly

Desktop

• OS: Ubuntu
• Version: 22.04.4 LTS

Error Log
building 'torchttcpp' extension
creating /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp
Emitting ninja build file /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o.d -pthread -B /home/ausstein/miniconda3/envs/torchHEOM/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -Icpp -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/TH -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/THC -I/home/ausstein/miniconda3/envs/torchHEOM/include/python3.11 -c -c /home/ausstein/github/torchTT/cpp/cpp_ext.cpp -o /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o -std=c++17 -Wno-c++11-narrowing -w -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=torchttcpp -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o
c++ -MMD -MF /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o.d -pthread -B /home/ausstein/miniconda3/envs/torchHEOM/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -O2 -isystem /home/ausstein/miniconda3/envs/torchHEOM/include -fPIC -Icpp -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/TH -I/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/THC -I/home/ausstein/miniconda3/envs/torchHEOM/include/python3.11 -c -c /home/ausstein/github/torchTT/cpp/cpp_ext.cpp -o /home/ausstein/github/torchTT/build/temp.linux-x86_64-cpython-311/cpp/cpp_ext.o -std=c++17 -Wno-c++11-narrowing -w -O3 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=torchttcpp -D_GLIBCXX_USE_CXX11_ABI=0
In file included from /usr/include/features.h:392,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/x86_64-pc-linux-gnu/bits/os_defines.h:39,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/x86_64-pc-linux-gnu/bits/c++config.h:679,
from /home/ausstein/local/gcc-13.1.0/include/c++/13.1.0/cstddef:49,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/c10/core/DeviceType.h:10,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/c10/core/Device.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/core/TensorBody.h:11,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/core/Tensor.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/ATen/Tensor.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/function_hook.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/cpp_hook.h:2,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/variable.h:6,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/autograd/autograd.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/autograd.h:3,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/csrc/api/include/torch/all.h:7,
from /home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/torch/include/torch/extension.h:5,
from /home/ausstein/github/torchTT/cpp/define.h:3,
from /home/ausstein/github/torchTT/cpp/full.h:1,
from /home/ausstein/github/torchTT/cpp/cpp_ext.cpp:1:
/usr/include/features-time64.h:20:10: fatal error: bits/wordsize.h: No such file or directory
20 | #include <bits/wordsize.h>
| ^~~~~~~~~~~~~~~~~
compilation terminated.
ninja: build stopped: subcommand failed.
:60: UserWarning:
C++ implementation not available. Falling back to pure Python.

  Error: Error compiling objects for extension
  Traceback (most recent call last):
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/miniconda3/envs/torchHEOM/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
      return self._build_with_temp_dir(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 409, in _build_with_temp_dir
      result_basename = _file_with_extension(tmp_dist_dir, result_extension)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/ausstein/my_tmp/pip-build-env-gvbj456p/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 128, in _file_with_extension
      raise ValueError(
  ValueError: No distribution was found. Ensure that `setup.py` is not empty and that it calls `setup()`.
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torchTT
Failed to build torchTT
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (torchTT)

[MazenAli]

I think this is due to a lack of pre-installed build tools, had the same issue. Here is a docker setup that works:

FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y \
    build-essential \
    cmake \
    g++ \
    ninja-build \
    libopenblas-dev \
    liblapack-dev \
    libfftw3-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt ./

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r requirements.txt

RUN rm requirements.txt

[ausstein]

@MazenAli as i said a previous version just a few commits back compiles and runs perfectly, are any of these build tools new requirements for v0.3?

[MazenAli]

I can't say for sure without checking the source code for changes. torchtt uses some c++ compiled routines for dmrg, or defaults to a python implementation. In the previous github version, I would get a warning that the implementation defaults to python. In this one, I don't. Which tells me that the github clone by default installed the python implementation whereas the pypi version by default tries to compile the c++ version.

Python dependency management, unfortunately, can't handle non-python build tools. The author could consider defaulting back to the python build with pypi for ease of install.

For you, the simplest workaround would be to use a docker container. Or use the old version ofc.

[ausstein]

I do have it compiled with c++ backend on the old version. For now I'll stay on that version.

…

On Tue, Dec 10, 2024, 9:05 PM Mazen Ali ***@***.***> wrote: I can't say for sure without checking the source code for changes. torchtt uses some c++ compiled routines for dmrg, or defaults to a python implementation. In the previous github version, I would get a warning that the implementation defaults to python. In this one, I don't. Which tells me that the github clone by default installed the python implementation whereas the pypi version by default tries to compile the c++ version. Python dependency management, unfortunately, can't handle non-python build tools. The author could consider defaulting back to the python build with pypi for ease of install. For you, the simplest workaround would be to use a docker container. Or use the old version ofc. — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVFIYQW2XZPG2BOWSA22MM32E5CSFAVCNFSM6AAAAABTLLPLF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZSG42TMNJTGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ion-g-ion]

I did some changes to the setup.py to address this issue. Now it should work fine even if the c++ extensions cannot be compiled.

BR,
Ion